[{"data":1,"prerenderedAt":115309},["ShallowReactive",2],{"/blog/tags/vllm/":3},[4,624,1827,11137,11807,12650,14057,16108,19407,21300,21581,22464,23781,25740,30131,33598,34914,35966,36043,36487,40769,46093,46127,46597,47208,47533,47674,48051,48850,49199,51681,54430,57512,57580,59388,64608,65996,66582,68510,69034,69885,70669,72183,72373,74440,74513,74903,75866,76757,77023,81003,83574,83617,83658,90257,91996,95214,115038],{"id":5,"title":6,"body":7,"comments":602,"date":603,"description":604,"draft":602,"extension":605,"external":606,"image":607,"meta":608,"navigation":609,"path":610,"seo":611,"stem":612,"tags":613,"__hash__":623},"blog/2026/06/18/running-mmlu-5shot-on-nemotron-nano-omni-with-a-dgx-spark.md","Running MMLU 5-shot on Nemotron Nano Omni with a DGX Spark",{"type":8,"value":9,"toc":592},"minimark",[10,44,55,60,72,75,121,134,138,141,147,168,173,192,196,199,209,219,227,236,243,251,263,284,288,291,296,318,321,326,330,341,346,349,353,364,369,372,377,381,399,402,455,458,556,559,563,580,588],[11,12,13,14,18,19,26,27,33,34,37,38,43],"p",{},"This is the first proper LLM ",[15,16,17],"strong",{},"evaluation"," I've run end-to-end, and I learned a ton — both about how evals actually work under the hood and about the infrastructure quirks of running one against a model hosted on my ",[20,21,25],"a",{"href":22,"rel":23},"https://www.nvidia.com/en-us/products/workstations/dgx-spark/",[24],"nofollow","DGX Spark",". I scored ",[15,28,29],{},[30,31,32],"code",{},"nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4"," on ",[15,35,36],{},"MMLU 5-shot"," and submitted the result to ",[20,39,42],{"href":40,"rel":41},"https://www.localmaxxing.com/en/evals/mmlu-5shot",[24],"localmaxxing.com",". This post walks through the result with interactive charts, a few example questions, and the setup that got me there.",[11,45,46,49,50,54],{},[15,47,48],{},"Headline: 73.5%"," (95% CI 72.8–74.2, over all 14,042 questions). Random guessing is 25%, so for a sparse MoE model with only ~3B ",[51,52,53],"em",{},"active"," parameters running at 4-bit, that's a genuinely strong score.",[56,57,59],"h2",{"id":58},"what-mmlu-actually-is","What MMLU actually is",[11,61,62,67,68,71],{},[20,63,66],{"href":64,"rel":65},"https://github.com/hendrycks/test",[24],"MMLU"," (Massive Multitask Language Understanding) is ",[15,69,70],{},"14,042 multiple-choice questions across 57 subjects"," — everything from abstract algebra to professional law to marketing — grouped into four broad categories (STEM, Humanities, Social Sciences, Other). It's a broad knowledge-and-reasoning test.",[11,73,74],{},"A couple of things that surprised me as a first-timer:",[76,77,78,89],"ul",{},[79,80,81,84,85,88],"li",{},[15,82,83],{},"\"5-shot\""," means every question is preceded by ",[15,86,87],{},"5 fully worked examples"," from the same subject, so the model learns the answer format before it sees the real question. Those few-shot prefixes make the prompts long (more on that below).",[79,90,91,94,95,98,99,102,103,106,107,106,110,106,113,116,117,120],{},[15,92,93],{},"The model never \"writes\" an answer."," This is the part I didn't expect. The standard MMLU task is scored by ",[15,96,97],{},"log-likelihood",": we feed the prompt ending in ",[30,100,101],{},"Answer:"," and measure the probability the model assigns to each of ",[30,104,105],{},"A",", ",[30,108,109],{},"B",[30,111,112],{},"C",[30,114,115],{},"D",". Whichever letter gets the highest probability is the model's pick. Because we're reading probabilities directly — not generating text — ",[15,118,119],{},"temperature and sampling settings are irrelevant",", and the model's \"reasoning\" mode never even fires. The score is just the fraction of questions where the top-probability letter matches the answer key.",[11,122,123,124,129,130,133],{},"I ran this with ",[20,125,128],{"href":126,"rel":127},"https://github.com/EleutherAI/lm-evaluation-harness",[24],"EleutherAI's lm-evaluation-harness"," (",[30,131,132],{},"v0.4.9.1",", the version localmaxxing pins for this suite).",[56,135,137],{"id":136},"where-the-model-is-strong-and-weak","Where the model is strong and weak",[11,139,140],{},"Here's accuracy broken down by category. Social Sciences and \"Other\" are clearly its strong suits; STEM and Humanities trail.",[142,143,144],"client-only",{},[145,146],"mmlu-category-chart",{},[11,148,149,150,155,156,159,160,163,164,167],{},"The category averages hide a lot of variance, though. Here's every one of the 57 subjects, sorted weakest to strongest and colored by category (",[151,152,154],"span",{"style":153},"color:#4C72B0","■"," STEM · ",[151,157,154],{"style":158},"color:#DD8452"," Humanities · ",[151,161,154],{"style":162},"color:#55A868"," Social Sciences · ",[151,165,154],{"style":166},"color:#C44E52"," Other). Hover any bar for the exact accuracy and sample size:",[142,169,170],{},[171,172],"mmlu-subject-chart",{},[11,174,175,176,179,180,183,184,187,188,191],{},"The pattern is intuitive: it crushes broad, verbal, \"general knowledge\" subjects (",[15,177,178],{},"high-school government & politics 93%, psychology 90%, biology 89%",") and struggles with dense symbolic reasoning and niche trivia (",[15,181,182],{},"global facts 43%, abstract algebra 49%, high-school math 49%, formal logic 53%","). Humanities gets dragged down by ",[30,185,186],{},"formal_logic"," and ",[30,189,190],{},"professional_law",", which are really logic/reasoning tests in disguise.",[56,193,195],{"id":194},"a-few-questions-up-close","A few questions, up close",[11,197,198],{},"Numbers are abstract, so here are three actual questions to give a feel for what the model is being asked.",[11,200,201,204,205,208],{},[15,202,203],{},"It nailed this one (high confidence, correct)"," — ",[51,206,207],{},"miscellaneous",":",[210,211,212],"blockquote",{},[11,213,214,215,218],{},"What kind of angle is formed where two perpendicular lines meet?\nA. obtuse · B. acute · ",[15,216,217],{},"C. right ✓ (model picked C)"," · D. invisible",[11,220,221,204,224,208],{},[15,222,223],{},"A genuinely hard miss",[51,225,226],{},"abstract algebra",[210,228,229],{},[11,230,231,232,235],{},"Find the degree for the given field extension Q(√2, √3, √18) over Q.\nA. 0 · ",[15,233,234],{},"B. 4 ✓"," · C. 2 (model picked C) · D. 6",[11,237,238,239,242],{},"That one requires actually reasoning about field extensions — and note √18 = 3√2 is ",[51,240,241],{},"not"," independent of √2, a classic trap. The model fell for it.",[11,244,245,204,248,208],{},[15,246,247],{},"And one that taught me evals aren't ground truth",[51,249,250],{},"computer security",[210,252,253],{},[11,254,255,256,258,259,262],{},"Three of the following are classic security properties; which one is ",[15,257,241],{},"?\nA. Confidentiality · ",[15,260,261],{},"B. Availability ✓ (answer key)"," · C. Correctness (model picked C) · D. Integrity",[11,264,265,266,269,270,273,274,276,277,280,281,283],{},"The model picked ",[15,267,268],{},"C. Correctness"," — and it's ",[51,271,272],{},"right",". The classic security triad is ",[15,275,112],{},"onfidentiality, ",[15,278,279],{},"I","ntegrity, ",[15,282,105],{},"vailability (the \"CIA triad\"), so \"Correctness\" is the one that isn't classic. The answer key says B (Availability), which is simply wrong. MMLU has a well-documented amount of label noise like this, and it's a good reminder that a few points of any benchmark score are just dataset errors. The model got \"penalized\" here for being more correct than the test.",[56,285,287],{"id":286},"how-long-are-these-prompts-anyway","How long are these prompts, anyway?",[11,289,290],{},"Because of the 5-shot prefix, the prompts aren't short. Here's the token-length distribution (measured with the model's own tokenizer):",[142,292,293],{},[294,295],"mmlu-length-hist",{},[11,297,298,299,302,303,306,307,310,311,313,314,317],{},"Mean is ~702 tokens, but the long tail matters: the dense history and law subjects push the ",[15,300,301],{},"longest 5-shot prompt to 3,111 tokens",". This bit me before I started — lm-eval's default ",[30,304,305],{},"max_length"," is ",[15,308,309],{},"2048",", which would have silently truncated ~2.7% of prompts (concentrated in exactly those long-context subjects) and quietly cost me points. Bumping ",[30,312,305],{}," to ",[15,315,316],{},"4096"," captured 100% of prompts with zero truncation.",[11,319,320],{},"Does length actually hurt accuracy? A little, but it's mostly a confound — the longest prompts belong to the hardest subjects:",[142,322,323],{},[324,325],"mmlu-length-accuracy",{},[56,327,329],{"id":328},"how-confident-is-it-and-is-that-confidence-trustworthy","How confident is it — and is that confidence trustworthy?",[11,331,332,333,336,337,340],{},"Since scoring is probabilistic, I can measure ",[15,334,335],{},"confidence"," as the gap between the top choice's log-probability and the runner-up's. A well-behaved model should be ",[51,338,339],{},"more"," confident when it's right. It is — the \"correct\" mass sits clearly to the right of \"incorrect\":",[142,342,343],{},[344,345],"mmlu-confidence-chart",{},[11,347,348],{},"That separation is exactly what you want to see: when the model is unsure (small gap), it's much more likely to be wrong. It \"knows when it knows.\"",[56,350,352],{"id":351},"is-it-biased-toward-any-answer-letter","Is it biased toward any answer letter?",[11,354,355,356,359,360,363],{},"A classic failure mode is favoring a position (e.g., always leaning \"C\") regardless of content. Comparing how often each letter is the ",[51,357,358],{},"correct"," answer versus how often the model ",[51,361,362],{},"picks"," it, the bias here is mild — a slight lean toward A/B and away from D:",[142,365,366],{},[367,368],"mmlu-bias-chart",{},[11,370,371],{},"And when it's wrong, what does it confuse for what? The diagonal is correct picks; off-diagonal shows the (fairly uniform) confusions:",[142,373,374],{},[375,376],"mmlu-confusion-chart",{},[56,378,380],{"id":379},"the-setup-a-dgx-spark-k3s-vllm-and-an-off-box-eval-client","The setup: a DGX Spark, k3s, vLLM, and an off-box eval client",[11,382,383,384,386,387,390,391,394,395,398],{},"The infrastructure was half the adventure. The model runs on my ",[15,385,25],{}," (GB10 Grace Blackwell, 128 GB unified memory) as a ",[15,388,389],{},"vLLM"," pod inside a k3s cluster, served NVFP4-quantized with FP8 KV cache. The eval ",[51,392,393],{},"client"," (lm-eval) ran on my Mac, hitting vLLM over the LAN via the OpenAI-compatible ",[30,396,397],{},"/v1/completions"," endpoint.",[11,400,401],{},"A few hard-won lessons:",[76,403,404,425,439,445],{},[79,405,406,409,410,413,414,417,418,421,422,424],{},[15,407,408],{},"Use the completions endpoint, not chat."," Log-likelihood / MCQ scoring needs prompt log-probs (",[30,411,412],{},"echo"," + ",[30,415,416],{},"logprobs","), which chat-completion APIs don't expose. ",[30,419,420],{},"local-completions"," against ",[30,423,397],{}," is the way.",[79,426,427,434,435,438],{},[15,428,429,430,433],{},"Run the eval client ",[51,431,432],{},"off"," the Spark."," vLLM pins ~103 GB of the 128 GB unified memory. My first instinct — run lm-eval on the Spark itself — pushed it into a swap death-spiral that throttled vLLM to ",[51,436,437],{},"25 seconds per request",". Moving the client to my Mac (the GPU work stays on the Spark; only orchestration + tokenization moves) fixed it instantly.",[79,440,441,444],{},[15,442,443],{},"The tokenizer must match the server."," I pointed lm-eval at the model's own tokenizer so the context-length bookkeeping for log-prob scoring lined up byte-for-byte with vLLM. Mismatches there silently corrupt scores.",[79,446,447,450,451,454],{},[15,448,449],{},"Checkpoint long runs."," The Spark rebooted ~90 minutes in. ",[30,452,453],{},"--use_cache"," meant the completed requests were banked, and a resilient wrapper resumed from the cache and finished the remaining work without losing anything.",[11,456,457],{},"The actual command, for the curious:",[459,460,465],"pre",{"className":461,"code":462,"language":463,"meta":464,"style":464},"language-bash shiki shiki-themes github-light github-dark monokai","lm_eval --model local-completions \\\n  --model_args model=nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4,\\\nbase_url=http://\u003Cspark>:8000/v1/completions,tokenizer=\u003Clocal-tokenizer>,\\\nnum_concurrent=16,max_length=4096,tokenized_requests=False \\\n  --use_cache ~/mmlu_cache --cache_requests true \\\n  --tasks mmlu --num_fewshot 5 --output_path ~/mmlu-full --log_samples\n","bash","",[30,466,467,486,498,507,515,532],{"__ignoreMap":464},[151,468,471,475,479,483],{"class":469,"line":470},"line",1,[151,472,474],{"class":473},"srTi1","lm_eval",[151,476,478],{"class":477},"s7F3e"," --model",[151,480,482],{"class":481},"sstjo"," local-completions",[151,484,485],{"class":477}," \\\n",[151,487,489,492,495],{"class":469,"line":488},2,[151,490,491],{"class":477},"  --model_args",[151,493,494],{"class":481}," model=nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4,",[151,496,497],{"class":477},"\\\n",[151,499,501,505],{"class":469,"line":500},3,[151,502,504],{"class":503},"sMOD_","base_url=http://\u003Cspark>:8000/v1/completions,tokenizer=\u003Clocal-tokenizer>,",[151,506,497],{"class":477},[151,508,510,513],{"class":469,"line":509},4,[151,511,512],{"class":503},"num_concurrent=16,max_length=4096,tokenized_requests=False ",[151,514,497],{"class":477},[151,516,518,521,524,527,530],{"class":469,"line":517},5,[151,519,520],{"class":477},"  --use_cache",[151,522,523],{"class":481}," ~/mmlu_cache",[151,525,526],{"class":477}," --cache_requests",[151,528,529],{"class":477}," true",[151,531,485],{"class":477},[151,533,535,538,541,544,547,550,553],{"class":469,"line":534},6,[151,536,537],{"class":477},"  --tasks",[151,539,540],{"class":481}," mmlu",[151,542,543],{"class":477}," --num_fewshot",[151,545,546],{"class":477}," 5",[151,548,549],{"class":477}," --output_path",[151,551,552],{"class":481}," ~/mmlu-full",[151,554,555],{"class":477}," --log_samples\n",[11,557,558],{},"End to end, the full 14,042-question run took roughly two hours of GPU time on the GB10 at a sustained ~6,500 prompt-tokens/sec.",[56,560,562],{"id":561},"takeaways","Takeaways",[76,564,565,571,574,577],{},[79,566,567,570],{},[15,568,569],{},"73.5% on MMLU 5-shot"," is a strong result for a ~3B-active, 4-bit model — competitive with much larger dense models from a year ago.",[79,572,573],{},"Its profile is \"broad knowledge generalist\": excellent at verbal/social subjects, weaker at symbolic math and logic.",[79,575,576],{},"Evals are not ground truth — label noise is real, and a model can be marked wrong for being right.",[79,578,579],{},"The plumbing matters as much as the model: endpoint choice, where the client runs, tokenizer alignment, and checkpointing all materially affect whether you get a clean number.",[11,581,582,583,587],{},"The run is submitted to the ",[20,584,586],{"href":40,"rel":585},[24],"localmaxxing MMLU 5-shot leaderboard",". All 14,042 per-question records (prompt, choices, the four log-likelihoods, prediction, confidence) are retained, so every chart above is reproducible from the raw data.",[589,590,591],"style",{},"html pre.shiki code .srTi1, html code.shiki .srTi1{--shiki-default:#6F42C1;--shiki-dark:#B392F0;--shiki-sepia:#A6E22E}html pre.shiki code .s7F3e, html code.shiki .s7F3e{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#AE81FF}html pre.shiki code .sstjo, html code.shiki .sstjo{--shiki-default:#032F62;--shiki-dark:#9ECBFF;--shiki-sepia:#E6DB74}html pre.shiki code .sMOD_, html code.shiki .sMOD_{--shiki-default:#24292E;--shiki-dark:#E1E4E8;--shiki-sepia:#F8F8F2}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html .sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html.sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}",{"title":464,"searchDepth":488,"depth":488,"links":593},[594,595,596,597,598,599,600,601],{"id":58,"depth":488,"text":59},{"id":136,"depth":488,"text":137},{"id":194,"depth":488,"text":195},{"id":286,"depth":488,"text":287},{"id":328,"depth":488,"text":329},{"id":351,"depth":488,"text":352},{"id":379,"depth":488,"text":380},{"id":561,"depth":488,"text":562},false,"2026-06-18","My first LLM eval end-to-end: scoring NVIDIA's Nemotron-3-Nano-Omni-30B (NVFP4) on MMLU 5-shot via vLLM on a DGX Spark, with interactive charts breaking down where it's strong, where it's weak, and how confident it is.","md",null,"/static/mmlu/cover.png",{},true,"/2026/06/18/running-mmlu-5shot-on-nemotron-nano-omni-with-a-dgx-spark",{"title":6,"description":604},"2026/06/18/running-mmlu-5shot-on-nemotron-nano-omni-with-a-dgx-spark",[614,615,616,617,618,619,620,621,622],"ai","llm","eval","mmlu","vllm","nemotron","dgx-spark","k8s","localmaxxing","PsXJuIY08tve_S6kaBuTrKgd4824cbS7pNHkWQDcwBc",{"id":625,"title":626,"body":627,"comments":609,"date":1814,"description":1815,"draft":602,"extension":605,"external":606,"image":1816,"meta":1817,"navigation":609,"path":1818,"seo":1819,"stem":1820,"tags":1821,"__hash__":1826},"blog/2026/03/09/blog-meta-about-openclaw-blog-cleanup.md","Behind the Scenes: AI-Assisted Blog Cleanup with OpenClaw",{"type":8,"value":628,"toc":1762},[629,633,644,647,650,653,657,664,696,699,711,722,724,728,735,740,758,764,768,785,789,793,804,809,811,815,818,822,825,829,839,842,868,872,875,895,899,906,912,914,918,925,929,948,952,955,969,975,977,981,988,992,1000,1004,1030,1032,1036,1039,1043,1061,1067,1071,1074,1096,1101,1105,1108,1119,1124,1126,1130,1198,1211,1213,1217,1223,1227,1235,1239,1250,1254,1265,1269,1280,1282,1286,1289,1293,1307,1311,1327,1331,1347,1351,1367,1369,1373,1376,1422,1429,1431,1435,1438,1442,1445,1459,1463,1466,1486,1490,1493,1497,1504,1563,1567,1570,1572,1576,1684,1686,1690,1697,1700,1714,1724,1726,1730,1733,1744,1747,1749,1759],[56,630,632],{"id":631},"introduction","Introduction",[11,634,635,636,639,640,643],{},"You might have noticed some subtle improvements to articles on this blog over the past few days. Typos fixed, security best practices added, version disclaimers included. But you probably don't know ",[15,637,638],{},"how"," these changes are happening or ",[15,641,642],{},"why",".",[11,645,646],{},"This article pulls back the curtain on my new AI-powered blog maintenance system built with OpenClaw — a multi-agent orchestration framework running entirely on my DGX Spark through Telegram. No cloud APIs, no paid services, just local LLMs and file-based communication doing the heavy lifting.",[11,648,649],{},"Let me show you how it works.",[651,652],"hr",{},[56,654,656],{"id":655},"the-problem-a-blog-that-needs-love-but-not-constant-manual-review","The Problem: A Blog That Needs Love (But Not Constant Manual Review)",[11,658,659,660,663],{},"My personal blog at ",[30,661,662],{},"briancaffey.github.io"," has 56 articles spanning from 2016 to 2026. Over time, they accumulate issues that I don't have the bandwidth to fix manually:",[76,665,666,672,678,684,690],{},[79,667,668,671],{},[15,669,670],{},"Typos and grammar errors"," — slipped through during rushed writing sessions",[79,673,674,677],{},[15,675,676],{},"Broken links"," — external resources archived or moved",[79,679,680,683],{},[15,681,682],{},"Outdated versions"," — software references from years ago",[79,685,686,689],{},[15,687,688],{},"Missing security notes"," — best practices that should accompany technical tutorials",[79,691,692,695],{},[15,693,694],{},"Inconsistent formatting"," — across articles written at different times",[11,697,698],{},"I could fix these myself, but that means:",[700,701,702,705,708],"ol",{},[79,703,704],{},"Reading through 56 articles (hundreds of hours)",[79,706,707],{},"Testing every code snippet to ensure it still works",[79,709,710],{},"Manually creating and reviewing PRs for each change",[11,712,713,714,717,718,721],{},"That's not sustainable. I needed a system that could ",[15,715,716],{},"continuously monitor"," my blog and ",[15,719,720],{},"proactively suggest improvements",". Enter OpenClaw.",[651,723],{},[56,725,727],{"id":726},"the-solution-a-multi-agent-system-with-specialized-roles","The Solution: A Multi-Agent System with Specialized Roles",[11,729,730,731,734],{},"OpenClaw is designed around the principle of ",[15,732,733],{},"sequential orchestration"," — one agent at a time, working through tasks in an organized pipeline. For the blog cleanup project, I've set up three specialized roles:",[736,737,739],"h3",{"id":738},"_1-orchestrator-agent-me-klaw","1. Orchestrator Agent (Me - Klaw)",[76,741,742,749,752,755],{},[79,743,744,745,748],{},"Reads the task queue (",[30,746,747],{},"todo/blog-cleanup.md",")",[79,750,751],{},"Assigns articles to review workers",[79,753,754],{},"Manages context between sessions via file-based communication",[79,756,757],{},"Creates PR branches and opens pull requests",[11,759,760,763],{},[15,761,762],{},"Running on:"," DGX Spark, accessed through Telegram",[736,765,767],{"id":766},"_2-review-worker-agent","2. Review Worker Agent",[76,769,770,773,776,782],{},[79,771,772],{},"Performs systematic article reviews using a standardized checklist",[79,774,775],{},"Checks for: typos, broken links, code accuracy, outdated content, security issues",[79,777,778,779,748],{},"Documents findings in structured review files (",[30,780,781],{},"todo/reviews/",[79,783,784],{},"Prioritizes fixes by impact and urgency",[11,786,787,763],{},[15,788,762],{},[736,790,792],{"id":791},"_3-pr-creation-agent","3. PR Creation Agent",[76,794,795,798,801],{},[79,796,797],{},"Commits changes to git branches",[79,799,800],{},"Creates pull requests with detailed descriptions",[79,802,803],{},"Links back to the project task queue for tracking",[11,805,806,808],{},[15,807,762],{}," DGX Spark, accessed through Telegram (same hardware!)",[651,810],{},[56,812,814],{"id":813},"how-it-works-the-review-pipeline","How It Works: The Review Pipeline",[11,816,817],{},"Here's what happens when I review an article:",[736,819,821],{"id":820},"step-1-article-selection","Step 1: Article Selection",[11,823,824],{},"I pick an article from the task queue. For example, the NVIDIA NIM tutorial was chosen first because it's recent and technical (high priority for accuracy).",[736,826,828],{"id":827},"step-2-systematic-analysis","Step 2: Systematic Analysis",[11,830,831,832,835,836,838],{},"The Review Worker reads the entire article using our ",[15,833,834],{},"file-based context management"," system. No context overflow — everything is written to files (",[30,837,781],{},") so each agent session stays lean.",[11,840,841],{},"The review checklist covers eight categories:",[700,843,844,847,850,853,856,859,862,865],{},[79,845,846],{},"Typos & Grammar (highest priority)",[79,848,849],{},"Broken Links & References",[79,851,852],{},"Code Accuracy & Completeness",[79,854,855],{},"Outdated Content",[79,857,858],{},"SEO & Metadata",[79,860,861],{},"Logical Flow & Clarity",[79,863,864],{},"Security Considerations",[79,866,867],{},"Accessibility & Formatting",[736,869,871],{"id":870},"step-3-finding-issues","Step 3: Finding Issues",[11,873,874],{},"For the NVIDIA NIM article, we found:",[76,876,877,883,889],{},[79,878,879,882],{},[15,880,881],{},"Typos:"," \"get go through\" → \"go through\", \"caputres\" → \"captures\"",[79,884,885,888],{},[15,886,887],{},"Security gap:"," No note about environment variables and credential handling",[79,890,891,894],{},[15,892,893],{},"Version drift:"," Kubernetes version shown would age quickly for readers",[736,896,898],{"id":897},"step-4-creating-the-pr","Step 4: Creating the PR",[11,900,901,902,905],{},"The PR Creation Agent commits fixes to a new branch (",[30,903,904],{},"blog-cleanup/nvidia-nim-improvements",") and opens pull request #8 with detailed descriptions of what changed and why.",[11,907,908,911],{},[15,909,910],{},"Result:"," I reviewed it on GitHub, approved the changes, merged it — all in under an hour.",[651,913],{},[56,915,917],{"id":916},"the-magic-running-locally-through-telegram","The Magic: Running Locally Through Telegram",[11,919,920,921,924],{},"Here's where this gets exciting: ",[15,922,923],{},"everything runs on my DGX Spark",", accessed through Telegram. No cloud APIs. No paid services. Just local LLMs doing the work.",[736,926,928],{"id":927},"hardware-setup","Hardware Setup",[76,930,931,936,942],{},[79,932,933,935],{},[15,934,25],{}," — NVIDIA's compact AI workstation with GB10 chip",[79,937,938,941],{},[15,939,940],{},"Qwen3.5-35B-A3B (3 billion active parameters)"," — My primary model running via LM Studio",[79,943,944,947],{},[15,945,946],{},"1 concurrent LLM request"," — The DGX Spark limitation that actually makes this system simpler",[736,949,951],{"id":950},"why-telegram","Why Telegram?",[11,953,954],{},"Telegram is my communication hub. I can:",[76,956,957,960,963,966],{},[79,958,959],{},"Start new review sessions with a single message",[79,961,962],{},"Get status updates and PR links instantly",[79,964,965],{},"Steer the project direction without leaving chat",[79,967,968],{},"Keep everything in one place (messages, files, code)",[11,970,971,974],{},[15,972,973],{},"No context switching."," No juggling between IDEs, terminals, browsers. Just Telegram.",[651,976],{},[56,978,980],{"id":979},"file-based-communication-the-secret-sauce","File-Based Communication: The Secret Sauce",[11,982,983,984,987],{},"The key to making this work is ",[15,985,986],{},"file-based state management",". Instead of trying to pass massive context between agents (which would overflow tokens and lose focus), we use files as the \"bus\" that all agents read/write to.",[736,989,991],{"id":990},"core-files","Core Files",[459,993,998],{"className":994,"code":996,"language":997},[995],"language-text","todo/\n├── blog-cleanup.md          # Master task queue with 5 phases\n├── blog-cleanup-inventory.md # Article inventory and categorization\n└── reviews/\n    ├── 2026-02-nvidia-nim-review.md\n    ├── 2025-07-nuxt4-upgrade-review.md\n    └── 2025-09-hnfm-review.md\n\nTODO.md                      # Root-level project tracking (visible on GitHub)\n","text",[30,999,996],{"__ignoreMap":464},[736,1001,1003],{"id":1002},"why-this-works","Why This Works",[700,1005,1006,1012,1018,1024],{},[79,1007,1008,1011],{},[15,1009,1010],{},"No context overflow"," — Each agent session stays focused and lean",[79,1013,1014,1017],{},[15,1015,1016],{},"Persistent state"," — Reviews survive between sessions",[79,1019,1020,1023],{},[15,1021,1022],{},"Human-readable"," — I can inspect findings without debugging logs",[79,1025,1026,1029],{},[15,1027,1028],{},"Version controlled"," — All files tracked in git, review history preserved",[651,1031],{},[56,1033,1035],{"id":1034},"what-kind-of-improvements-am-i-finding","What Kind of Improvements Am I Finding?",[11,1037,1038],{},"Let me show you the types of fixes OpenClaw is making:",[736,1040,1042],{"id":1041},"typos-grammar-most-common","Typos & Grammar (Most Common)",[76,1044,1045,1055,1058],{},[79,1046,1047,1048,1051,1052,1054],{},"\"tha new ",[30,1049,1050],{},"/app"," directory\" → \"the new ",[30,1053,1050],{}," directory\"",[79,1056,1057],{},"\"rarranged my folder to match watch\" → \"rearranged my folder to match what\"",[79,1059,1060],{},"\"get go through\" → \"go through\"",[11,1062,1063,1066],{},[15,1064,1065],{},"Impact:"," Small changes, but they matter for credibility. Readers notice when articles feel polished.",[736,1068,1070],{"id":1069},"security-best-practices-high-value","Security Best Practices (High Value)",[11,1072,1073],{},"Added a new section to the NVIDIA NIM tutorial:",[459,1075,1079],{"className":1076,"code":1077,"language":1078,"meta":464,"style":464},"language-markdown shiki shiki-themes github-light github-dark monokai","## Security Best Practices & Disclaimers\n\n**Security Note:** This tutorial uses environment variables for configuration, which is good practice. Never commit `.env` files with real credentials to version control. Use GCP IAM roles and service accounts for production deployments.\n","markdown",[30,1080,1081,1086,1091],{"__ignoreMap":464},[151,1082,1083],{"class":469,"line":470},[151,1084,1085],{},"## Security Best Practices & Disclaimers\n",[151,1087,1088],{"class":469,"line":488},[151,1089,1090],{"emptyLinePlaceholder":609},"\n",[151,1092,1093],{"class":469,"line":500},[151,1094,1095],{},"**Security Note:** This tutorial uses environment variables for configuration, which is good practice. Never commit `.env` files with real credentials to version control. Use GCP IAM roles and service accounts for production deployments.\n",[11,1097,1098,1100],{},[15,1099,1065],{}," Helps readers avoid common security mistakes. Takes 5 minutes to write but saves hours of potential damage.",[736,1102,1104],{"id":1103},"version-disclaimers-long-term-value","Version Disclaimers (Long-Term Value)",[11,1106,1107],{},"Added notes like:",[210,1109,1110],{},[11,1111,1112,1113,1118],{},"\"Note: The Kubernetes version shown (1.35.0) is current as of February 2026. Check ",[20,1114,1117],{"href":1115,"rel":1116},"https://cloud.google.com/kubernetes-engine/docs/release-notes",[24],"GKE release notes"," for the latest versions.\"",[11,1120,1121,1123],{},[15,1122,1065],{}," Sets reader expectations about rapidly-changing cloud services. Article remains useful even as specific versions age.",[651,1125],{},[56,1127,1129],{"id":1128},"the-results-so-far-3-articles-3-prs-merged","The Results So Far (3 Articles, 3 PRs Merged!)",[1131,1132,1133,1152],"table",{},[1134,1135,1136],"thead",{},[1137,1138,1139,1143,1146,1149],"tr",{},[1140,1141,1142],"th",{},"Article",[1140,1144,1145],{},"Priority",[1140,1147,1148],{},"Fixes Made",[1140,1150,1151],{},"Status",[1153,1154,1155,1170,1184],"tbody",{},[1137,1156,1157,1161,1164,1167],{},[1158,1159,1160],"td",{},"NVIDIA NIM on GCP",[1158,1162,1163],{},"High",[1158,1165,1166],{},"Typos + Security notes + Version disclaimer",[1158,1168,1169],{},"✅ Merged #8",[1137,1171,1172,1175,1178,1181],{},[1158,1173,1174],{},"Nuxt 4 Upgrade",[1158,1176,1177],{},"Medium",[1158,1179,1180],{},"2 typos fixed",[1158,1182,1183],{},"✅ Merged #9",[1137,1185,1186,1189,1192,1195],{},[1158,1187,1188],{},"hnfm Hackathon Project",[1158,1190,1191],{},"Critical",[1158,1193,1194],{},"Completed cut-off sentence, formatting fixes",[1158,1196,1197],{},"✅ Merged #10",[11,1199,1200,1203,1204,1207,1210],{},[15,1201,1202],{},"Time spent:"," ~30 minutes per article (including review and PR creation)",[1205,1206],"br",{},[15,1208,1209],{},"Manual effort:"," I only reviewed the PRs on GitHub. Everything else was automated through OpenClaw.",[651,1212],{},[56,1214,1216],{"id":1215},"why-this-matters-local-ai-for-real-work","Why This Matters: Local AI for Real Work",[11,1218,1219,1220,208],{},"I could have used an LLM API service to do this review work. But there are several reasons I chose ",[15,1221,1222],{},"local inference",[736,1224,1226],{"id":1225},"cost","Cost",[76,1228,1229,1232],{},[79,1230,1231],{},"Cloud APIs charge per token — reviewing 56 articles would add up quickly",[79,1233,1234],{},"Local inference has zero marginal cost after hardware investment",[736,1236,1238],{"id":1237},"privacy","Privacy",[76,1240,1241,1244,1247],{},[79,1242,1243],{},"No blog content leaves my machine",[79,1245,1246],{},"No proprietary data sent to third-party services",[79,1248,1249],{},"Full control over what gets processed",[736,1251,1253],{"id":1252},"ownership","Ownership",[76,1255,1256,1259,1262],{},[79,1257,1258],{},"I own the models, the prompts, the workflow",[79,1260,1261],{},"Can modify anything without API rate limits or terms of service constraints",[79,1263,1264],{},"The system evolves with my needs, not a vendor's roadmap",[736,1266,1268],{"id":1267},"learning","Learning",[76,1270,1271,1274,1277],{},[79,1272,1273],{},"Running this locally forces me to understand how agents work",[79,1275,1276],{},"File-based communication teaches state management patterns",[79,1278,1279],{},"Debugging failures builds deeper intuition about LLM behavior",[651,1281],{},[56,1283,1285],{"id":1284},"whats-next-53-articles-remaining","What's Next? (53 Articles Remaining!)",[11,1287,1288],{},"The systematic review is just getting started. Here's the plan:",[736,1290,1292],{"id":1291},"phase-1-quick-wins-complete","Phase 1: Quick Wins ✅ Complete",[76,1294,1295,1298,1301],{},[79,1296,1297],{},"High-priority recent articles (2025-2026) with obvious issues",[79,1299,1300],{},"Typos, missing links, version disclaimers",[79,1302,1303,1306],{},[15,1304,1305],{},"Status:"," 3/56 complete",[736,1308,1310],{"id":1309},"phase-2-deep-dives-in-progress","Phase 2: Deep Dives 🔄 In Progress",[76,1312,1313,1316,1319,1322],{},[79,1314,1315],{},"Older articles from 2016-2018 that need major updates",[79,1317,1318],{},"Django version migrations (1.x → 4.x)",[79,1320,1321],{},"Code snippet verification (do commands still work?)",[79,1323,1324,1326],{},[15,1325,1305],{}," Starting now",[736,1328,1330],{"id":1329},"phase-3-link-validation-pending","Phase 3: Link Validation ⬜ Pending",[76,1332,1333,1336,1339,1342],{},[79,1334,1335],{},"Automated link checking for all external references",[79,1337,1338],{},"Archive.org snapshots for broken resources",[79,1340,1341],{},"Updating or removing dead links",[79,1343,1344,1346],{},[15,1345,1305],{}," To be built",[736,1348,1350],{"id":1349},"phase-4-image-generation-planned","Phase 4: Image Generation ⬜ Planned",[76,1352,1353,1356,1359,1362],{},[79,1354,1355],{},"Generate consistent header images using InvokeAI + Flux Krea",[79,1357,1358],{},"Create visual branding for the cleanup series",[79,1360,1361],{},"Add alt-text for accessibility",[79,1363,1364,1366],{},[15,1365,1305],{}," TODO — see below!",[651,1368],{},[56,1370,1372],{"id":1371},"the-todo-header-images","The TODO: Header Images",[11,1374,1375],{},"One thing I haven't done yet is generate custom header images for each article. This is on my next sprint:",[459,1377,1379],{"className":1076,"code":1378,"language":1078,"meta":464,"style":464},"### TODO: Image Generation System\n- [ ] Set up InvokeAI integration with Flux Krea model\n- [ ] Create image prompts based on article topics\n- [ ] Generate 56 unique header images (one per article)\n- [ ] Add alt-text for accessibility\n- [ ] Update all articles with new images\n\n**Estimated time:** 2-3 days of focused work\n",[30,1380,1381,1386,1391,1396,1401,1406,1411,1416],{"__ignoreMap":464},[151,1382,1383],{"class":469,"line":470},[151,1384,1385],{},"### TODO: Image Generation System\n",[151,1387,1388],{"class":469,"line":488},[151,1389,1390],{},"- [ ] Set up InvokeAI integration with Flux Krea model\n",[151,1392,1393],{"class":469,"line":500},[151,1394,1395],{},"- [ ] Create image prompts based on article topics\n",[151,1397,1398],{"class":469,"line":509},[151,1399,1400],{},"- [ ] Generate 56 unique header images (one per article)\n",[151,1402,1403],{"class":469,"line":517},[151,1404,1405],{},"- [ ] Add alt-text for accessibility\n",[151,1407,1408],{"class":469,"line":534},[151,1409,1410],{},"- [ ] Update all articles with new images\n",[151,1412,1414],{"class":469,"line":1413},7,[151,1415,1090],{"emptyLinePlaceholder":609},[151,1417,1419],{"class":469,"line":1418},8,[151,1420,1421],{},"**Estimated time:** 2-3 days of focused work\n",[11,1423,1424,1425,1428],{},"I'll write another article once this is done, showing the image generation pipeline and the results. For now, I'm using placeholders or leaving ",[30,1426,1427],{},"image:"," blank in front-matter.",[651,1430],{},[56,1432,1434],{"id":1433},"how-you-can-use-this-system","How You Can Use This System",[11,1436,1437],{},"If you're running a blog (or any content repository) and want to apply similar automation:",[736,1439,1441],{"id":1440},"_1-start-with-a-review-checklist","1. Start with a Review Checklist",[11,1443,1444],{},"Define what \"good\" looks like for your content:",[76,1446,1447,1450,1453,1456],{},[79,1448,1449],{},"Typos and grammar?",[79,1451,1452],{},"Broken links?",[79,1454,1455],{},"Outdated references?",[79,1457,1458],{},"Security best practices?",[736,1460,1462],{"id":1461},"_2-build-file-based-state-management","2. Build File-Based State Management",[11,1464,1465],{},"Use files instead of trying to pass context between agents:",[76,1467,1468,1474,1480],{},[79,1469,1470,1473],{},[30,1471,1472],{},"TODO.md"," — Active tasks",[79,1475,1476,1479],{},[30,1477,1478],{},"reviews/"," — Review findings per article",[79,1481,1482,1485],{},[30,1483,1484],{},"inventory.md"," — Article metadata and status",[736,1487,1489],{"id":1488},"_3-use-sequential-orchestration","3. Use Sequential Orchestration",[11,1491,1492],{},"One agent at a time, working through tasks in order. It's simpler than parallel systems and easier to debug.",[736,1494,1496],{"id":1495},"_4-automate-pr-creation","4. Automate PR Creation",[11,1498,1499,1500,1503],{},"Use the GitHub CLI (",[30,1501,1502],{},"gh",") to create branches and open pull requests:",[459,1505,1507],{"className":461,"code":1506,"language":463,"meta":464,"style":464},"git checkout -b blog-cleanup/article-name-improvements\n# ... make fixes ...\ngit push origin blog-cleanup/article-name-improvements\ngh pr create --title \"Blog cleanup: Fix in article name\" --body \"...\"\n",[30,1508,1509,1523,1529,1541],{"__ignoreMap":464},[151,1510,1511,1514,1517,1520],{"class":469,"line":470},[151,1512,1513],{"class":473},"git",[151,1515,1516],{"class":481}," checkout",[151,1518,1519],{"class":477}," -b",[151,1521,1522],{"class":481}," blog-cleanup/article-name-improvements\n",[151,1524,1525],{"class":469,"line":488},[151,1526,1528],{"class":1527},"s8-w5","# ... make fixes ...\n",[151,1530,1531,1533,1536,1539],{"class":469,"line":500},[151,1532,1513],{"class":473},[151,1534,1535],{"class":481}," push",[151,1537,1538],{"class":481}," origin",[151,1540,1522],{"class":481},[151,1542,1543,1545,1548,1551,1554,1557,1560],{"class":469,"line":509},[151,1544,1502],{"class":473},[151,1546,1547],{"class":481}," pr",[151,1549,1550],{"class":481}," create",[151,1552,1553],{"class":477}," --title",[151,1555,1556],{"class":481}," \"Blog cleanup: Fix in article name\"",[151,1558,1559],{"class":477}," --body",[151,1561,1562],{"class":481}," \"...\"\n",[736,1564,1566],{"id":1565},"_5-run-locally-when-possible","5. Run Locally When Possible",[11,1568,1569],{},"If you have the hardware, run inference locally for cost savings and privacy. DGX Spark is great for this, but even a single RTX 4090 can handle Qwen3.5-35B at reasonable speed.",[651,1571],{},[56,1573,1575],{"id":1574},"tools-technologies-used","Tools & Technologies Used",[1131,1577,1578,1591],{},[1134,1579,1580],{},[1137,1581,1582,1585,1588],{},[1140,1583,1584],{},"Component",[1140,1586,1587],{},"Technology",[1140,1589,1590],{},"Purpose",[1153,1592,1593,1606,1619,1632,1644,1657,1670],{},[1137,1594,1595,1600,1603],{},[1158,1596,1597],{},[15,1598,1599],{},"Framework",[1158,1601,1602],{},"OpenClaw",[1158,1604,1605],{},"Multi-agent orchestration system",[1137,1607,1608,1613,1616],{},[1158,1609,1610],{},[15,1611,1612],{},"Model",[1158,1614,1615],{},"Qwen3.5-35B-A3B (3B active params)",[1158,1617,1618],{},"Primary LLM for review and PR creation",[1137,1620,1621,1626,1629],{},[1158,1622,1623],{},[15,1624,1625],{},"Inference",[1158,1627,1628],{},"LM Studio",[1158,1630,1631],{},"Local model serving on DGX Spark",[1137,1633,1634,1639,1641],{},[1158,1635,1636],{},[15,1637,1638],{},"Hardware",[1158,1640,25],{},[1158,1642,1643],{},"Compact AI workstation with GB10 chip",[1137,1645,1646,1651,1654],{},[1158,1647,1648],{},[15,1649,1650],{},"Communication",[1158,1652,1653],{},"Telegram",[1158,1655,1656],{},"Access point to all agents (no context switching)",[1137,1658,1659,1664,1667],{},[1158,1660,1661],{},[15,1662,1663],{},"Version Control",[1158,1665,1666],{},"Git + GitHub",[1158,1668,1669],{},"PR workflow and change tracking",[1137,1671,1672,1677,1681],{},[1158,1673,1674],{},[15,1675,1676],{},"CLI Tool",[1158,1678,1679],{},[30,1680,1502],{},[1158,1682,1683],{},"Create branches, open pull requests from terminal",[651,1685],{},[56,1687,1689],{"id":1688},"final-thoughts-ai-as-a-collaborative-partner","Final Thoughts: AI as a Collaborative Partner",[11,1691,1692,1693,1696],{},"This blog cleanup project isn't about replacing human review — it's about ",[15,1694,1695],{},"augmenting my capabilities",". OpenClaw does the tedious work of reading through articles and finding issues. I do the final approval on GitHub, where I have full context and can make judgment calls.",[11,1698,1699],{},"The result:",[76,1701,1702,1705,1708,1711],{},[79,1703,1704],{},"Higher quality content (typos fixed, security notes added)",[79,1706,1707],{},"Better long-term value (version disclaimers keep articles relevant)",[79,1709,1710],{},"More time for writing new content instead of maintenance",[79,1712,1713],{},"A system that scales as my blog grows",[11,1715,1716,1719,1720,1723],{},[15,1717,1718],{},"This is what local AI can do:"," not just chat or generate text, but actually ",[15,1721,1722],{},"work alongside you"," on real projects. No cloud APIs. No monthly fees. Just your hardware, your data, and your goals.",[651,1725],{},[56,1727,1729],{"id":1728},"what-do-you-think","What Do You Think?",[11,1731,1732],{},"I'd love to hear feedback:",[76,1734,1735,1738,1741],{},[79,1736,1737],{},"Is this the kind of transparency helpful for readers?",[79,1739,1740],{},"Should I write more \"behind the scenes\" articles like this one?",[79,1742,1743],{},"Any suggestions for improvements to the OpenClaw system itself?",[11,1745,1746],{},"Drop a comment below or reach out on Twitter/X. The blog cleanup is ongoing — there's always room for iteration!",[651,1748],{},[11,1750,1751],{},[51,1752,1753,1754,1758],{},"This article was reviewed and improved using OpenClaw, just like all my other content now. See ",[20,1755,747],{"href":1756,"rel":1757},"https://github.com/briancaffey/briancaffey.github.io/blob/master/todo/blog-cleanup.md",[24]," to follow along with the project.",[589,1760,1761],{},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html .sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html.sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html pre.shiki code .srTi1, html code.shiki .srTi1{--shiki-default:#6F42C1;--shiki-dark:#B392F0;--shiki-sepia:#A6E22E}html pre.shiki code .sstjo, html code.shiki .sstjo{--shiki-default:#032F62;--shiki-dark:#9ECBFF;--shiki-sepia:#E6DB74}html pre.shiki code .s7F3e, html code.shiki .s7F3e{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#AE81FF}html pre.shiki code .s8-w5, html code.shiki .s8-w5{--shiki-default:#6A737D;--shiki-dark:#6A737D;--shiki-sepia:#88846F}",{"title":464,"searchDepth":488,"depth":488,"links":1763},[1764,1765,1766,1771,1777,1781,1785,1790,1791,1797,1803,1804,1811,1812,1813],{"id":631,"depth":488,"text":632},{"id":655,"depth":488,"text":656},{"id":726,"depth":488,"text":727,"children":1767},[1768,1769,1770],{"id":738,"depth":500,"text":739},{"id":766,"depth":500,"text":767},{"id":791,"depth":500,"text":792},{"id":813,"depth":488,"text":814,"children":1772},[1773,1774,1775,1776],{"id":820,"depth":500,"text":821},{"id":827,"depth":500,"text":828},{"id":870,"depth":500,"text":871},{"id":897,"depth":500,"text":898},{"id":916,"depth":488,"text":917,"children":1778},[1779,1780],{"id":927,"depth":500,"text":928},{"id":950,"depth":500,"text":951},{"id":979,"depth":488,"text":980,"children":1782},[1783,1784],{"id":990,"depth":500,"text":991},{"id":1002,"depth":500,"text":1003},{"id":1034,"depth":488,"text":1035,"children":1786},[1787,1788,1789],{"id":1041,"depth":500,"text":1042},{"id":1069,"depth":500,"text":1070},{"id":1103,"depth":500,"text":1104},{"id":1128,"depth":488,"text":1129},{"id":1215,"depth":488,"text":1216,"children":1792},[1793,1794,1795,1796],{"id":1225,"depth":500,"text":1226},{"id":1237,"depth":500,"text":1238},{"id":1252,"depth":500,"text":1253},{"id":1267,"depth":500,"text":1268},{"id":1284,"depth":488,"text":1285,"children":1798},[1799,1800,1801,1802],{"id":1291,"depth":500,"text":1292},{"id":1309,"depth":500,"text":1310},{"id":1329,"depth":500,"text":1330},{"id":1349,"depth":500,"text":1350},{"id":1371,"depth":488,"text":1372},{"id":1433,"depth":488,"text":1434,"children":1805},[1806,1807,1808,1809,1810],{"id":1440,"depth":500,"text":1441},{"id":1461,"depth":500,"text":1462},{"id":1488,"depth":500,"text":1489},{"id":1495,"depth":500,"text":1496},{"id":1565,"depth":500,"text":1566},{"id":1574,"depth":488,"text":1575},{"id":1688,"depth":488,"text":1689},{"id":1728,"depth":488,"text":1729},"2026-03-09","How I'm using OpenClaw, a multi-agent system running on DGX Spark, to systematically review and improve my personal blog articles through Telegram","/static/blog-meta/klaw.png",{},"/2026/03/09/blog-meta-about-openclaw-blog-cleanup",{"title":626,"description":1815},"2026/03/09/blog-meta-about-openclaw-blog-cleanup",[614,1822,1823,1824,1825,620],"agents","openclaw","automation","blog","xX5pZPouy4EaeGDsFHZm513gOjKaKK68cx8eupTQmU4",{"id":1828,"title":1829,"body":1830,"comments":609,"date":5679,"description":11122,"draft":602,"extension":605,"external":11123,"image":11127,"meta":11128,"navigation":609,"path":11129,"seo":11130,"stem":11131,"tags":11132,"__hash__":11136},"blog/2026/02/20/nvidia-nim-on-google-cloud-gcp.md","Deploying an AI model on GKE with NVIDIA NIM",{"type":8,"value":1831,"toc":11094},[1832,1835,1844,1850,1853,1857,1860,1964,1968,2040,2044,2050,2056,2059,2121,2250,2253,2374,2378,2503,2507,2546,2549,2654,2658,2661,2715,2721,2835,2838,2951,2955,2958,3200,3204,3294,3297,3460,3463,3844,3848,3853,3856,3861,3864,4279,4282,4287,4290,4368,4371,4532,4536,4553,4557,4560,4588,4591,4611,4615,4633,4636,4656,4660,4726,4770,4802,4809,4813,4886,4963,4967,5017,5020,5178,5181,5212,5215,5399,5406,5452,5455,5460,6167,6170,6189,6192,6421,6424,9028,9031,9068,9071,9327,9331,9334,9356,9359,9555,9558,9563,9840,9853,9858,9862,9872,9876,9882,9885,9910,9913,9998,10001,10004,10009,10013,10016,10022,10025,10046,10050,10053,10060,10067,10078,10081,10084,10089,10102,10105,10123,10142,10147,10153,10158,10166,10169,10172,10175,10182,10192,10195,10198,10201,10224,10226,10229,10246,10249,10252,10255,10258,10265,10269,10272,10277,10280,10294,10299,10308,10335,10346,10433,10443,10558,10565,10623,10628,10655,10660,10703,10708,10748,10751,10753,10756,10759,10762,10776,10778,10784,10794,10799,10805,10808,10814,10820,10826,10832,10838,10844,10847,10853,10859,10865,10868,10874,10877,10883,10886,10889,10895,10901,10904,10910,10917,10923,10926,10932,10935,10941,10944,10950,10953,10958,10964,10967,10973,10979,10982,10992,10996,11006,11016,11028,11032,11038,11044,11082,11085,11088,11091],[11,1833,1834],{},"This article is an overview of my experience doing a CodeLab from Google and NVIDIA that goes through setting up an NVIDIA Inference Microservice (NIM) on GKE. I came at this with lots of experience using NIMs locally and years of experience developing on AWS, it was an awesome introduction to awesome development tools of Google Cloud Platform!",[11,1836,1837,1838,1843],{},"This is a LONG and detailed article that is a complete walkthrough of the original CodeLab which can be found here: ",[20,1839,1842],{"href":1840,"rel":1841},"https://codelabs.developers.google.com/codelabs/nvidia-nim-google-cloud",[24],"Deploy an AI model on GKE with NVIDIA NIM",". Then I made my own adaptation of this lesson by redeploying the same application using Pulumi's Infrastructure as Code for automated spin up and tear down of the cloud resources: node pools, gpu pools, kubernetes clusters and LLM deployment.",[11,1845,1846,1847],{},"First I'll go through the tutorial, then I'll use Codex to get help defining all of the components with code! The walkthrough contains some mistakes I made and other random issues I had running through this, as well as a bunch of logs, outputs and screenshots. ",[15,1848,1849],{},"Feel free to skip to the end if you want to see how I'm planning to take my most recent vibe-coded app from my home network to the cloud with NVIDIA NIMs on GKE!",[11,1851,1852],{},"Let's go!",[56,1854,1856],{"id":1855},"cloud-shell-terminal","Cloud Shell Terminal",[11,1858,1859],{},"The first part of this tutorial requires exporting these environment variables. I'm doing all of this in Google's Cloud Terminal shell which is available in the GCP cloud console.",[459,1861,1863],{"className":461,"code":1862,"language":463,"meta":464,"style":464},"export PROJECT_ID=waywo-487618\nexport REGION=us-east4\nexport ZONE=us-east4-a\nexport CLUSTER_NAME=nim-demo\nexport NODE_POOL_MACHINE_TYPE=g2-standard-16\nexport CLUSTER_MACHINE_TYPE=e2-standard-4\nexport GPU_TYPE=nvidia-l4\nexport GPU_COUNT=1\n",[30,1864,1865,1880,1892,1904,1916,1928,1940,1952],{"__ignoreMap":464},[151,1866,1867,1871,1874,1877],{"class":469,"line":470},[151,1868,1870],{"class":1869},"sC2Qs","export",[151,1872,1873],{"class":503}," PROJECT_ID",[151,1875,1876],{"class":1869},"=",[151,1878,1879],{"class":503},"waywo-487618\n",[151,1881,1882,1884,1887,1889],{"class":469,"line":488},[151,1883,1870],{"class":1869},[151,1885,1886],{"class":503}," REGION",[151,1888,1876],{"class":1869},[151,1890,1891],{"class":503},"us-east4\n",[151,1893,1894,1896,1899,1901],{"class":469,"line":500},[151,1895,1870],{"class":1869},[151,1897,1898],{"class":503}," ZONE",[151,1900,1876],{"class":1869},[151,1902,1903],{"class":503},"us-east4-a\n",[151,1905,1906,1908,1911,1913],{"class":469,"line":509},[151,1907,1870],{"class":1869},[151,1909,1910],{"class":503}," CLUSTER_NAME",[151,1912,1876],{"class":1869},[151,1914,1915],{"class":503},"nim-demo\n",[151,1917,1918,1920,1923,1925],{"class":469,"line":517},[151,1919,1870],{"class":1869},[151,1921,1922],{"class":503}," NODE_POOL_MACHINE_TYPE",[151,1924,1876],{"class":1869},[151,1926,1927],{"class":503},"g2-standard-16\n",[151,1929,1930,1932,1935,1937],{"class":469,"line":534},[151,1931,1870],{"class":1869},[151,1933,1934],{"class":503}," CLUSTER_MACHINE_TYPE",[151,1936,1876],{"class":1869},[151,1938,1939],{"class":503},"e2-standard-4\n",[151,1941,1942,1944,1947,1949],{"class":469,"line":1413},[151,1943,1870],{"class":1869},[151,1945,1946],{"class":503}," GPU_TYPE",[151,1948,1876],{"class":1869},[151,1950,1951],{"class":503},"nvidia-l4\n",[151,1953,1954,1956,1959,1961],{"class":469,"line":1418},[151,1955,1870],{"class":1869},[151,1957,1958],{"class":503}," GPU_COUNT",[151,1960,1876],{"class":1869},[151,1962,1963],{"class":477},"1\n",[736,1965,1967],{"id":1966},"create-cluster","Create cluster",[459,1969,1971],{"className":461,"code":1970,"language":463,"meta":464,"style":464},"gcloud container clusters create ${CLUSTER_NAME} \\\n    --project=${PROJECT_ID} \\\n    --location=${ZONE} \\\n    --release-channel=rapid \\\n    --machine-type=${CLUSTER_MACHINE_TYPE} \\\n    --num-nodes=1\n",[30,1972,1973,1991,2004,2016,2023,2035],{"__ignoreMap":464},[151,1974,1975,1978,1981,1984,1986,1989],{"class":469,"line":470},[151,1976,1977],{"class":473},"gcloud",[151,1979,1980],{"class":481}," container",[151,1982,1983],{"class":481}," clusters",[151,1985,1550],{"class":481},[151,1987,1988],{"class":503}," ${CLUSTER_NAME} ",[151,1990,497],{"class":477},[151,1992,1993,1996,1999,2002],{"class":469,"line":488},[151,1994,1995],{"class":477},"    --project=${",[151,1997,1998],{"class":503},"PROJECT_ID",[151,2000,2001],{"class":477},"}",[151,2003,485],{"class":477},[151,2005,2006,2009,2012,2014],{"class":469,"line":500},[151,2007,2008],{"class":477},"    --location=${",[151,2010,2011],{"class":503},"ZONE",[151,2013,2001],{"class":477},[151,2015,485],{"class":477},[151,2017,2018,2021],{"class":469,"line":509},[151,2019,2020],{"class":477},"    --release-channel=rapid",[151,2022,485],{"class":477},[151,2024,2025,2028,2031,2033],{"class":469,"line":517},[151,2026,2027],{"class":477},"    --machine-type=${",[151,2029,2030],{"class":503},"CLUSTER_MACHINE_TYPE",[151,2032,2001],{"class":477},[151,2034,485],{"class":477},[151,2036,2037],{"class":469,"line":534},[151,2038,2039],{"class":477},"    --num-nodes=1\n",[736,2041,2043],{"id":2042},"message","Message",[459,2045,2048],{"className":2046,"code":2047,"language":997},[995],"Note: Your Pod address range (`--cluster-ipv4-cidr`) can accommodate at most 1008 node(s).\nAPI [container.googleapis.com] not enabled on project [waywo-487618]. Would you like to enable and retry (this will take a few minutes)? (y/N)?\n",[30,2049,2047],{"__ignoreMap":464},[459,2051,2054],{"className":2052,"code":2053,"language":997},[995],"ERROR: (gcloud.container.clusters.create) FAILED_PRECONDITION: Billing account for project '1068422711119' is not found. Billing must be enabled for activation of service(s) 'container.googleapis.com,artifactregistry.googleapis.com,compute.googleapis.com,containerregistry.googleapis.com,dns.googleapis.com' to proceed.\n",[30,2055,2053],{"__ignoreMap":464},[11,2057,2058],{},"Of course I forgot to enable billing! I did that and also enabled the following services:",[459,2060,2062],{"className":461,"code":2061,"language":463,"meta":464,"style":464},"gcloud services enable container.googleapis.com\ngcloud services enable artifactregistry.googleapis.com\ngcloud services enable compute.googleapis.com\ngcloud services enable containerregistry.googleapis.com\ngcloud services enable dns.googleapis.com\n",[30,2063,2064,2077,2088,2099,2110],{"__ignoreMap":464},[151,2065,2066,2068,2071,2074],{"class":469,"line":470},[151,2067,1977],{"class":473},[151,2069,2070],{"class":481}," services",[151,2072,2073],{"class":481}," enable",[151,2075,2076],{"class":481}," container.googleapis.com\n",[151,2078,2079,2081,2083,2085],{"class":469,"line":488},[151,2080,1977],{"class":473},[151,2082,2070],{"class":481},[151,2084,2073],{"class":481},[151,2086,2087],{"class":481}," artifactregistry.googleapis.com\n",[151,2089,2090,2092,2094,2096],{"class":469,"line":500},[151,2091,1977],{"class":473},[151,2093,2070],{"class":481},[151,2095,2073],{"class":481},[151,2097,2098],{"class":481}," compute.googleapis.com\n",[151,2100,2101,2103,2105,2107],{"class":469,"line":509},[151,2102,1977],{"class":473},[151,2104,2070],{"class":481},[151,2106,2073],{"class":481},[151,2108,2109],{"class":481}," containerregistry.googleapis.com\n",[151,2111,2112,2114,2116,2118],{"class":469,"line":517},[151,2113,1977],{"class":473},[151,2115,2070],{"class":481},[151,2117,2073],{"class":481},[151,2119,2120],{"class":481}," dns.googleapis.com\n",[459,2122,2124],{"className":461,"code":2123,"language":463,"meta":464,"style":464},"brian@cloudshell:~ (waywo-487618)$ gcloud config set project 1068422711119\nProject 'waywo-487618' lacks an 'environment' tag. Please create or add a tag with key 'environment' and a value like 'Production', 'Development', 'Test', or 'Staging'. Add an 'environment' tag using `gcloud resource-manager tags bindings create`. See https://cloud.google.com/resource-manager/docs/creating-managing-projects#designate_project_environments_with_tags for details.\nUpdated property [core/project].\n",[30,2125,2126,2134,2239],{"__ignoreMap":464},[151,2127,2128,2131],{"class":469,"line":470},[151,2129,2130],{"class":473},"brian@cloudshell:~",[151,2132,2133],{"class":503}," (waywo-487618)$ gcloud config set project 1068422711119\n",[151,2135,2136,2139,2142,2145,2148,2151,2154,2157,2159,2162,2165,2168,2171,2174,2177,2179,2182,2184,2187,2190,2193,2196,2199,2201,2204,2207,2209,2211,2213,2216,2219,2221,2224,2227,2230,2233,2236],{"class":469,"line":488},[151,2137,2138],{"class":473},"Project",[151,2140,2141],{"class":481}," 'waywo-487618'",[151,2143,2144],{"class":481}," lacks",[151,2146,2147],{"class":481}," an",[151,2149,2150],{"class":481}," 'environment'",[151,2152,2153],{"class":481}," tag.",[151,2155,2156],{"class":481}," Please",[151,2158,1550],{"class":481},[151,2160,2161],{"class":481}," or",[151,2163,2164],{"class":481}," add",[151,2166,2167],{"class":481}," a",[151,2169,2170],{"class":481}," tag",[151,2172,2173],{"class":481}," with",[151,2175,2176],{"class":481}," key",[151,2178,2150],{"class":481},[151,2180,2181],{"class":481}," and",[151,2183,2167],{"class":481},[151,2185,2186],{"class":481}," value",[151,2188,2189],{"class":481}," like",[151,2191,2192],{"class":481}," 'Production',",[151,2194,2195],{"class":481}," 'Development',",[151,2197,2198],{"class":481}," 'Test',",[151,2200,2161],{"class":481},[151,2202,2203],{"class":481}," 'Staging'.",[151,2205,2206],{"class":481}," Add",[151,2208,2147],{"class":481},[151,2210,2150],{"class":481},[151,2212,2170],{"class":481},[151,2214,2215],{"class":481}," using",[151,2217,2218],{"class":481}," `",[151,2220,1977],{"class":473},[151,2222,2223],{"class":481}," resource-manager tags bindings create`",[151,2225,643],{"class":2226},"sTrkL",[151,2228,2229],{"class":481}," See",[151,2231,2232],{"class":481}," https://cloud.google.com/resource-manager/docs/creating-managing-projects#designate_project_environments_with_tags",[151,2234,2235],{"class":481}," for",[151,2237,2238],{"class":481}," details.\n",[151,2240,2241,2244,2247],{"class":469,"line":500},[151,2242,2243],{"class":473},"Updated",[151,2245,2246],{"class":481}," property",[151,2248,2249],{"class":503}," [core/project].\n",[11,2251,2252],{},"I tried step one and got this:",[459,2254,2256],{"className":461,"code":2255,"language":463,"meta":464,"style":464},"brian@cloudshell:~ (1068422711119)$ gcloud resource-manager tags keys create environment \\ --parent=projects/1068422711119 \\ --description=\"environment tag\" ERROR: (gcloud.resource-manager.tags.keys.create) The value of ``core/project'' property is set to project number.To use this command, set ``--project'' flag to PROJECT ID or set ``core/project'' property to PROJECT ID. brian@cloudshell:~ (1068422711119)$\n",[30,2257,2258],{"__ignoreMap":464},[151,2259,2260,2262,2265,2268,2271,2273,2276,2279,2282,2284,2287,2290,2293,2296,2299,2302,2304,2307,2310,2313,2316,2319,2322,2325,2328,2330,2333,2336,2338,2341,2343,2346,2349,2351,2353,2355,2357,2359,2361,2363,2365,2368,2371],{"class":469,"line":470},[151,2261,2130],{"class":473},[151,2263,2264],{"class":503}," (1068422711119)$ gcloud resource-manager tags keys create environment ",[151,2266,2267],{"class":477},"\\ ",[151,2269,2270],{"class":503},"--parent",[151,2272,1876],{"class":1869},[151,2274,2275],{"class":481},"projects/1068422711119",[151,2277,2278],{"class":477}," \\ ",[151,2280,2281],{"class":503},"--description",[151,2283,1876],{"class":1869},[151,2285,2286],{"class":481},"\"environment tag\"",[151,2288,2289],{"class":473}," ERROR:",[151,2291,2292],{"class":503}," (gcloud.resource-manager.tags.keys.create) The value of ",[151,2294,2295],{"class":481},"``",[151,2297,2298],{"class":473},"core/project",[151,2300,2301],{"class":473},"''",[151,2303,2246],{"class":481},[151,2305,2306],{"class":481}," is",[151,2308,2309],{"class":481}," set",[151,2311,2312],{"class":481}," to",[151,2314,2315],{"class":481}," project",[151,2317,2318],{"class":481}," number.To",[151,2320,2321],{"class":481}," use",[151,2323,2324],{"class":481}," this",[151,2326,2327],{"class":481}," command,",[151,2329,2309],{"class":481},[151,2331,2332],{"class":481}," ``",[151,2334,2335],{"class":473},"--project",[151,2337,2301],{"class":473},[151,2339,2340],{"class":481}," flag",[151,2342,2312],{"class":481},[151,2344,2345],{"class":481}," PROJECT",[151,2347,2348],{"class":481}," ID",[151,2350,2161],{"class":481},[151,2352,2309],{"class":481},[151,2354,2332],{"class":481},[151,2356,2298],{"class":473},[151,2358,2301],{"class":473},[151,2360,2246],{"class":481},[151,2362,2312],{"class":481},[151,2364,2345],{"class":481},[151,2366,2367],{"class":481}," ID.",[151,2369,2370],{"class":481}," brian@cloudshell:~",[151,2372,2373],{"class":503}," (1068422711119)$\n",[736,2375,2377],{"id":2376},"create-tag-key","Create tag key",[459,2379,2381],{"className":461,"code":2380,"language":463,"meta":464,"style":464},"brian@cloudshell:~ (waywo-487618)$ gcloud resource-manager tags keys create environment \\\n    --parent=projects/waywo-487618 \\\n    --description=\"environment tag\"\nWaiting for TagKey [environment] to be created...done.\ncreateTime: '2026-02-20T20:38:34.308275Z'\ndescription: environment tag\netag: iCR0vsXqBiuUgnmnyGGP3Q==\nname: tagKeys/281478748509463\nnamespacedName: waywo-487618/environment\nparent: projects/1068422711119\nshortName: environment\nupdateTime: '2026-02-20T20:38:34.308275Z'\nbrian@cloudshell:~ (waywo-487618)$\n",[30,2382,2383,2392,2404,2412,2425,2433,2444,2452,2460,2469,2478,2487,2495],{"__ignoreMap":464},[151,2384,2385,2387,2390],{"class":469,"line":470},[151,2386,2130],{"class":473},[151,2388,2389],{"class":503}," (waywo-487618)$ gcloud resource-manager tags keys create environment ",[151,2391,497],{"class":477},[151,2393,2394,2397,2399,2402],{"class":469,"line":488},[151,2395,2396],{"class":503},"    --parent",[151,2398,1876],{"class":1869},[151,2400,2401],{"class":481},"projects/waywo-487618",[151,2403,485],{"class":473},[151,2405,2406,2409],{"class":469,"line":500},[151,2407,2408],{"class":477},"    --description=",[151,2410,2411],{"class":481},"\"environment tag\"\n",[151,2413,2414,2417,2419,2422],{"class":469,"line":509},[151,2415,2416],{"class":473},"Waiting",[151,2418,2235],{"class":481},[151,2420,2421],{"class":481}," TagKey",[151,2423,2424],{"class":503}," [environment] to be created...done.\n",[151,2426,2427,2430],{"class":469,"line":517},[151,2428,2429],{"class":473},"createTime:",[151,2431,2432],{"class":481}," '2026-02-20T20:38:34.308275Z'\n",[151,2434,2435,2438,2441],{"class":469,"line":534},[151,2436,2437],{"class":473},"description:",[151,2439,2440],{"class":481}," environment",[151,2442,2443],{"class":481}," tag\n",[151,2445,2446,2449],{"class":469,"line":1413},[151,2447,2448],{"class":473},"etag:",[151,2450,2451],{"class":481}," iCR0vsXqBiuUgnmnyGGP3Q==\n",[151,2453,2454,2457],{"class":469,"line":1418},[151,2455,2456],{"class":473},"name:",[151,2458,2459],{"class":481}," tagKeys/281478748509463\n",[151,2461,2463,2466],{"class":469,"line":2462},9,[151,2464,2465],{"class":473},"namespacedName:",[151,2467,2468],{"class":481}," waywo-487618/environment\n",[151,2470,2472,2475],{"class":469,"line":2471},10,[151,2473,2474],{"class":473},"parent:",[151,2476,2477],{"class":481}," projects/1068422711119\n",[151,2479,2481,2484],{"class":469,"line":2480},11,[151,2482,2483],{"class":473},"shortName:",[151,2485,2486],{"class":481}," environment\n",[151,2488,2490,2493],{"class":469,"line":2489},12,[151,2491,2492],{"class":473},"updateTime:",[151,2494,2432],{"class":481},[151,2496,2498,2500],{"class":469,"line":2497},13,[151,2499,2130],{"class":473},[151,2501,2502],{"class":503}," (waywo-487618)$\n",[736,2504,2506],{"id":2505},"create-tag-value","Create tag value",[459,2508,2510],{"className":461,"code":2509,"language":463,"meta":464,"style":464},"gcloud resource-manager tags values create development \\\n    --parent=tagKeys/281478748509463 \\\n    --description=\"Development environment\"\n",[30,2511,2512,2532,2539],{"__ignoreMap":464},[151,2513,2514,2516,2519,2522,2525,2527,2530],{"class":469,"line":470},[151,2515,1977],{"class":473},[151,2517,2518],{"class":481}," resource-manager",[151,2520,2521],{"class":481}," tags",[151,2523,2524],{"class":481}," values",[151,2526,1550],{"class":481},[151,2528,2529],{"class":481}," development",[151,2531,485],{"class":477},[151,2533,2534,2537],{"class":469,"line":488},[151,2535,2536],{"class":477},"    --parent=tagKeys/281478748509463",[151,2538,485],{"class":477},[151,2540,2541,2543],{"class":469,"line":500},[151,2542,2408],{"class":477},[151,2544,2545],{"class":481},"\"Development environment\"\n",[11,2547,2548],{},"That worked!",[459,2550,2552],{"className":461,"code":2551,"language":463,"meta":464,"style":464},"brian@cloudshell:~ (waywo-487618)$ gcloud resource-manager tags values create development \\\n    --parent=tagKeys/281478748509463 \\\n    --description=\"Development environment\"\nWaiting for TagValue [development] to be created...done.\ncreateTime: '2026-02-20T20:44:44.825712Z'\ndescription: Development environment\netag: /z9jZec+0keOOXuAsiUnWw==\nname: tagValues/281482499700539\nnamespacedName: waywo-487618/environment/development\nparent: tagKeys/281478748509463\nshortName: development\nupdateTime: '2026-02-20T20:44:44.825712Z'\nbrian@cloudshell:~ (waywo-487618)$\n",[30,2553,2554,2563,2574,2580,2592,2599,2608,2615,2622,2629,2635,2642,2648],{"__ignoreMap":464},[151,2555,2556,2558,2561],{"class":469,"line":470},[151,2557,2130],{"class":473},[151,2559,2560],{"class":503}," (waywo-487618)$ gcloud resource-manager tags values create development ",[151,2562,497],{"class":477},[151,2564,2565,2567,2569,2572],{"class":469,"line":488},[151,2566,2396],{"class":503},[151,2568,1876],{"class":1869},[151,2570,2571],{"class":481},"tagKeys/281478748509463",[151,2573,485],{"class":473},[151,2575,2576,2578],{"class":469,"line":500},[151,2577,2408],{"class":477},[151,2579,2545],{"class":481},[151,2581,2582,2584,2586,2589],{"class":469,"line":509},[151,2583,2416],{"class":473},[151,2585,2235],{"class":481},[151,2587,2588],{"class":481}," TagValue",[151,2590,2591],{"class":503}," [development] to be created...done.\n",[151,2593,2594,2596],{"class":469,"line":517},[151,2595,2429],{"class":473},[151,2597,2598],{"class":481}," '2026-02-20T20:44:44.825712Z'\n",[151,2600,2601,2603,2606],{"class":469,"line":534},[151,2602,2437],{"class":473},[151,2604,2605],{"class":481}," Development",[151,2607,2486],{"class":481},[151,2609,2610,2612],{"class":469,"line":1413},[151,2611,2448],{"class":473},[151,2613,2614],{"class":481}," /z9jZec+0keOOXuAsiUnWw==\n",[151,2616,2617,2619],{"class":469,"line":1418},[151,2618,2456],{"class":473},[151,2620,2621],{"class":481}," tagValues/281482499700539\n",[151,2623,2624,2626],{"class":469,"line":2462},[151,2625,2465],{"class":473},[151,2627,2628],{"class":481}," waywo-487618/environment/development\n",[151,2630,2631,2633],{"class":469,"line":2471},[151,2632,2474],{"class":473},[151,2634,2459],{"class":481},[151,2636,2637,2639],{"class":469,"line":2480},[151,2638,2483],{"class":473},[151,2640,2641],{"class":481}," development\n",[151,2643,2644,2646],{"class":469,"line":2489},[151,2645,2492],{"class":473},[151,2647,2598],{"class":481},[151,2649,2650,2652],{"class":469,"line":2497},[151,2651,2130],{"class":473},[151,2653,2502],{"class":503},[736,2655,2657],{"id":2656},"enable-the-services","Enable the services",[11,2659,2660],{},"Now that I have added an environment tag, I can enable the services needed for creating the",[459,2662,2663],{"className":461,"code":2061,"language":463,"meta":464,"style":464},[30,2664,2665,2675,2685,2695,2705],{"__ignoreMap":464},[151,2666,2667,2669,2671,2673],{"class":469,"line":470},[151,2668,1977],{"class":473},[151,2670,2070],{"class":481},[151,2672,2073],{"class":481},[151,2674,2076],{"class":481},[151,2676,2677,2679,2681,2683],{"class":469,"line":488},[151,2678,1977],{"class":473},[151,2680,2070],{"class":481},[151,2682,2073],{"class":481},[151,2684,2087],{"class":481},[151,2686,2687,2689,2691,2693],{"class":469,"line":500},[151,2688,1977],{"class":473},[151,2690,2070],{"class":481},[151,2692,2073],{"class":481},[151,2694,2098],{"class":481},[151,2696,2697,2699,2701,2703],{"class":469,"line":509},[151,2698,1977],{"class":473},[151,2700,2070],{"class":481},[151,2702,2073],{"class":481},[151,2704,2109],{"class":481},[151,2706,2707,2709,2711,2713],{"class":469,"line":517},[151,2708,1977],{"class":473},[151,2710,2070],{"class":481},[151,2712,2073],{"class":481},[151,2714,2120],{"class":481},[11,2716,2717],{},[2718,2719],"img",{"alt":2718,"src":2720},"/static/codelab/nvgcp_terminal_1.png",[459,2722,2724],{"className":461,"code":2723,"language":463,"meta":464,"style":464},"brian@cloudshell:~ (waywo-487618)$ gcloud container clusters create ${CLUSTER_NAME} \\\n    --project=${PROJECT_ID} \\\n    --location=${ZONE} \\\n    --release-channel=rapid \\\n    --machine-type=${CLUSTER_MACHINE_TYPE} \\\n    --num-nodes=1\nNote: Your Pod address range (`--cluster-ipv4-cidr`) can accommodate at most 1008 node(s).\nCreating cluster nim-demo in us-east4-a... Cluster is being health-checked...working.\n",[30,2725,2726,2735,2747,2757,2763,2773,2777,2807],{"__ignoreMap":464},[151,2727,2728,2730,2733],{"class":469,"line":470},[151,2729,2130],{"class":473},[151,2731,2732],{"class":503}," (waywo-487618)$ gcloud container clusters create ${CLUSTER_NAME} ",[151,2734,497],{"class":477},[151,2736,2737,2740,2742,2745],{"class":469,"line":488},[151,2738,2739],{"class":503},"    --project",[151,2741,1876],{"class":1869},[151,2743,2744],{"class":503},"${PROJECT_ID} ",[151,2746,497],{"class":473},[151,2748,2749,2751,2753,2755],{"class":469,"line":500},[151,2750,2008],{"class":477},[151,2752,2011],{"class":503},[151,2754,2001],{"class":477},[151,2756,485],{"class":477},[151,2758,2759,2761],{"class":469,"line":509},[151,2760,2020],{"class":477},[151,2762,485],{"class":477},[151,2764,2765,2767,2769,2771],{"class":469,"line":517},[151,2766,2027],{"class":477},[151,2768,2030],{"class":503},[151,2770,2001],{"class":477},[151,2772,485],{"class":477},[151,2774,2775],{"class":469,"line":534},[151,2776,2039],{"class":477},[151,2778,2779,2782,2785,2788,2791,2794,2796,2799,2802,2804],{"class":469,"line":1413},[151,2780,2781],{"class":473},"Note:",[151,2783,2784],{"class":481}," Your",[151,2786,2787],{"class":481}," Pod",[151,2789,2790],{"class":481}," address",[151,2792,2793],{"class":481}," range",[151,2795,129],{"class":503},[151,2797,2798],{"class":481},"`",[151,2800,2801],{"class":473},"--cluster-ipv4-cidr",[151,2803,2798],{"class":481},[151,2805,2806],{"class":503},") can accommodate at most 1008 node(s).\n",[151,2808,2809,2812,2815,2818,2821,2824,2827,2829,2832],{"class":469,"line":1418},[151,2810,2811],{"class":473},"Creating",[151,2813,2814],{"class":481}," cluster",[151,2816,2817],{"class":481}," nim-demo",[151,2819,2820],{"class":481}," in",[151,2822,2823],{"class":481}," us-east4-a...",[151,2825,2826],{"class":481}," Cluster",[151,2828,2306],{"class":481},[151,2830,2831],{"class":481}," being",[151,2833,2834],{"class":481}," health-checked...working.\n",[11,2836,2837],{},"Cluster is being created, cluster is being health checked, Kubernetes Control Plane is healthy, exciting!",[459,2839,2841],{"className":461,"code":2840,"language":463,"meta":464,"style":464},"brian@cloudshell:~ (waywo-487618)$ gcloud container clusters create ${CLUSTER_NAME} \\\n    --project=${PROJECT_ID} \\\n    --location=${ZONE} \\\n    --release-channel=rapid \\\n    --machine-type=${CLUSTER_MACHINE_TYPE} \\\n    --num-nodes=1\nNote: Your Pod address range (`--cluster-ipv4-cidr`) can accommodate at most 1008 node(s).\nCreating cluster nim-demo in us-east4-a... Cluster is being health-checked (Kubernetes Control Plane is healthy)...working...\n",[30,2842,2843,2851,2861,2871,2877,2887,2891,2913],{"__ignoreMap":464},[151,2844,2845,2847,2849],{"class":469,"line":470},[151,2846,2130],{"class":473},[151,2848,2732],{"class":503},[151,2850,497],{"class":477},[151,2852,2853,2855,2857,2859],{"class":469,"line":488},[151,2854,2739],{"class":503},[151,2856,1876],{"class":1869},[151,2858,2744],{"class":503},[151,2860,497],{"class":473},[151,2862,2863,2865,2867,2869],{"class":469,"line":500},[151,2864,2008],{"class":477},[151,2866,2011],{"class":503},[151,2868,2001],{"class":477},[151,2870,485],{"class":477},[151,2872,2873,2875],{"class":469,"line":509},[151,2874,2020],{"class":477},[151,2876,485],{"class":477},[151,2878,2879,2881,2883,2885],{"class":469,"line":517},[151,2880,2027],{"class":477},[151,2882,2030],{"class":503},[151,2884,2001],{"class":477},[151,2886,485],{"class":477},[151,2888,2889],{"class":469,"line":534},[151,2890,2039],{"class":477},[151,2892,2893,2895,2897,2899,2901,2903,2905,2907,2909,2911],{"class":469,"line":1413},[151,2894,2781],{"class":473},[151,2896,2784],{"class":481},[151,2898,2787],{"class":481},[151,2900,2790],{"class":481},[151,2902,2793],{"class":481},[151,2904,129],{"class":503},[151,2906,2798],{"class":481},[151,2908,2801],{"class":473},[151,2910,2798],{"class":481},[151,2912,2806],{"class":503},[151,2914,2915,2917,2919,2921,2923,2925,2927,2929,2931,2934,2937,2940,2943,2945,2948],{"class":469,"line":1418},[151,2916,2811],{"class":473},[151,2918,2814],{"class":481},[151,2920,2817],{"class":481},[151,2922,2820],{"class":481},[151,2924,2823],{"class":481},[151,2926,2826],{"class":481},[151,2928,2306],{"class":481},[151,2930,2831],{"class":481},[151,2932,2933],{"class":481}," health-checked",[151,2935,2936],{"class":503}," (Kubernetes ",[151,2938,2939],{"class":481},"Control",[151,2941,2942],{"class":481}," Plane",[151,2944,2306],{"class":481},[151,2946,2947],{"class":481}," healthy",[151,2949,2950],{"class":503},")...working...\n",[736,2952,2954],{"id":2953},"created","Created!",[11,2956,2957],{},"After a few short minutes my cluster was created!",[459,2959,2961],{"className":461,"code":2960,"language":463,"meta":464,"style":464},"brian@cloudshell:~ (waywo-487618)$ gcloud container clusters create ${CLUSTER_NAME} \\\n    --project=${PROJECT_ID} \\\n    --location=${ZONE} \\\n    --release-channel=rapid \\\n    --machine-type=${CLUSTER_MACHINE_TYPE} \\\n    --num-nodes=1\nNote: Your Pod address range (`--cluster-ipv4-cidr`) can accommodate at most 1008 node(s).\nCreating cluster nim-demo in us-east4-a... Cluster is being health-checked (Kubernetes Control Plane is healthy)...done.\nCreated [https://container.googleapis.com/v1/projects/waywo-487618/zones/us-east4-a/clusters/nim-demo].\nTo inspect the contents of your cluster, go to: https://console.cloud.google.com/kubernetes/workload_/gcloud/us-east4-a/nim-demo?project=waywo-487618\nkubeconfig entry generated for nim-demo.\nNAME: nim-demo\nLOCATION: us-east4-a\nMASTER_VERSION: 1.35.0-gke.2398000\nMASTER_IP: 34.186.97.5\nMACHINE_TYPE: e2-standard-4\nNODE_VERSION: 1.35.0-gke.2398000\nNUM_NODES: 1\nSTATUS: RUNNING\nSTACK_TYPE: IPV4\n",[30,2962,2963,2971,2981,2991,2997,3007,3011,3033,3066,3074,3106,3122,3130,3138,3147,3156,3165,3173,3182,3191],{"__ignoreMap":464},[151,2964,2965,2967,2969],{"class":469,"line":470},[151,2966,2130],{"class":473},[151,2968,2732],{"class":503},[151,2970,497],{"class":477},[151,2972,2973,2975,2977,2979],{"class":469,"line":488},[151,2974,2739],{"class":503},[151,2976,1876],{"class":1869},[151,2978,2744],{"class":503},[151,2980,497],{"class":473},[151,2982,2983,2985,2987,2989],{"class":469,"line":500},[151,2984,2008],{"class":477},[151,2986,2011],{"class":503},[151,2988,2001],{"class":477},[151,2990,485],{"class":477},[151,2992,2993,2995],{"class":469,"line":509},[151,2994,2020],{"class":477},[151,2996,485],{"class":477},[151,2998,2999,3001,3003,3005],{"class":469,"line":517},[151,3000,2027],{"class":477},[151,3002,2030],{"class":503},[151,3004,2001],{"class":477},[151,3006,485],{"class":477},[151,3008,3009],{"class":469,"line":534},[151,3010,2039],{"class":477},[151,3012,3013,3015,3017,3019,3021,3023,3025,3027,3029,3031],{"class":469,"line":1413},[151,3014,2781],{"class":473},[151,3016,2784],{"class":481},[151,3018,2787],{"class":481},[151,3020,2790],{"class":481},[151,3022,2793],{"class":481},[151,3024,129],{"class":503},[151,3026,2798],{"class":481},[151,3028,2801],{"class":473},[151,3030,2798],{"class":481},[151,3032,2806],{"class":503},[151,3034,3035,3037,3039,3041,3043,3045,3047,3049,3051,3053,3055,3057,3059,3061,3063],{"class":469,"line":1418},[151,3036,2811],{"class":473},[151,3038,2814],{"class":481},[151,3040,2817],{"class":481},[151,3042,2820],{"class":481},[151,3044,2823],{"class":481},[151,3046,2826],{"class":481},[151,3048,2306],{"class":481},[151,3050,2831],{"class":481},[151,3052,2933],{"class":481},[151,3054,2936],{"class":503},[151,3056,2939],{"class":481},[151,3058,2942],{"class":481},[151,3060,2306],{"class":481},[151,3062,2947],{"class":481},[151,3064,3065],{"class":503},")...done.\n",[151,3067,3068,3071],{"class":469,"line":2462},[151,3069,3070],{"class":473},"Created",[151,3072,3073],{"class":503}," [https://container.googleapis.com/v1/projects/waywo-487618/zones/us-east4-a/clusters/nim-demo].\n",[151,3075,3076,3079,3082,3085,3088,3091,3094,3097,3100,3103],{"class":469,"line":2471},[151,3077,3078],{"class":473},"To",[151,3080,3081],{"class":481}," inspect",[151,3083,3084],{"class":481}," the",[151,3086,3087],{"class":481}," contents",[151,3089,3090],{"class":481}," of",[151,3092,3093],{"class":481}," your",[151,3095,3096],{"class":481}," cluster,",[151,3098,3099],{"class":481}," go",[151,3101,3102],{"class":481}," to:",[151,3104,3105],{"class":481}," https://console.cloud.google.com/kubernetes/workload_/gcloud/us-east4-a/nim-demo?project=waywo-487618\n",[151,3107,3108,3111,3114,3117,3119],{"class":469,"line":2480},[151,3109,3110],{"class":473},"kubeconfig",[151,3112,3113],{"class":481}," entry",[151,3115,3116],{"class":481}," generated",[151,3118,2235],{"class":481},[151,3120,3121],{"class":481}," nim-demo.\n",[151,3123,3124,3127],{"class":469,"line":2489},[151,3125,3126],{"class":473},"NAME:",[151,3128,3129],{"class":481}," nim-demo\n",[151,3131,3132,3135],{"class":469,"line":2497},[151,3133,3134],{"class":473},"LOCATION:",[151,3136,3137],{"class":481}," us-east4-a\n",[151,3139,3141,3144],{"class":469,"line":3140},14,[151,3142,3143],{"class":473},"MASTER_VERSION:",[151,3145,3146],{"class":481}," 1.35.0-gke.2398000\n",[151,3148,3150,3153],{"class":469,"line":3149},15,[151,3151,3152],{"class":473},"MASTER_IP:",[151,3154,3155],{"class":477}," 34.186.97.5\n",[151,3157,3159,3162],{"class":469,"line":3158},16,[151,3160,3161],{"class":473},"MACHINE_TYPE:",[151,3163,3164],{"class":481}," e2-standard-4\n",[151,3166,3168,3171],{"class":469,"line":3167},17,[151,3169,3170],{"class":473},"NODE_VERSION:",[151,3172,3146],{"class":481},[151,3174,3176,3179],{"class":469,"line":3175},18,[151,3177,3178],{"class":473},"NUM_NODES:",[151,3180,3181],{"class":477}," 1\n",[151,3183,3185,3188],{"class":469,"line":3184},19,[151,3186,3187],{"class":473},"STATUS:",[151,3189,3190],{"class":481}," RUNNING\n",[151,3192,3194,3197],{"class":469,"line":3193},20,[151,3195,3196],{"class":473},"STACK_TYPE:",[151,3198,3199],{"class":481}," IPV4\n",[736,3201,3203],{"id":3202},"step-4-create-the-gpu-pool","Step 4: Create the GPU Pool",[459,3205,3207],{"className":461,"code":3206,"language":463,"meta":464,"style":464},"gcloud container node-pools create gpupool \\\n    --accelerator type=${GPU_TYPE},count=${GPU_COUNT},gpu-driver-version=latest \\\n    --project=${PROJECT_ID} \\\n    --location=${ZONE} \\\n    --cluster=${CLUSTER_NAME} \\\n    --machine-type=${NODE_POOL_MACHINE_TYPE} \\\n    --num-nodes=1\n",[30,3208,3209,3225,3247,3257,3267,3279,3290],{"__ignoreMap":464},[151,3210,3211,3213,3215,3218,3220,3223],{"class":469,"line":470},[151,3212,1977],{"class":473},[151,3214,1980],{"class":481},[151,3216,3217],{"class":481}," node-pools",[151,3219,1550],{"class":481},[151,3221,3222],{"class":481}," gpupool",[151,3224,485],{"class":477},[151,3226,3227,3230,3233,3236,3239,3242,3245],{"class":469,"line":488},[151,3228,3229],{"class":477},"    --accelerator",[151,3231,3232],{"class":481}," type=",[151,3234,3235],{"class":503},"${GPU_TYPE}",[151,3237,3238],{"class":481},",count=",[151,3240,3241],{"class":503},"${GPU_COUNT}",[151,3243,3244],{"class":481},",gpu-driver-version=latest",[151,3246,485],{"class":477},[151,3248,3249,3251,3253,3255],{"class":469,"line":500},[151,3250,1995],{"class":477},[151,3252,1998],{"class":503},[151,3254,2001],{"class":477},[151,3256,485],{"class":477},[151,3258,3259,3261,3263,3265],{"class":469,"line":509},[151,3260,2008],{"class":477},[151,3262,2011],{"class":503},[151,3264,2001],{"class":477},[151,3266,485],{"class":477},[151,3268,3269,3272,3275,3277],{"class":469,"line":517},[151,3270,3271],{"class":477},"    --cluster=${",[151,3273,3274],{"class":503},"CLUSTER_NAME",[151,3276,2001],{"class":477},[151,3278,485],{"class":477},[151,3280,3281,3283,3286,3288],{"class":469,"line":534},[151,3282,2027],{"class":477},[151,3284,3285],{"class":503},"NODE_POOL_MACHINE_TYPE",[151,3287,2001],{"class":477},[151,3289,485],{"class":477},[151,3291,3292],{"class":469,"line":1413},[151,3293,2039],{"class":477},[11,3295,3296],{},"Here’s what it says:",[459,3298,3300],{"className":461,"code":3299,"language":463,"meta":464,"style":464},"brian@cloudshell:~ (waywo-487618)$ gcloud container node-pools create gpupool \\\n    --accelerator type=${GPU_TYPE},count=${GPU_COUNT},gpu-driver-version=latest \\\n    --project=${PROJECT_ID} \\\n    --location=${ZONE} \\\n    --cluster=${CLUSTER_NAME} \\\n    --machine-type=${NODE_POOL_MACHINE_TYPE} \\\n    --num-nodes=1\nNote: Machines with GPUs have certain limitations which may affect your workflow. Learn more at https://cloud.google.com/kubernetes-engine/docs/how-to/gpus\nNote: Starting in GKE 1.30.1-gke.115600, if you don't specify a driver version, GKE installs the default GPU driver for your node's GKE version.\nCreating node pool gpupool...working.\n",[30,3301,3302,3311,3327,3337,3347,3357,3367,3371,3418,3447],{"__ignoreMap":464},[151,3303,3304,3306,3309],{"class":469,"line":470},[151,3305,2130],{"class":473},[151,3307,3308],{"class":503}," (waywo-487618)$ gcloud container node-pools create gpupool ",[151,3310,497],{"class":477},[151,3312,3313,3315,3317,3319,3321,3323,3325],{"class":469,"line":488},[151,3314,3229],{"class":473},[151,3316,3232],{"class":481},[151,3318,3235],{"class":503},[151,3320,3238],{"class":481},[151,3322,3241],{"class":503},[151,3324,3244],{"class":481},[151,3326,485],{"class":477},[151,3328,3329,3331,3333,3335],{"class":469,"line":500},[151,3330,1995],{"class":477},[151,3332,1998],{"class":503},[151,3334,2001],{"class":477},[151,3336,485],{"class":477},[151,3338,3339,3341,3343,3345],{"class":469,"line":509},[151,3340,2008],{"class":477},[151,3342,2011],{"class":503},[151,3344,2001],{"class":477},[151,3346,485],{"class":477},[151,3348,3349,3351,3353,3355],{"class":469,"line":517},[151,3350,3271],{"class":477},[151,3352,3274],{"class":503},[151,3354,2001],{"class":477},[151,3356,485],{"class":477},[151,3358,3359,3361,3363,3365],{"class":469,"line":534},[151,3360,2027],{"class":477},[151,3362,3285],{"class":503},[151,3364,2001],{"class":477},[151,3366,485],{"class":477},[151,3368,3369],{"class":469,"line":1413},[151,3370,2039],{"class":477},[151,3372,3373,3375,3378,3380,3383,3386,3389,3392,3395,3398,3401,3403,3406,3409,3412,3415],{"class":469,"line":1418},[151,3374,2781],{"class":473},[151,3376,3377],{"class":481}," Machines",[151,3379,2173],{"class":481},[151,3381,3382],{"class":481}," GPUs",[151,3384,3385],{"class":481}," have",[151,3387,3388],{"class":481}," certain",[151,3390,3391],{"class":481}," limitations",[151,3393,3394],{"class":481}," which",[151,3396,3397],{"class":481}," may",[151,3399,3400],{"class":481}," affect",[151,3402,3093],{"class":481},[151,3404,3405],{"class":481}," workflow.",[151,3407,3408],{"class":481}," Learn",[151,3410,3411],{"class":481}," more",[151,3413,3414],{"class":481}," at",[151,3416,3417],{"class":481}," https://cloud.google.com/kubernetes-engine/docs/how-to/gpus\n",[151,3419,3420,3422,3425,3427,3430,3433,3436,3439,3442,3444],{"class":469,"line":2462},[151,3421,2781],{"class":473},[151,3423,3424],{"class":481}," Starting",[151,3426,2820],{"class":481},[151,3428,3429],{"class":481}," GKE",[151,3431,3432],{"class":481}," 1.30.1-gke.115600,",[151,3434,3435],{"class":481}," if",[151,3437,3438],{"class":481}," you",[151,3440,3441],{"class":481}," don't specify a driver version, GKE installs the default GPU driver for your node's",[151,3443,3429],{"class":481},[151,3445,3446],{"class":481}," version.\n",[151,3448,3449,3451,3454,3457],{"class":469,"line":2471},[151,3450,2811],{"class":473},[151,3452,3453],{"class":481}," node",[151,3455,3456],{"class":481}," pool",[151,3458,3459],{"class":481}," gpupool...working.\n",[11,3461,3462],{},"After a few minutes I saw the following:",[459,3464,3466],{"className":461,"code":3465,"language":463,"meta":464,"style":464},"brian@cloudshell:~ (waywo-487618)$ gcloud container node-pools create gpupool \\\n    --accelerator type=${GPU_TYPE},count=${GPU_COUNT},gpu-driver-version=latest \\\n    --project=${PROJECT_ID} \\\n    --location=${ZONE} \\\n    --cluster=${CLUSTER_NAME} \\\n    --machine-type=${NODE_POOL_MACHINE_TYPE} \\\n    --num-nodes=1\nNote: Machines with GPUs have certain limitations which may affect your workflow. Learn more at https://cloud.google.com/kubernetes-engine/docs/how-to/gpus\nNote: Starting in GKE 1.30.1-gke.115600, if you don't specify a driver version, GKE installs the default GPU driver for your node's GKE version.\nCreating node pool gpupool...done.\nERROR: (gcloud.container.node-pools.create) Operation [\u003COperation\n clusterConditions: [\u003CStatusCondition\n canonicalCode: CanonicalCodeValueValuesEnum(RESOURCE_EXHAUSTED, 9)\n code: CodeValueValuesEnum(GCE_QUOTA_EXCEEDED, 3)\n message: \"Instance 'gke-nim-demo-gpupool-1490dbfa-qtlc' creation failed: Quota 'GPUS_ALL_REGIONS' exceeded.  Limit: 0.0 globally.\">, \u003CStatusCondition\n canonicalCode: CanonicalCodeValueValuesEnum(RESOURCE_EXHAUSTED, 9)\n message: \"Insufficient quota to satisfy the request: Not all instances running in IGM after 43.560778184s. Expected 1, running 0, transitioning 1. Current errors: [GCE_QUOTA_EXCEEDED]: Instance 'gke-nim-demo-gpupool-1490dbfa-qtlc' creation failed: Quota 'GPUS_ALL_REGIONS' exceeded.  Limit: 0.0 globally.\">]\n detail: \"Insufficient quota to satisfy the request: Not all instances running in IGM after 43.560778184s. Expected 1, running 0, transitioning 1. Current errors: [GCE_QUOTA_EXCEEDED]: Instance 'gke-nim-demo-gpupool-1490dbfa-qtlc' creation failed: Quota 'GPUS_ALL_REGIONS' exceeded.  Limit: 0.0 globally.\"\n endTime: '2026-02-20T20:55:56.546684253Z'\n error: \u003CStatus\n code: 8\n details: []\n message: \"Instance 'gke-nim-demo-gpupool-1490dbfa-qtlc' creation failed: Quota 'GPUS_ALL_REGIONS' exceeded.  Limit: 0.0 globally.\">\n name: 'operation-1771620906928-11ea3375-fd21-4184-85c3-9aa57fcd80c2'\n nodepoolConditions: []\n operationType: OperationTypeValueValuesEnum(CREATE_NODE_POOL, 7)\n selfLink: 'https://container.googleapis.com/v1/projects/1068422711119/zones/us-east4-a/operations/operation-1771620906928-11ea3375-fd21-4184-85c3-9aa57fcd80c2'\n startTime: '2026-02-20T20:55:06.928676262Z'\n status: StatusValueValuesEnum(DONE, 3)\n statusMessage: \"Insufficient quota to satisfy the request: Not all instances running in IGM after 43.560778184s. Expected 1, running 0, transitioning 1. Current errors: [GCE_QUOTA_EXCEEDED]: Instance 'gke-nim-demo-gpupool-1490dbfa-qtlc' creation failed: Quota 'GPUS_ALL_REGIONS' exceeded.  Limit: 0.0 globally.\"\n targetLink: 'https://container.googleapis.com/v1/projects/1068422711119/zones/us-east4-a/clusters/nim-demo/nodePools/gpupool'\n zone: 'us-east4-a'>] finished with error: Insufficient quota to satisfy the request: Not all instances running in IGM after 43.560778184s. Expected 1, running 0, transitioning 1. Current errors: [GCE_QUOTA_EXCEEDED]: Instance 'gke-nim-demo-gpupool-1490dbfa-qtlc' creation failed: Quota 'GPUS_ALL_REGIONS' exceeded.  Limit: 0.0 globally.\n",[30,3467,3468,3476,3492,3502,3512,3522,3532,3536,3570,3592,3603,3617,3627,3641,3653,3670,3680,3692,3700,3708,3718,3727,3733,3743,3752,3758,3771,3780,3789,3801,3809,3818],{"__ignoreMap":464},[151,3469,3470,3472,3474],{"class":469,"line":470},[151,3471,2130],{"class":473},[151,3473,3308],{"class":503},[151,3475,497],{"class":477},[151,3477,3478,3480,3482,3484,3486,3488,3490],{"class":469,"line":488},[151,3479,3229],{"class":473},[151,3481,3232],{"class":481},[151,3483,3235],{"class":503},[151,3485,3238],{"class":481},[151,3487,3241],{"class":503},[151,3489,3244],{"class":481},[151,3491,485],{"class":477},[151,3493,3494,3496,3498,3500],{"class":469,"line":500},[151,3495,1995],{"class":477},[151,3497,1998],{"class":503},[151,3499,2001],{"class":477},[151,3501,485],{"class":477},[151,3503,3504,3506,3508,3510],{"class":469,"line":509},[151,3505,2008],{"class":477},[151,3507,2011],{"class":503},[151,3509,2001],{"class":477},[151,3511,485],{"class":477},[151,3513,3514,3516,3518,3520],{"class":469,"line":517},[151,3515,3271],{"class":477},[151,3517,3274],{"class":503},[151,3519,2001],{"class":477},[151,3521,485],{"class":477},[151,3523,3524,3526,3528,3530],{"class":469,"line":534},[151,3525,2027],{"class":477},[151,3527,3285],{"class":503},[151,3529,2001],{"class":477},[151,3531,485],{"class":477},[151,3533,3534],{"class":469,"line":1413},[151,3535,2039],{"class":477},[151,3537,3538,3540,3542,3544,3546,3548,3550,3552,3554,3556,3558,3560,3562,3564,3566,3568],{"class":469,"line":1418},[151,3539,2781],{"class":473},[151,3541,3377],{"class":481},[151,3543,2173],{"class":481},[151,3545,3382],{"class":481},[151,3547,3385],{"class":481},[151,3549,3388],{"class":481},[151,3551,3391],{"class":481},[151,3553,3394],{"class":481},[151,3555,3397],{"class":481},[151,3557,3400],{"class":481},[151,3559,3093],{"class":481},[151,3561,3405],{"class":481},[151,3563,3408],{"class":481},[151,3565,3411],{"class":481},[151,3567,3414],{"class":481},[151,3569,3417],{"class":481},[151,3571,3572,3574,3576,3578,3580,3582,3584,3586,3588,3590],{"class":469,"line":2462},[151,3573,2781],{"class":473},[151,3575,3424],{"class":481},[151,3577,2820],{"class":481},[151,3579,3429],{"class":481},[151,3581,3432],{"class":481},[151,3583,3435],{"class":481},[151,3585,3438],{"class":481},[151,3587,3441],{"class":481},[151,3589,3429],{"class":481},[151,3591,3446],{"class":481},[151,3593,3594,3596,3598,3600],{"class":469,"line":2471},[151,3595,2811],{"class":473},[151,3597,3453],{"class":481},[151,3599,3456],{"class":481},[151,3601,3602],{"class":481}," gpupool...done.\n",[151,3604,3605,3608,3611,3614],{"class":469,"line":2480},[151,3606,3607],{"class":473},"ERROR:",[151,3609,3610],{"class":503}," (gcloud.container.node-pools.create) Operation [",[151,3612,3613],{"class":1869},"\u003C",[151,3615,3616],{"class":503},"Operation\n",[151,3618,3619,3622,3624],{"class":469,"line":2489},[151,3620,3621],{"class":503}," clusterConditions: [",[151,3623,3613],{"class":1869},[151,3625,3626],{"class":503},"StatusCondition\n",[151,3628,3629,3632,3635,3638],{"class":469,"line":2497},[151,3630,3631],{"class":503}," canonicalCode: CanonicalCodeValueValuesEnum(RESOURCE_EXHAUSTED",[151,3633,3634],{"class":1869},",",[151,3636,3637],{"class":477}," 9",[151,3639,3640],{"class":503},")\n",[151,3642,3643,3646,3648,3651],{"class":469,"line":3140},[151,3644,3645],{"class":503}," code: CodeValueValuesEnum(GCE_QUOTA_EXCEEDED",[151,3647,3634],{"class":1869},[151,3649,3650],{"class":477}," 3",[151,3652,3640],{"class":503},[151,3654,3655,3658,3661,3664,3666,3668],{"class":469,"line":3149},[151,3656,3657],{"class":503}," message: ",[151,3659,3660],{"class":481},"\"Instance 'gke-nim-demo-gpupool-1490dbfa-qtlc' creation failed: Quota 'GPUS_ALL_REGIONS' exceeded.  Limit: 0.0 globally.\"",[151,3662,3663],{"class":1869},">",[151,3665,106],{"class":503},[151,3667,3613],{"class":1869},[151,3669,3626],{"class":503},[151,3671,3672,3674,3676,3678],{"class":469,"line":3158},[151,3673,3631],{"class":503},[151,3675,3634],{"class":1869},[151,3677,3637],{"class":477},[151,3679,3640],{"class":503},[151,3681,3682,3684,3687,3689],{"class":469,"line":3167},[151,3683,3657],{"class":503},[151,3685,3686],{"class":481},"\"Insufficient quota to satisfy the request: Not all instances running in IGM after 43.560778184s. Expected 1, running 0, transitioning 1. Current errors: [GCE_QUOTA_EXCEEDED]: Instance 'gke-nim-demo-gpupool-1490dbfa-qtlc' creation failed: Quota 'GPUS_ALL_REGIONS' exceeded.  Limit: 0.0 globally.\"",[151,3688,3663],{"class":1869},[151,3690,3691],{"class":503},"]\n",[151,3693,3694,3697],{"class":469,"line":3175},[151,3695,3696],{"class":503}," detail: ",[151,3698,3699],{"class":481},"\"Insufficient quota to satisfy the request: Not all instances running in IGM after 43.560778184s. Expected 1, running 0, transitioning 1. Current errors: [GCE_QUOTA_EXCEEDED]: Instance 'gke-nim-demo-gpupool-1490dbfa-qtlc' creation failed: Quota 'GPUS_ALL_REGIONS' exceeded.  Limit: 0.0 globally.\"\n",[151,3701,3702,3705],{"class":469,"line":3184},[151,3703,3704],{"class":503}," endTime: ",[151,3706,3707],{"class":481},"'2026-02-20T20:55:56.546684253Z'\n",[151,3709,3710,3713,3715],{"class":469,"line":3193},[151,3711,3712],{"class":503}," error: ",[151,3714,3613],{"class":1869},[151,3716,3717],{"class":503},"Status\n",[151,3719,3721,3724],{"class":469,"line":3720},21,[151,3722,3723],{"class":503}," code: ",[151,3725,3726],{"class":477},"8\n",[151,3728,3730],{"class":469,"line":3729},22,[151,3731,3732],{"class":503}," details: []\n",[151,3734,3736,3738,3740],{"class":469,"line":3735},23,[151,3737,3657],{"class":503},[151,3739,3660],{"class":481},[151,3741,3742],{"class":1869},">\n",[151,3744,3746,3749],{"class":469,"line":3745},24,[151,3747,3748],{"class":503}," name: ",[151,3750,3751],{"class":481},"'operation-1771620906928-11ea3375-fd21-4184-85c3-9aa57fcd80c2'\n",[151,3753,3755],{"class":469,"line":3754},25,[151,3756,3757],{"class":503}," nodepoolConditions: []\n",[151,3759,3761,3764,3766,3769],{"class":469,"line":3760},26,[151,3762,3763],{"class":503}," operationType: OperationTypeValueValuesEnum(CREATE_NODE_POOL",[151,3765,3634],{"class":1869},[151,3767,3768],{"class":477}," 7",[151,3770,3640],{"class":503},[151,3772,3774,3777],{"class":469,"line":3773},27,[151,3775,3776],{"class":503}," selfLink: ",[151,3778,3779],{"class":481},"'https://container.googleapis.com/v1/projects/1068422711119/zones/us-east4-a/operations/operation-1771620906928-11ea3375-fd21-4184-85c3-9aa57fcd80c2'\n",[151,3781,3783,3786],{"class":469,"line":3782},28,[151,3784,3785],{"class":503}," startTime: ",[151,3787,3788],{"class":481},"'2026-02-20T20:55:06.928676262Z'\n",[151,3790,3792,3795,3797,3799],{"class":469,"line":3791},29,[151,3793,3794],{"class":503}," status: StatusValueValuesEnum(DONE",[151,3796,3634],{"class":1869},[151,3798,3650],{"class":477},[151,3800,3640],{"class":503},[151,3802,3804,3807],{"class":469,"line":3803},30,[151,3805,3806],{"class":503}," statusMessage: ",[151,3808,3699],{"class":481},[151,3810,3812,3815],{"class":469,"line":3811},31,[151,3813,3814],{"class":503}," targetLink: ",[151,3816,3817],{"class":481},"'https://container.googleapis.com/v1/projects/1068422711119/zones/us-east4-a/clusters/nim-demo/nodePools/gpupool'\n",[151,3819,3821,3824,3827,3829,3832,3835,3838,3841],{"class":469,"line":3820},32,[151,3822,3823],{"class":503}," zone: ",[151,3825,3826],{"class":481},"'us-east4-a'",[151,3828,3663],{"class":1869},[151,3830,3831],{"class":503},"] finished with error: Insufficient quota to satisfy the request: Not all instances running in IGM after 43.560778184s. Expected 1, running 0, transitioning 1. Current errors: [GCE_QUOTA_EXCEEDED]: Instance ",[151,3833,3834],{"class":481},"'gke-nim-demo-gpupool-1490dbfa-qtlc'",[151,3836,3837],{"class":503}," creation failed: Quota ",[151,3839,3840],{"class":481},"'GPUS_ALL_REGIONS'",[151,3842,3843],{"class":503}," exceeded.  Limit: 0.0 globally.\n",[736,3845,3847],{"id":3846},"cloud-assist-to-the-rescue","Cloud Assist to the Rescue!",[11,3849,3850],{},[2718,3851],{"alt":2718,"src":3852},"/static/codelab/nvgcp_cloud_assist.png",[11,3854,3855],{},"OK, so I need to increase my GPU quota!",[11,3857,3858],{},[2718,3859],{"alt":2718,"src":3860},"/static/codelab/nvgcp_cloud_assist_2.png",[11,3862,3863],{},"I searched for GPUS_ALL_REGIONS and requested a quote increase for 1 GPU across all regions, then a few moments later got this nice email:",[459,3865,3867],{"className":461,"code":3866,"language":463,"meta":464,"style":464},"Hello,\n\nYour quota request for waywo-487618 has been approved and your project quota has been adjusted according to the following requested limits:\n\n+------------------+------------+--------+-----------------+----------------+\n| NAME             | DIMENSIONS | REGION | REQUESTED LIMIT | APPROVED LIMIT |\n+------------------+------------+--------+-----------------+----------------+\n| GPUS_ALL_REGIONS |            | GLOBAL |               1 |              1 |\n+------------------+------------+--------+-----------------+----------------+\n\nAfter approval, Quotas can take up to 15 min to be fully visible in the Cloud Console and available to you.\n\nTo verify, please navigate to\nhttps://console.cloud.google.com/iam-admin/quotas?project=waywo-487618.\n\nIf you find actual limits are greater than expected, this is normal if previous requests were approved with higher limits. Otherwise, please let us know if approved changes are not reflected in your project.\n\nIf you want to increase your quota further, please file a new request.\n\nBest regards and happy computing!\n\nSincerely,\nGoogle Cloud Support\n",[30,3868,3869,3874,3878,3934,3938,3943,3980,3984,4011,4015,4019,4078,4082,4098,4106,4110,4202,4206,4240,4244,4260,4264,4269],{"__ignoreMap":464},[151,3870,3871],{"class":469,"line":470},[151,3872,3873],{"class":473},"Hello,\n",[151,3875,3876],{"class":469,"line":488},[151,3877,1090],{"emptyLinePlaceholder":609},[151,3879,3880,3883,3886,3889,3891,3894,3897,3900,3903,3905,3907,3909,3911,3913,3915,3918,3921,3923,3925,3928,3931],{"class":469,"line":500},[151,3881,3882],{"class":473},"Your",[151,3884,3885],{"class":481}," quota",[151,3887,3888],{"class":481}," request",[151,3890,2235],{"class":481},[151,3892,3893],{"class":481}," waywo-487618",[151,3895,3896],{"class":481}," has",[151,3898,3899],{"class":481}," been",[151,3901,3902],{"class":481}," approved",[151,3904,2181],{"class":481},[151,3906,3093],{"class":481},[151,3908,2315],{"class":481},[151,3910,3885],{"class":481},[151,3912,3896],{"class":481},[151,3914,3899],{"class":481},[151,3916,3917],{"class":481}," adjusted",[151,3919,3920],{"class":481}," according",[151,3922,2312],{"class":481},[151,3924,3084],{"class":481},[151,3926,3927],{"class":481}," following",[151,3929,3930],{"class":481}," requested",[151,3932,3933],{"class":481}," limits:\n",[151,3935,3936],{"class":469,"line":509},[151,3937,1090],{"emptyLinePlaceholder":609},[151,3939,3940],{"class":469,"line":517},[151,3941,3942],{"class":473},"+------------------+------------+--------+-----------------+----------------+\n",[151,3944,3945,3948,3951,3954,3957,3960,3962,3964,3967,3970,3972,3975,3977],{"class":469,"line":534},[151,3946,3947],{"class":1869},"|",[151,3949,3950],{"class":473}," NAME",[151,3952,3953],{"class":1869},"             |",[151,3955,3956],{"class":473}," DIMENSIONS",[151,3958,3959],{"class":1869}," |",[151,3961,1886],{"class":473},[151,3963,3959],{"class":1869},[151,3965,3966],{"class":473}," REQUESTED",[151,3968,3969],{"class":481}," LIMIT",[151,3971,3959],{"class":1869},[151,3973,3974],{"class":473}," APPROVED",[151,3976,3969],{"class":481},[151,3978,3979],{"class":1869}," |\n",[151,3981,3982],{"class":469,"line":1413},[151,3983,3942],{"class":473},[151,3985,3986,3988,3991,3993,3996,3999,4001,4004,4006,4009],{"class":469,"line":1418},[151,3987,3947],{"class":1869},[151,3989,3990],{"class":473}," GPUS_ALL_REGIONS",[151,3992,3959],{"class":1869},[151,3994,3995],{"class":1869},"            |",[151,3997,3998],{"class":473}," GLOBAL",[151,4000,3959],{"class":1869},[151,4002,4003],{"class":473},"               1",[151,4005,3959],{"class":1869},[151,4007,4008],{"class":473},"              1",[151,4010,3979],{"class":1869},[151,4012,4013],{"class":469,"line":2462},[151,4014,3942],{"class":473},[151,4016,4017],{"class":469,"line":2471},[151,4018,1090],{"emptyLinePlaceholder":609},[151,4020,4021,4024,4027,4030,4033,4036,4039,4041,4044,4047,4049,4052,4055,4058,4060,4062,4065,4068,4070,4073,4075],{"class":469,"line":2480},[151,4022,4023],{"class":473},"After",[151,4025,4026],{"class":481}," approval,",[151,4028,4029],{"class":481}," Quotas",[151,4031,4032],{"class":481}," can",[151,4034,4035],{"class":481}," take",[151,4037,4038],{"class":481}," up",[151,4040,2312],{"class":481},[151,4042,4043],{"class":477}," 15",[151,4045,4046],{"class":481}," min",[151,4048,2312],{"class":481},[151,4050,4051],{"class":481}," be",[151,4053,4054],{"class":481}," fully",[151,4056,4057],{"class":481}," visible",[151,4059,2820],{"class":481},[151,4061,3084],{"class":481},[151,4063,4064],{"class":481}," Cloud",[151,4066,4067],{"class":481}," Console",[151,4069,2181],{"class":481},[151,4071,4072],{"class":481}," available",[151,4074,2312],{"class":481},[151,4076,4077],{"class":481}," you.\n",[151,4079,4080],{"class":469,"line":2489},[151,4081,1090],{"emptyLinePlaceholder":609},[151,4083,4084,4086,4089,4092,4095],{"class":469,"line":2497},[151,4085,3078],{"class":473},[151,4087,4088],{"class":481}," verify,",[151,4090,4091],{"class":481}," please",[151,4093,4094],{"class":481}," navigate",[151,4096,4097],{"class":481}," to\n",[151,4099,4100,4103],{"class":469,"line":3140},[151,4101,4102],{"class":473},"https://console.cloud.google.com/iam-admin/quotas?project",[151,4104,4105],{"class":481},"=waywo-487618.\n",[151,4107,4108],{"class":469,"line":3149},[151,4109,1090],{"emptyLinePlaceholder":609},[151,4111,4112,4115,4117,4120,4123,4126,4129,4132,4135,4138,4140,4142,4145,4147,4150,4153,4156,4158,4160,4163,4166,4169,4171,4174,4177,4180,4182,4184,4187,4189,4192,4195,4197,4199],{"class":469,"line":3158},[151,4113,4114],{"class":473},"If",[151,4116,3438],{"class":481},[151,4118,4119],{"class":481}," find",[151,4121,4122],{"class":481}," actual",[151,4124,4125],{"class":481}," limits",[151,4127,4128],{"class":481}," are",[151,4130,4131],{"class":481}," greater",[151,4133,4134],{"class":481}," than",[151,4136,4137],{"class":481}," expected,",[151,4139,2324],{"class":481},[151,4141,2306],{"class":481},[151,4143,4144],{"class":481}," normal",[151,4146,3435],{"class":481},[151,4148,4149],{"class":481}," previous",[151,4151,4152],{"class":481}," requests",[151,4154,4155],{"class":481}," were",[151,4157,3902],{"class":481},[151,4159,2173],{"class":481},[151,4161,4162],{"class":481}," higher",[151,4164,4165],{"class":481}," limits.",[151,4167,4168],{"class":481}," Otherwise,",[151,4170,4091],{"class":481},[151,4172,4173],{"class":481}," let",[151,4175,4176],{"class":481}," us",[151,4178,4179],{"class":481}," know",[151,4181,3435],{"class":481},[151,4183,3902],{"class":481},[151,4185,4186],{"class":481}," changes",[151,4188,4128],{"class":481},[151,4190,4191],{"class":481}," not",[151,4193,4194],{"class":481}," reflected",[151,4196,2820],{"class":481},[151,4198,3093],{"class":481},[151,4200,4201],{"class":481}," project.\n",[151,4203,4204],{"class":469,"line":3167},[151,4205,1090],{"emptyLinePlaceholder":609},[151,4207,4208,4210,4212,4215,4217,4220,4222,4224,4227,4229,4232,4234,4237],{"class":469,"line":3175},[151,4209,4114],{"class":473},[151,4211,3438],{"class":481},[151,4213,4214],{"class":481}," want",[151,4216,2312],{"class":481},[151,4218,4219],{"class":481}," increase",[151,4221,3093],{"class":481},[151,4223,3885],{"class":481},[151,4225,4226],{"class":481}," further,",[151,4228,4091],{"class":481},[151,4230,4231],{"class":481}," file",[151,4233,2167],{"class":481},[151,4235,4236],{"class":481}," new",[151,4238,4239],{"class":481}," request.\n",[151,4241,4242],{"class":469,"line":3184},[151,4243,1090],{"emptyLinePlaceholder":609},[151,4245,4246,4249,4252,4254,4257],{"class":469,"line":3193},[151,4247,4248],{"class":473},"Best",[151,4250,4251],{"class":481}," regards",[151,4253,2181],{"class":481},[151,4255,4256],{"class":481}," happy",[151,4258,4259],{"class":481}," computing!\n",[151,4261,4262],{"class":469,"line":3720},[151,4263,1090],{"emptyLinePlaceholder":609},[151,4265,4266],{"class":469,"line":3729},[151,4267,4268],{"class":473},"Sincerely,\n",[151,4270,4271,4274,4276],{"class":469,"line":3735},[151,4272,4273],{"class":473},"Google",[151,4275,4064],{"class":481},[151,4277,4278],{"class":481}," Support\n",[11,4280,4281],{},"Awesome! There it is!",[11,4283,4284],{},[2718,4285],{"alt":2718,"src":4286},"/static/codelab/nvgcp_rate_limit.png",[11,4288,4289],{},"Now let’s try to create the GPU pool again",[459,4291,4292],{"className":461,"code":3206,"language":463,"meta":464,"style":464},[30,4293,4294,4308,4324,4334,4344,4354,4364],{"__ignoreMap":464},[151,4295,4296,4298,4300,4302,4304,4306],{"class":469,"line":470},[151,4297,1977],{"class":473},[151,4299,1980],{"class":481},[151,4301,3217],{"class":481},[151,4303,1550],{"class":481},[151,4305,3222],{"class":481},[151,4307,485],{"class":477},[151,4309,4310,4312,4314,4316,4318,4320,4322],{"class":469,"line":488},[151,4311,3229],{"class":477},[151,4313,3232],{"class":481},[151,4315,3235],{"class":503},[151,4317,3238],{"class":481},[151,4319,3241],{"class":503},[151,4321,3244],{"class":481},[151,4323,485],{"class":477},[151,4325,4326,4328,4330,4332],{"class":469,"line":500},[151,4327,1995],{"class":477},[151,4329,1998],{"class":503},[151,4331,2001],{"class":477},[151,4333,485],{"class":477},[151,4335,4336,4338,4340,4342],{"class":469,"line":509},[151,4337,2008],{"class":477},[151,4339,2011],{"class":503},[151,4341,2001],{"class":477},[151,4343,485],{"class":477},[151,4345,4346,4348,4350,4352],{"class":469,"line":517},[151,4347,3271],{"class":477},[151,4349,3274],{"class":503},[151,4351,2001],{"class":477},[151,4353,485],{"class":477},[151,4355,4356,4358,4360,4362],{"class":469,"line":534},[151,4357,2027],{"class":477},[151,4359,3285],{"class":503},[151,4361,2001],{"class":477},[151,4363,485],{"class":477},[151,4365,4366],{"class":469,"line":1413},[151,4367,2039],{"class":477},[11,4369,4370],{},"Somehow I got logged out, tried again and apparently I had already created the nodepool!",[459,4372,4374],{"className":461,"code":4373,"language":463,"meta":464,"style":464},"brian@cloudshell:~ (waywo-487618)$ gcloud container node-pools create gpupool \\\n    --accelerator type=${GPU_TYPE},count=${GPU_COUNT},gpu-driver-version=latest \\\n    --project=${PROJECT_ID} \\\n    --location=${ZONE} \\\n    --cluster=${CLUSTER_NAME} \\\n    --machine-type=${NODE_POOL_MACHINE_TYPE} \\\n    --num-nodes=1\nNote: Machines with GPUs have certain limitations which may affect your workflow. Learn more at https://cloud.google.com/kubernetes-engine/docs/how-to/gpus\nNote: Starting in GKE 1.30.1-gke.115600, if you don't specify a driver version, GKE installs the default GPU driver for your node's GKE version.\nERROR: (gcloud.container.node-pools.create) ResponseError: code=409, message=Already exists: projects/waywo-487618/zones/us-east4-a/clusters/nim-demo/nodePools/gpupool.\nbrian@cloudshell:~ (waywo-487618)$\n",[30,4375,4376,4384,4400,4410,4420,4430,4440,4444,4478,4500,4526],{"__ignoreMap":464},[151,4377,4378,4380,4382],{"class":469,"line":470},[151,4379,2130],{"class":473},[151,4381,3308],{"class":503},[151,4383,497],{"class":477},[151,4385,4386,4388,4390,4392,4394,4396,4398],{"class":469,"line":488},[151,4387,3229],{"class":473},[151,4389,3232],{"class":481},[151,4391,3235],{"class":503},[151,4393,3238],{"class":481},[151,4395,3241],{"class":503},[151,4397,3244],{"class":481},[151,4399,485],{"class":477},[151,4401,4402,4404,4406,4408],{"class":469,"line":500},[151,4403,1995],{"class":477},[151,4405,1998],{"class":503},[151,4407,2001],{"class":477},[151,4409,485],{"class":477},[151,4411,4412,4414,4416,4418],{"class":469,"line":509},[151,4413,2008],{"class":477},[151,4415,2011],{"class":503},[151,4417,2001],{"class":477},[151,4419,485],{"class":477},[151,4421,4422,4424,4426,4428],{"class":469,"line":517},[151,4423,3271],{"class":477},[151,4425,3274],{"class":503},[151,4427,2001],{"class":477},[151,4429,485],{"class":477},[151,4431,4432,4434,4436,4438],{"class":469,"line":534},[151,4433,2027],{"class":477},[151,4435,3285],{"class":503},[151,4437,2001],{"class":477},[151,4439,485],{"class":477},[151,4441,4442],{"class":469,"line":1413},[151,4443,2039],{"class":477},[151,4445,4446,4448,4450,4452,4454,4456,4458,4460,4462,4464,4466,4468,4470,4472,4474,4476],{"class":469,"line":1418},[151,4447,2781],{"class":473},[151,4449,3377],{"class":481},[151,4451,2173],{"class":481},[151,4453,3382],{"class":481},[151,4455,3385],{"class":481},[151,4457,3388],{"class":481},[151,4459,3391],{"class":481},[151,4461,3394],{"class":481},[151,4463,3397],{"class":481},[151,4465,3400],{"class":481},[151,4467,3093],{"class":481},[151,4469,3405],{"class":481},[151,4471,3408],{"class":481},[151,4473,3411],{"class":481},[151,4475,3414],{"class":481},[151,4477,3417],{"class":481},[151,4479,4480,4482,4484,4486,4488,4490,4492,4494,4496,4498],{"class":469,"line":2462},[151,4481,2781],{"class":473},[151,4483,3424],{"class":481},[151,4485,2820],{"class":481},[151,4487,3429],{"class":481},[151,4489,3432],{"class":481},[151,4491,3435],{"class":481},[151,4493,3438],{"class":481},[151,4495,3441],{"class":481},[151,4497,3429],{"class":481},[151,4499,3446],{"class":481},[151,4501,4502,4504,4507,4509,4512,4515,4517,4520,4523],{"class":469,"line":2471},[151,4503,3607],{"class":473},[151,4505,4506],{"class":503}," (gcloud.container.node-pools.create) ResponseError: code",[151,4508,1876],{"class":1869},[151,4510,4511],{"class":481},"409,",[151,4513,4514],{"class":503}," message",[151,4516,1876],{"class":1869},[151,4518,4519],{"class":481},"Already",[151,4521,4522],{"class":473}," exists:",[151,4524,4525],{"class":481}," projects/waywo-487618/zones/us-east4-a/clusters/nim-demo/nodePools/gpupool.\n",[151,4527,4528,4530],{"class":469,"line":2480},[151,4529,2130],{"class":473},[151,4531,2502],{"class":503},[736,4533,4535],{"id":4534},"good-to-go","Good to go!",[459,4537,4539],{"className":461,"code":4538,"language":463,"meta":464,"style":464},"export NGC_CLI_API_KEY=\"\u003CYOUR NGC API KEY>\"\n",[30,4540,4541],{"__ignoreMap":464},[151,4542,4543,4545,4548,4550],{"class":469,"line":470},[151,4544,1870],{"class":1869},[151,4546,4547],{"class":503}," NGC_CLI_API_KEY",[151,4549,1876],{"class":1869},[151,4551,4552],{"class":481},"\"\u003CYOUR NGC API KEY>\"\n",[736,4554,4556],{"id":4555},"export-the-ngc_api_key-env-var","Export the NGC_API_KEY env var",[11,4558,4559],{},"Get the NIM-LLM Helm chart",[459,4561,4563],{"className":461,"code":4562,"language":463,"meta":464,"style":464},"helm fetch https://helm.ngc.nvidia.com/nim/charts/nim-llm-1.3.0.tgz --username='$oauthtoken' --password=$NGC_CLI_API_KEY\n",[30,4564,4565],{"__ignoreMap":464},[151,4566,4567,4570,4573,4576,4579,4582,4585],{"class":469,"line":470},[151,4568,4569],{"class":473},"helm",[151,4571,4572],{"class":481}," fetch",[151,4574,4575],{"class":481}," https://helm.ngc.nvidia.com/nim/charts/nim-llm-1.3.0.tgz",[151,4577,4578],{"class":477}," --username=",[151,4580,4581],{"class":481},"'$oauthtoken'",[151,4583,4584],{"class":477}," --password=",[151,4586,4587],{"class":503},"$NGC_CLI_API_KEY\n",[11,4589,4590],{},"This just downloads a file to our current working directory:",[459,4592,4594],{"className":461,"code":4593,"language":463,"meta":464,"style":464},"brian@cloudshell:~ (waywo-487618)$ ls\nnim-llm-1.3.0.tgz  README-cloudshell.txt\n",[30,4595,4596,4603],{"__ignoreMap":464},[151,4597,4598,4600],{"class":469,"line":470},[151,4599,2130],{"class":473},[151,4601,4602],{"class":503}," (waywo-487618)$ ls\n",[151,4604,4605,4608],{"class":469,"line":488},[151,4606,4607],{"class":473},"nim-llm-1.3.0.tgz",[151,4609,4610],{"class":481},"  README-cloudshell.txt\n",[736,4612,4614],{"id":4613},"create-a-namespace","Create a namespace",[459,4616,4618],{"className":461,"code":4617,"language":463,"meta":464,"style":464},"kubectl create namespace nim\n",[30,4619,4620],{"__ignoreMap":464},[151,4621,4622,4625,4627,4630],{"class":469,"line":470},[151,4623,4624],{"class":473},"kubectl",[151,4626,1550],{"class":481},[151,4628,4629],{"class":481}," namespace",[151,4631,4632],{"class":481}," nim\n",[736,4634,2954],{"id":4635},"created-1",[459,4637,4639],{"className":461,"code":4638,"language":463,"meta":464,"style":464},"brian@cloudshell:~ (waywo-487618)$ kubectl create namespace nim\nnamespace/nim created\n",[30,4640,4641,4648],{"__ignoreMap":464},[151,4642,4643,4645],{"class":469,"line":470},[151,4644,2130],{"class":473},[151,4646,4647],{"class":503}," (waywo-487618)$ kubectl create namespace nim\n",[151,4649,4650,4653],{"class":469,"line":488},[151,4651,4652],{"class":473},"namespace/nim",[151,4654,4655],{"class":481}," created\n",[736,4657,4659],{"id":4658},"configure-secrets","Configure secrets",[459,4661,4663],{"className":461,"code":4662,"language":463,"meta":464,"style":464},"kubectl create secret docker-registry registry-secret --docker-server=nvcr.io --docker-username='$oauthtoken'     --docker-password=$NGC_CLI_API_KEY -n nim\n\nkubectl create secret generic ngc-api --from-literal=NGC_API_KEY=$NGC_CLI_API_KEY -n nim\n\n",[30,4664,4665,4699,4703],{"__ignoreMap":464},[151,4666,4667,4669,4671,4674,4677,4680,4683,4686,4688,4691,4694,4697],{"class":469,"line":470},[151,4668,4624],{"class":473},[151,4670,1550],{"class":481},[151,4672,4673],{"class":481}," secret",[151,4675,4676],{"class":481}," docker-registry",[151,4678,4679],{"class":481}," registry-secret",[151,4681,4682],{"class":477}," --docker-server=nvcr.io",[151,4684,4685],{"class":477}," --docker-username=",[151,4687,4581],{"class":481},[151,4689,4690],{"class":477},"     --docker-password=",[151,4692,4693],{"class":503},"$NGC_CLI_API_KEY",[151,4695,4696],{"class":477}," -n",[151,4698,4632],{"class":481},[151,4700,4701],{"class":469,"line":488},[151,4702,1090],{"emptyLinePlaceholder":609},[151,4704,4705,4707,4709,4711,4714,4717,4720,4722,4724],{"class":469,"line":500},[151,4706,4624],{"class":473},[151,4708,1550],{"class":481},[151,4710,4673],{"class":481},[151,4712,4713],{"class":481}," generic",[151,4715,4716],{"class":481}," ngc-api",[151,4718,4719],{"class":477}," --from-literal=NGC_API_KEY=",[151,4721,4693],{"class":503},[151,4723,4696],{"class":477},[151,4725,4632],{"class":481},[459,4727,4729],{"className":461,"code":4728,"language":463,"meta":464,"style":464},"brian@cloudshell:~ (waywo-487618)$ kubectl create secret docker-registry registry-secret --docker-server=nvcr.io --docker-username='$oauthtoken'     --docker-password=$NGC_CLI_API_KEY -n nim\nsecret/registry-secret created\n",[30,4730,4731,4763],{"__ignoreMap":464},[151,4732,4733,4735,4738,4740,4743,4746,4748,4750,4753,4755,4758,4761],{"class":469,"line":470},[151,4734,2130],{"class":473},[151,4736,4737],{"class":503}," (waywo-487618)$ kubectl create secret docker-registry registry-secret --docker-server",[151,4739,1876],{"class":1869},[151,4741,4742],{"class":481},"nvcr.io",[151,4744,4745],{"class":503}," --docker-username",[151,4747,1876],{"class":1869},[151,4749,4581],{"class":481},[151,4751,4752],{"class":503},"     --docker-password",[151,4754,1876],{"class":1869},[151,4756,4757],{"class":503},"$NGC_CLI_API_KEY ",[151,4759,4760],{"class":473},"-n",[151,4762,4632],{"class":481},[151,4764,4765,4768],{"class":469,"line":488},[151,4766,4767],{"class":473},"secret/registry-secret",[151,4769,4655],{"class":481},[459,4771,4773],{"className":461,"code":4772,"language":463,"meta":464,"style":464},"brian@cloudshell:~ (waywo-487618)$ kubectl create secret generic ngc-api --from-literal=NGC_API_KEY=$NGC_CLI_API_KEY -n nim\nsecret/ngc-api created\n",[30,4774,4775,4795],{"__ignoreMap":464},[151,4776,4777,4779,4782,4784,4787,4789,4791,4793],{"class":469,"line":470},[151,4778,2130],{"class":473},[151,4780,4781],{"class":503}," (waywo-487618)$ kubectl create secret generic ngc-api --from-literal",[151,4783,1876],{"class":1869},[151,4785,4786],{"class":503},"NGC_API_KEY",[151,4788,1876],{"class":1869},[151,4790,4757],{"class":503},[151,4792,4760],{"class":473},[151,4794,4632],{"class":481},[151,4796,4797,4800],{"class":469,"line":488},[151,4798,4799],{"class":473},"secret/ngc-api",[151,4801,4655],{"class":481},[11,4803,4804,4805,4808],{},"Cool, this makes sense, I need the NGC API key to both download the container from ",[15,4806,4807],{},"nvcr"," (NVIDIA container registry), AND download the model weights from nvcr.",[736,4810,4812],{"id":4811},"nim-configuration","NIM Configuration",[459,4814,4816],{"className":461,"code":4815,"language":463,"meta":464,"style":464},"cat \u003C\u003CEOF > nim_custom_value.yaml\nimage:\n  repository: \"nvcr.io/nim/meta/llama3-8b-instruct\" # container location\n  tag: 1.0.0 # NIM version you want to deploy\nmodel:\n  ngcAPISecret: ngc-api  # name of a secret in the cluster that includes a key named NGC_CLI_API_KEY and is an NGC API key\npersistence:\n  enabled: true\nimagePullSecrets:\n  -   name: registry-secret # name of a secret used to pull nvcr.io images, see https://kubernetes.io/docs/tasks/    configure-pod-container/pull-image-private-registry/\nEOF\n",[30,4817,4818,4836,4841,4846,4851,4856,4861,4866,4871,4876,4881],{"__ignoreMap":464},[151,4819,4820,4823,4826,4830,4833],{"class":469,"line":470},[151,4821,4822],{"class":473},"cat",[151,4824,4825],{"class":1869}," \u003C\u003C",[151,4827,4829],{"class":4828},"sinWB","EOF",[151,4831,4832],{"class":1869}," >",[151,4834,4835],{"class":481}," nim_custom_value.yaml\n",[151,4837,4838],{"class":469,"line":488},[151,4839,4840],{"class":481},"image:\n",[151,4842,4843],{"class":469,"line":500},[151,4844,4845],{"class":481},"  repository: \"nvcr.io/nim/meta/llama3-8b-instruct\" # container location\n",[151,4847,4848],{"class":469,"line":509},[151,4849,4850],{"class":481},"  tag: 1.0.0 # NIM version you want to deploy\n",[151,4852,4853],{"class":469,"line":517},[151,4854,4855],{"class":481},"model:\n",[151,4857,4858],{"class":469,"line":534},[151,4859,4860],{"class":481},"  ngcAPISecret: ngc-api  # name of a secret in the cluster that includes a key named NGC_CLI_API_KEY and is an NGC API key\n",[151,4862,4863],{"class":469,"line":1413},[151,4864,4865],{"class":481},"persistence:\n",[151,4867,4868],{"class":469,"line":1418},[151,4869,4870],{"class":481},"  enabled: true\n",[151,4872,4873],{"class":469,"line":2462},[151,4874,4875],{"class":481},"imagePullSecrets:\n",[151,4877,4878],{"class":469,"line":2471},[151,4879,4880],{"class":481},"  -   name: registry-secret # name of a secret used to pull nvcr.io images, see https://kubernetes.io/docs/tasks/    configure-pod-container/pull-image-private-registry/\n",[151,4882,4883],{"class":469,"line":2480},[151,4884,4885],{"class":4828},"EOF\n",[459,4887,4889],{"className":461,"code":4888,"language":463,"meta":464,"style":464},"brian@cloudshell:~ (waywo-487618)$ cat \u003C\u003CEOF > nim_custom_value.yaml\nimage:\n  repository: \"nvcr.io/nim/meta/llama3-8b-instruct\" # container location\n  tag: 1.0.0 # NIM version you want to deploy\nmodel:\n  ngcAPISecret: ngc-api  # name of a secret in the cluster that includes a key named NGC_CLI_API_KEY and is an NGC API key\npersistence:\n  enabled: true\nimagePullSecrets:\n  -   name: registry-secret # name of a secret used to pull nvcr.io images, see https://kubernetes.io/docs/tasks/    configure-pod-container/pull-image-private-registry/\nEOF\nbrian@cloudshell:~ (waywo-487618)$ ls\nnim_custom_value.yaml  nim-llm-1.3.0.tgz  README-cloudshell.txt\n",[30,4890,4891,4907,4911,4915,4919,4923,4927,4931,4935,4939,4943,4947,4953],{"__ignoreMap":464},[151,4892,4893,4895,4898,4901,4903,4905],{"class":469,"line":470},[151,4894,2130],{"class":473},[151,4896,4897],{"class":503}," (waywo-487618)$ cat ",[151,4899,4900],{"class":1869},"\u003C\u003C",[151,4902,4829],{"class":4828},[151,4904,4832],{"class":1869},[151,4906,4835],{"class":481},[151,4908,4909],{"class":469,"line":488},[151,4910,4840],{"class":481},[151,4912,4913],{"class":469,"line":500},[151,4914,4845],{"class":481},[151,4916,4917],{"class":469,"line":509},[151,4918,4850],{"class":481},[151,4920,4921],{"class":469,"line":517},[151,4922,4855],{"class":481},[151,4924,4925],{"class":469,"line":534},[151,4926,4860],{"class":481},[151,4928,4929],{"class":469,"line":1413},[151,4930,4865],{"class":481},[151,4932,4933],{"class":469,"line":1418},[151,4934,4870],{"class":481},[151,4936,4937],{"class":469,"line":2462},[151,4938,4875],{"class":481},[151,4940,4941],{"class":469,"line":2471},[151,4942,4880],{"class":481},[151,4944,4945],{"class":469,"line":2480},[151,4946,4885],{"class":4828},[151,4948,4949,4951],{"class":469,"line":2489},[151,4950,2130],{"class":473},[151,4952,4602],{"class":503},[151,4954,4955,4958,4961],{"class":469,"line":2497},[151,4956,4957],{"class":473},"nim_custom_value.yaml",[151,4959,4960],{"class":481},"  nim-llm-1.3.0.tgz",[151,4962,4610],{"class":481},[736,4964,4966],{"id":4965},"create-the-deployment","Create the deployment",[459,4968,4970],{"className":461,"code":4969,"language":463,"meta":464,"style":464},"brian@cloudshell:~ (waywo-487618)$ helm install my-nim nim-llm-1.1.2.tgz -f nim_custom_value.yaml --namespace nim\nError: INSTALLATION FAILED: non-absolute URLs should be in form of repo_name/path_to_chart, got: nim-llm-1.1.2.tgz\n",[30,4971,4972,4979],{"__ignoreMap":464},[151,4973,4974,4976],{"class":469,"line":470},[151,4975,2130],{"class":473},[151,4977,4978],{"class":503}," (waywo-487618)$ helm install my-nim nim-llm-1.1.2.tgz -f nim_custom_value.yaml --namespace nim\n",[151,4980,4981,4984,4987,4990,4993,4996,4999,5001,5003,5006,5008,5011,5014],{"class":469,"line":488},[151,4982,4983],{"class":473},"Error:",[151,4985,4986],{"class":481}," INSTALLATION",[151,4988,4989],{"class":481}," FAILED:",[151,4991,4992],{"class":481}," non-absolute",[151,4994,4995],{"class":481}," URLs",[151,4997,4998],{"class":481}," should",[151,5000,4051],{"class":481},[151,5002,2820],{"class":481},[151,5004,5005],{"class":481}," form",[151,5007,3090],{"class":481},[151,5009,5010],{"class":481}," repo_name/path_to_chart,",[151,5012,5013],{"class":481}," got:",[151,5015,5016],{"class":481}," nim-llm-1.1.2.tgz\n",[11,5018,5019],{},"Oops, I got an error. Let’s ask Cloud Assist!",[459,5021,5023],{"className":461,"code":5022,"language":463,"meta":464,"style":464},"The error message \"Error: INSTALLATION FAILED: non-absolute URLs should be in form of repo_name/path_to_chart, got: nim-llm-1.1.2.tgz\" indicates that Helm expects a chart reference in a specific format, but it received a local file path.\n\nHere's how to resolve it:\n\nIf nim-llm-1.1.2.tgz is a local Helm chart package, you should use a relative path or an absolute path without the repo_name/ prefix.\n\nYou can try one of these options:\n\nUsing a relative path (if the file is in the current directory):\nhelm install my-nim ./nim-llm-1.1.2.tgz -f nim_custom_value.yaml --namespace nim\nGenerated code may be subject to license restrictions not shown here. Use code with care. Learn more\n\nAdding ./ explicitly tells Helm that nim-llm-1.1.2.tgz is a local file in the current directory.\nUsing an absolute path:\nIf the file is not in the current directory, provide its full path. For example:\nhelm install my-nim /path/to/your/nim-llm-1.1.2.tgz -f nim_custom_value.yaml --namespace nim\nGenerated code may be subject to license restrictions not shown here. Use code with care. Learn more\n\nReplace /path/to/your/ with the actual directory where nim-llm-1.1.2.tgz is located.\nThe key is to make it clear to Helm that you are providing a local chart archive, not a chart from a Helm repository.\n",[30,5024,5025,5087,5091,5099,5103,5108,5112,5117,5121,5126,5131,5136,5140,5145,5150,5155,5160,5164,5168,5173],{"__ignoreMap":464},[151,5026,5027,5030,5033,5035,5038,5041,5044,5047,5050,5052,5055,5058,5060,5062,5065,5068,5071,5074,5077,5079,5082,5084],{"class":469,"line":470},[151,5028,5029],{"class":473},"The",[151,5031,5032],{"class":481}," error",[151,5034,4514],{"class":481},[151,5036,5037],{"class":481}," \"Error: INSTALLATION FAILED: non-absolute URLs should be in form of repo_name/path_to_chart, got: nim-llm-1.1.2.tgz\"",[151,5039,5040],{"class":481}," indicates",[151,5042,5043],{"class":481}," that",[151,5045,5046],{"class":481}," Helm",[151,5048,5049],{"class":481}," expects",[151,5051,2167],{"class":481},[151,5053,5054],{"class":481}," chart",[151,5056,5057],{"class":481}," reference",[151,5059,2820],{"class":481},[151,5061,2167],{"class":481},[151,5063,5064],{"class":481}," specific",[151,5066,5067],{"class":481}," format,",[151,5069,5070],{"class":481}," but",[151,5072,5073],{"class":481}," it",[151,5075,5076],{"class":481}," received",[151,5078,2167],{"class":481},[151,5080,5081],{"class":481}," local",[151,5083,4231],{"class":481},[151,5085,5086],{"class":481}," path.\n",[151,5088,5089],{"class":469,"line":488},[151,5090,1090],{"emptyLinePlaceholder":609},[151,5092,5093,5096],{"class":469,"line":500},[151,5094,5095],{"class":473},"Here",[151,5097,5098],{"class":473},"'s how to resolve it:\n",[151,5100,5101],{"class":469,"line":509},[151,5102,1090],{"emptyLinePlaceholder":609},[151,5104,5105],{"class":469,"line":517},[151,5106,5107],{"class":473},"If nim-llm-1.1.2.tgz is a local Helm chart package, you should use a relative path or an absolute path without the repo_name/ prefix.\n",[151,5109,5110],{"class":469,"line":534},[151,5111,1090],{"emptyLinePlaceholder":609},[151,5113,5114],{"class":469,"line":1413},[151,5115,5116],{"class":473},"You can try one of these options:\n",[151,5118,5119],{"class":469,"line":1418},[151,5120,1090],{"emptyLinePlaceholder":609},[151,5122,5123],{"class":469,"line":2462},[151,5124,5125],{"class":473},"Using a relative path (if the file is in the current directory):\n",[151,5127,5128],{"class":469,"line":2471},[151,5129,5130],{"class":473},"helm install my-nim ./nim-llm-1.1.2.tgz -f nim_custom_value.yaml --namespace nim\n",[151,5132,5133],{"class":469,"line":2480},[151,5134,5135],{"class":473},"Generated code may be subject to license restrictions not shown here. Use code with care. Learn more\n",[151,5137,5138],{"class":469,"line":2489},[151,5139,1090],{"emptyLinePlaceholder":609},[151,5141,5142],{"class":469,"line":2497},[151,5143,5144],{"class":473},"Adding ./ explicitly tells Helm that nim-llm-1.1.2.tgz is a local file in the current directory.\n",[151,5146,5147],{"class":469,"line":3140},[151,5148,5149],{"class":473},"Using an absolute path:\n",[151,5151,5152],{"class":469,"line":3149},[151,5153,5154],{"class":473},"If the file is not in the current directory, provide its full path. For example:\n",[151,5156,5157],{"class":469,"line":3158},[151,5158,5159],{"class":473},"helm install my-nim /path/to/your/nim-llm-1.1.2.tgz -f nim_custom_value.yaml --namespace nim\n",[151,5161,5162],{"class":469,"line":3167},[151,5163,5135],{"class":473},[151,5165,5166],{"class":469,"line":3175},[151,5167,1090],{"emptyLinePlaceholder":609},[151,5169,5170],{"class":469,"line":3184},[151,5171,5172],{"class":473},"Replace /path/to/your/ with the actual directory where nim-llm-1.1.2.tgz is located.\n",[151,5174,5175],{"class":469,"line":3193},[151,5176,5177],{"class":473},"The key is to make it clear to Helm that you are providing a local chart archive, not a chart from a Helm repository.\n",[11,5179,5180],{},"Oh! The helm version is different than the one that was in the guide:",[459,5182,5184],{"className":461,"code":5183,"language":463,"meta":464,"style":464},"brian@cloudshell:~ (waywo-487618)$ helm install my-nim ./nim-llm-1.1.2.tgz -f nim_custom_value.yaml --namespace nim\nError: INSTALLATION FAILED: path \"./nim-llm-1.1.2.tgz\" not found\n",[30,5185,5186,5193],{"__ignoreMap":464},[151,5187,5188,5190],{"class":469,"line":470},[151,5189,2130],{"class":473},[151,5191,5192],{"class":503}," (waywo-487618)$ helm install my-nim ./nim-llm-1.1.2.tgz -f nim_custom_value.yaml --namespace nim\n",[151,5194,5195,5197,5199,5201,5204,5207,5209],{"class":469,"line":488},[151,5196,4983],{"class":473},[151,5198,4986],{"class":481},[151,5200,4989],{"class":481},[151,5202,5203],{"class":481}," path",[151,5205,5206],{"class":481}," \"./nim-llm-1.1.2.tgz\"",[151,5208,4191],{"class":481},[151,5210,5211],{"class":481}," found\n",[11,5213,5214],{},"Trying again with v1.3.0, we get a successful installation!",[459,5216,5218],{"className":461,"code":5217,"language":463,"meta":464,"style":464},"brian@cloudshell:~ (waywo-487618)$ helm install my-nim ./nim-llm-1.3.0.tgz -f nim_custom_value.yaml --namespace nim\nNAME: my-nim\nLAST DEPLOYED: Fri Feb 20 21:43:37 2026\nNAMESPACE: nim\nSTATUS: deployed\nREVISION: 1\nDESCRIPTION: Install complete\nNOTES:\nThank you for installing nim-llm.\n\n**************************************************\n| It may take some time for pods to become ready |\n| while model files download                     |\n**************************************************\n\nYour NIM version is: 1.0.0\nbrian@cloudshell:~ (waywo-487618)$\n",[30,5219,5220,5227,5234,5257,5264,5271,5278,5289,5294,5309,5313,5318,5350,5369,5373,5377,5393],{"__ignoreMap":464},[151,5221,5222,5224],{"class":469,"line":470},[151,5223,2130],{"class":473},[151,5225,5226],{"class":503}," (waywo-487618)$ helm install my-nim ./nim-llm-1.3.0.tgz -f nim_custom_value.yaml --namespace nim\n",[151,5228,5229,5231],{"class":469,"line":488},[151,5230,3126],{"class":473},[151,5232,5233],{"class":481}," my-nim\n",[151,5235,5236,5239,5242,5245,5248,5251,5254],{"class":469,"line":500},[151,5237,5238],{"class":473},"LAST",[151,5240,5241],{"class":481}," DEPLOYED:",[151,5243,5244],{"class":481}," Fri",[151,5246,5247],{"class":481}," Feb",[151,5249,5250],{"class":477}," 20",[151,5252,5253],{"class":481}," 21:43:37",[151,5255,5256],{"class":477}," 2026\n",[151,5258,5259,5262],{"class":469,"line":509},[151,5260,5261],{"class":473},"NAMESPACE:",[151,5263,4632],{"class":481},[151,5265,5266,5268],{"class":469,"line":517},[151,5267,3187],{"class":473},[151,5269,5270],{"class":481}," deployed\n",[151,5272,5273,5276],{"class":469,"line":534},[151,5274,5275],{"class":473},"REVISION:",[151,5277,3181],{"class":477},[151,5279,5280,5283,5286],{"class":469,"line":1413},[151,5281,5282],{"class":473},"DESCRIPTION:",[151,5284,5285],{"class":481}," Install",[151,5287,5288],{"class":481}," complete\n",[151,5290,5291],{"class":469,"line":1418},[151,5292,5293],{"class":473},"NOTES:\n",[151,5295,5296,5299,5301,5303,5306],{"class":469,"line":2462},[151,5297,5298],{"class":473},"Thank",[151,5300,3438],{"class":481},[151,5302,2235],{"class":481},[151,5304,5305],{"class":481}," installing",[151,5307,5308],{"class":481}," nim-llm.\n",[151,5310,5311],{"class":469,"line":2471},[151,5312,1090],{"emptyLinePlaceholder":609},[151,5314,5315],{"class":469,"line":2480},[151,5316,5317],{"class":1869},"**************************************************\n",[151,5319,5320,5322,5325,5327,5329,5332,5335,5337,5340,5342,5345,5348],{"class":469,"line":2489},[151,5321,3947],{"class":1869},[151,5323,5324],{"class":473}," It",[151,5326,3397],{"class":481},[151,5328,4035],{"class":481},[151,5330,5331],{"class":481}," some",[151,5333,5334],{"class":481}," time",[151,5336,2235],{"class":481},[151,5338,5339],{"class":481}," pods",[151,5341,2312],{"class":481},[151,5343,5344],{"class":481}," become",[151,5346,5347],{"class":481}," ready",[151,5349,3979],{"class":1869},[151,5351,5352,5354,5357,5360,5363,5366],{"class":469,"line":2497},[151,5353,3947],{"class":1869},[151,5355,5356],{"class":1869}," while",[151,5358,5359],{"class":473}," model",[151,5361,5362],{"class":481}," files",[151,5364,5365],{"class":481}," download",[151,5367,5368],{"class":1869},"                     |\n",[151,5370,5371],{"class":469,"line":3140},[151,5372,5317],{"class":473},[151,5374,5375],{"class":469,"line":3149},[151,5376,1090],{"emptyLinePlaceholder":609},[151,5378,5379,5381,5384,5387,5390],{"class":469,"line":3158},[151,5380,3882],{"class":473},[151,5382,5383],{"class":481}," NIM",[151,5385,5386],{"class":481}," version",[151,5388,5389],{"class":481}," is:",[151,5391,5392],{"class":477}," 1.0.0\n",[151,5394,5395,5397],{"class":469,"line":3167},[151,5396,2130],{"class":473},[151,5398,2502],{"class":503},[11,5400,5401,5402,5405],{},"Let’s check on the pod in the ",[30,5403,5404],{},"nim"," namespace we created:",[459,5407,5409],{"className":461,"code":5408,"language":463,"meta":464,"style":464},"brian@cloudshell:~ (waywo-487618)$ kubectl get pods -n nim\nNAME               READY   STATUS    RESTARTS   AGE\nmy-nim-nim-llm-0   0/1     Running   0          2m58s\n",[30,5410,5411,5418,5435],{"__ignoreMap":464},[151,5412,5413,5415],{"class":469,"line":470},[151,5414,2130],{"class":473},[151,5416,5417],{"class":503}," (waywo-487618)$ kubectl get pods -n nim\n",[151,5419,5420,5423,5426,5429,5432],{"class":469,"line":488},[151,5421,5422],{"class":473},"NAME",[151,5424,5425],{"class":481},"               READY",[151,5427,5428],{"class":481},"   STATUS",[151,5430,5431],{"class":481},"    RESTARTS",[151,5433,5434],{"class":481},"   AGE\n",[151,5436,5437,5440,5443,5446,5449],{"class":469,"line":500},[151,5438,5439],{"class":473},"my-nim-nim-llm-0",[151,5441,5442],{"class":481},"   0/1",[151,5444,5445],{"class":481},"     Running",[151,5447,5448],{"class":477},"   0",[151,5450,5451],{"class":481},"          2m58s\n",[11,5453,5454],{},"Awesome! Cloud Assist has access to my k8s cluster and was able to show me logs directly in the chat response!",[11,5456,5457],{},[2718,5458],{"alt":2718,"src":5459},"/static/codelab/nvgcp_cloud_assist_tool_use.png",[459,5461,5463],{"className":461,"code":5462,"language":463,"meta":464,"style":464},"Container image Copyright (c) 2016-2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.\n\nThis NIM container is governed by the NVIDIA AI Product Agreement here:\nhttps://www.nvidia.com/en-us/data-center/products/nvidia-ai-enterprise/eula/.\nA copy of this license can be found under /opt/nim/LICENSE.\n\nThe use of this model is governed by the AI Foundation Models Community License\nhere: https://docs.nvidia.com/ai-foundation-models-community-license.pdf.\n\nADDITIONAL INFORMATION: Meta Llama 3 Community License, Built with Meta Llama 3.\nA copy of the Llama 3 license can be found under /opt/nim/MODEL_LICENSE.\n\n2026-02-20 21:46:26,056 [INFO] PyTorch version 2.2.2 available.\n2026-02-20 21:46:26,645 [WARNING] [TRT-LLM] [W] Logger level already set from environment. Discard new verbosity: error\n2026-02-20 21:46:26,645 [INFO] [TRT-LLM] [I] Starting TensorRT-LLM init.\n[TensorRT-LLM][INFO] Set logger level by INFO\n2026-02-20 21:46:26,777 [INFO] [TRT-LLM] [I] TensorRT-LLM inited.\n[TensorRT-LLM] TensorRT-LLM version: 0.10.1.dev2024053000\n{\"level\": \"INFO\", \"time\": \"02-20 21:46:27.896\", \"file_path\": \"/usr/local/lib/python3.10/dist-packages/vllm_nvext/entrypoints/openai/api_server.py\", \"line_number\": \"489\", \"message\": \"NIM LLM API version 1.0.0\", \"exc_info\": \"None\", \"stack_info\": \"None\"}\n{\"level\": \"INFO\", \"time\": \"02-20 21:46:27.898\", \"file_path\": \"/usr/local/lib/python3.10/dist-packages/vllm_nvext/hub/ngc_profile.py\", \"line_number\": \"217\", \"message\": \"Running NIM without LoRA. Only looking for compatible profiles that do not support LoRA.\", \"exc_info\": \"None\", \"stack_info\": \"None\"}\n{\"level\": \"INFO\", \"time\": \"02-20 21:46:27.898\", \"file_path\": \"/usr/local/lib/python3.10/dist-packages/vllm_nvext/hub/ngc_profile.py\", \"line_number\": \"219\", \"message\": \"Detected 1 compatible profile(s).\", \"exc_info\": \"None\", \"stack_info\": \"None\"}\n{\"level\": \"INFO\", \"time\": \"02-20 21:46:27.898\", \"file_path\": \"/usr/local/lib/python3.10/dist-packages/vllm_nvext/hub/ngc_injector.py\", \"line_number\": \"106\", \"message\": \"Valid profile: 8835c31752fbc67ef658b20a9f78e056914fdef0660206d82f252d62fd96064d (vllm-fp16-tp1) on GPUs [0]\", \"exc_info\": \"None\", \"stack_info\": \"None\"}\n{\"level\": \"INFO\", \"time\": \"02-20 21:46:27.898\", \"file_path\": \"/usr/local/lib/python3.10/dist-packages/vllm_nvext/hub/ngc_injector.py\", \"line_number\": \"141\", \"message\": \"Selected profile: 8835c31752fbc67ef658b20a9f78e056914fdef0660206d82f252d62fd96064d (vllm-fp16-tp1)\", \"exc_info\": \"None\", \"stack_info\": \"None\"}\n{\"level\": \"INFO\", \"time\": \"02-20 21:46:28.585\", \"file_path\": \"/usr/local/lib/python3.10/dist-packages/vllm_nvext/hub/ngc_injector.py\", \"line_number\": \"146\", \"message\": \"Profile metadata: tp: 1\", \"exc_info\": \"None\", \"stack_info\": \"None\"}\n{\"level\": \"INFO\", \"time\": \"02-20 21:46:28.586\", \"file_path\": \"/usr/local/lib/python3.10/dist-packages/vllm_nvext/hub/ngc_injector.py\", \"line_number\": \"146\", \"message\": \"Profile metadata: llm_engine: vllm\", \"exc_info\": \"None\", \"stack_info\": \"None\"}\n{\"level\": \"INFO\", \"time\": \"02-20 21:46:28.586\", \"file_path\": \"/usr/local/lib/python3.10/dist-packages/vllm_nvext/hub/ngc_injector.py\", \"line_number\": \"146\", \"message\": \"Profile metadata: precision: fp16\", \"exc_info\": \"None\", \"stack_info\": \"None\"}\n{\"level\": \"INFO\", \"time\": \"02-20 21:46:28.586\", \"file_path\": \"/usr/local/lib/python3.10/dist-packages/vllm_nvext/hub/ngc_injector.py\", \"line_number\": \"146\", \"message\": \"Profile metadata: feat_lora: false\", \"exc_info\": \"None\", \"stack_info\": \"None\"}\n{\"level\": \"INFO\", \"time\": \"02-20 21:46:28.586\", \"file_path\": \"/usr/local/lib/python3.10/dist-packages/vllm_nvext/hub/ngc_injector.py\", \"line_number\": \"166\", \"message\": \"Preparing model workspace. This step might download additional files to run the model.\", \"exc_info\": \"None\", \"stack_info\": \"None\"}\n{\"level\": \"INFO\", \"time\": \"02-20 21:47:14.944\", \"file_path\": \"/usr/local/lib/python3.10/dist-packages/vllm_nvext/hub/ngc_injector.py\", \"line_number\": \"172\", \"message\": \"Model workspace is now ready. It took 46.358 seconds\", \"exc_info\": \"None\", \"stack_info\": \"None\"}\n{\"level\": \"INFO\", \"time\": \"02-20 21:47:14.949\", \"file_path\": \"/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py\", \"line_number\": \"98\", \"message\": \"Initializing an LLM engine (v0.4.1) with config: model='/tmp/meta--llama3-8b-instruct-tbtjj6ij', speculative_config=None,\n\n",[30,5464,5465,5491,5495,5529,5534,5561,5565,5599,5607,5611,5644,5671,5675,5686,5696,5705,5710,5720,5725,5774,5812,5848,5885,5921,5958,5994,6029,6064,6100,6137],{"__ignoreMap":464},[151,5466,5467,5470,5473,5476,5479,5482,5485,5488],{"class":469,"line":470},[151,5468,5469],{"class":473},"Container",[151,5471,5472],{"class":481}," image",[151,5474,5475],{"class":481}," Copyright",[151,5477,5478],{"class":503}," (c) 2016-2024, NVIDIA CORPORATION & ",[151,5480,5481],{"class":473},"AFFILIATES.",[151,5483,5484],{"class":481}," All",[151,5486,5487],{"class":481}," rights",[151,5489,5490],{"class":481}," reserved.\n",[151,5492,5493],{"class":469,"line":488},[151,5494,1090],{"emptyLinePlaceholder":609},[151,5496,5497,5500,5502,5504,5506,5509,5512,5514,5517,5520,5523,5526],{"class":469,"line":500},[151,5498,5499],{"class":473},"This",[151,5501,5383],{"class":481},[151,5503,1980],{"class":481},[151,5505,2306],{"class":481},[151,5507,5508],{"class":481}," governed",[151,5510,5511],{"class":481}," by",[151,5513,3084],{"class":481},[151,5515,5516],{"class":481}," NVIDIA",[151,5518,5519],{"class":481}," AI",[151,5521,5522],{"class":481}," Product",[151,5524,5525],{"class":481}," Agreement",[151,5527,5528],{"class":481}," here:\n",[151,5530,5531],{"class":469,"line":509},[151,5532,5533],{"class":473},"https://www.nvidia.com/en-us/data-center/products/nvidia-ai-enterprise/eula/.\n",[151,5535,5536,5538,5541,5543,5545,5548,5550,5552,5555,5558],{"class":469,"line":517},[151,5537,105],{"class":473},[151,5539,5540],{"class":481}," copy",[151,5542,3090],{"class":481},[151,5544,2324],{"class":481},[151,5546,5547],{"class":481}," license",[151,5549,4032],{"class":481},[151,5551,4051],{"class":481},[151,5553,5554],{"class":481}," found",[151,5556,5557],{"class":481}," under",[151,5559,5560],{"class":481}," /opt/nim/LICENSE.\n",[151,5562,5563],{"class":469,"line":534},[151,5564,1090],{"emptyLinePlaceholder":609},[151,5566,5567,5569,5571,5573,5575,5577,5579,5581,5583,5585,5587,5590,5593,5596],{"class":469,"line":1413},[151,5568,5029],{"class":473},[151,5570,2321],{"class":481},[151,5572,3090],{"class":481},[151,5574,2324],{"class":481},[151,5576,5359],{"class":481},[151,5578,2306],{"class":481},[151,5580,5508],{"class":481},[151,5582,5511],{"class":481},[151,5584,3084],{"class":481},[151,5586,5519],{"class":481},[151,5588,5589],{"class":481}," Foundation",[151,5591,5592],{"class":481}," Models",[151,5594,5595],{"class":481}," Community",[151,5597,5598],{"class":481}," License\n",[151,5600,5601,5604],{"class":469,"line":1418},[151,5602,5603],{"class":473},"here:",[151,5605,5606],{"class":481}," https://docs.nvidia.com/ai-foundation-models-community-license.pdf.\n",[151,5608,5609],{"class":469,"line":2462},[151,5610,1090],{"emptyLinePlaceholder":609},[151,5612,5613,5616,5619,5622,5625,5627,5629,5632,5635,5637,5639,5641],{"class":469,"line":2471},[151,5614,5615],{"class":473},"ADDITIONAL",[151,5617,5618],{"class":481}," INFORMATION:",[151,5620,5621],{"class":481}," Meta",[151,5623,5624],{"class":481}," Llama",[151,5626,3650],{"class":477},[151,5628,5595],{"class":481},[151,5630,5631],{"class":481}," License,",[151,5633,5634],{"class":481}," Built",[151,5636,2173],{"class":481},[151,5638,5621],{"class":481},[151,5640,5624],{"class":481},[151,5642,5643],{"class":481}," 3.\n",[151,5645,5646,5648,5650,5652,5654,5656,5658,5660,5662,5664,5666,5668],{"class":469,"line":2480},[151,5647,105],{"class":473},[151,5649,5540],{"class":481},[151,5651,3090],{"class":481},[151,5653,3084],{"class":481},[151,5655,5624],{"class":481},[151,5657,3650],{"class":477},[151,5659,5547],{"class":481},[151,5661,4032],{"class":481},[151,5663,4051],{"class":481},[151,5665,5554],{"class":481},[151,5667,5557],{"class":481},[151,5669,5670],{"class":481}," /opt/nim/MODEL_LICENSE.\n",[151,5672,5673],{"class":469,"line":2489},[151,5674,1090],{"emptyLinePlaceholder":609},[151,5676,5677,5680,5683],{"class":469,"line":2497},[151,5678,5679],{"class":473},"2026-02-20",[151,5681,5682],{"class":481}," 21:46:26,056",[151,5684,5685],{"class":503}," [INFO] PyTorch version 2.2.2 available.\n",[151,5687,5688,5690,5693],{"class":469,"line":3140},[151,5689,5679],{"class":473},[151,5691,5692],{"class":481}," 21:46:26,645",[151,5694,5695],{"class":503}," [WARNING] [TRT-LLM] [W] Logger level already set from environment. Discard new verbosity: error\n",[151,5697,5698,5700,5702],{"class":469,"line":3149},[151,5699,5679],{"class":473},[151,5701,5692],{"class":481},[151,5703,5704],{"class":503}," [INFO] [TRT-LLM] [I] Starting TensorRT-LLM init.\n",[151,5706,5707],{"class":469,"line":3158},[151,5708,5709],{"class":503},"[TensorRT-LLM][INFO] Set logger level by INFO\n",[151,5711,5712,5714,5717],{"class":469,"line":3167},[151,5713,5679],{"class":473},[151,5715,5716],{"class":481}," 21:46:26,777",[151,5718,5719],{"class":503}," [INFO] [TRT-LLM] [I] TensorRT-LLM inited.\n",[151,5721,5722],{"class":469,"line":3175},[151,5723,5724],{"class":503},"[TensorRT-LLM] TensorRT-LLM version: 0.10.1.dev2024053000\n",[151,5726,5727,5730,5733,5735,5738,5741,5744,5747,5750,5753,5756,5759,5762,5765,5768,5771],{"class":469,"line":3184},[151,5728,5729],{"class":503},"{",[151,5731,5732],{"class":473},"\"level\"",[151,5734,208],{"class":2226},[151,5736,5737],{"class":481}," \"INFO\",",[151,5739,5740],{"class":481}," \"time\":",[151,5742,5743],{"class":481}," \"02-20 21:46:27.896\",",[151,5745,5746],{"class":481}," \"file_path\":",[151,5748,5749],{"class":481}," \"/usr/local/lib/python3.10/dist-packages/vllm_nvext/entrypoints/openai/api_server.py\",",[151,5751,5752],{"class":481}," \"line_number\":",[151,5754,5755],{"class":481}," \"489\",",[151,5757,5758],{"class":481}," \"message\":",[151,5760,5761],{"class":481}," \"NIM LLM API version 1.0.0\",",[151,5763,5764],{"class":481}," \"exc_info\":",[151,5766,5767],{"class":481}," \"None\",",[151,5769,5770],{"class":481}," \"stack_info\":",[151,5772,5773],{"class":481}," \"None\"}\n",[151,5775,5776,5778,5780,5782,5784,5786,5789,5791,5794,5796,5799,5801,5804,5806,5808,5810],{"class":469,"line":3193},[151,5777,5729],{"class":503},[151,5779,5732],{"class":473},[151,5781,208],{"class":2226},[151,5783,5737],{"class":481},[151,5785,5740],{"class":481},[151,5787,5788],{"class":481}," \"02-20 21:46:27.898\",",[151,5790,5746],{"class":481},[151,5792,5793],{"class":481}," \"/usr/local/lib/python3.10/dist-packages/vllm_nvext/hub/ngc_profile.py\",",[151,5795,5752],{"class":481},[151,5797,5798],{"class":481}," \"217\",",[151,5800,5758],{"class":481},[151,5802,5803],{"class":481}," \"Running NIM without LoRA. Only looking for compatible profiles that do not support LoRA.\",",[151,5805,5764],{"class":481},[151,5807,5767],{"class":481},[151,5809,5770],{"class":481},[151,5811,5773],{"class":481},[151,5813,5814,5816,5818,5820,5822,5824,5826,5828,5830,5832,5835,5837,5840,5842,5844,5846],{"class":469,"line":3720},[151,5815,5729],{"class":503},[151,5817,5732],{"class":473},[151,5819,208],{"class":2226},[151,5821,5737],{"class":481},[151,5823,5740],{"class":481},[151,5825,5788],{"class":481},[151,5827,5746],{"class":481},[151,5829,5793],{"class":481},[151,5831,5752],{"class":481},[151,5833,5834],{"class":481}," \"219\",",[151,5836,5758],{"class":481},[151,5838,5839],{"class":481}," \"Detected 1 compatible profile(s).\",",[151,5841,5764],{"class":481},[151,5843,5767],{"class":481},[151,5845,5770],{"class":481},[151,5847,5773],{"class":481},[151,5849,5850,5852,5854,5856,5858,5860,5862,5864,5867,5869,5872,5874,5877,5879,5881,5883],{"class":469,"line":3729},[151,5851,5729],{"class":503},[151,5853,5732],{"class":473},[151,5855,208],{"class":2226},[151,5857,5737],{"class":481},[151,5859,5740],{"class":481},[151,5861,5788],{"class":481},[151,5863,5746],{"class":481},[151,5865,5866],{"class":481}," \"/usr/local/lib/python3.10/dist-packages/vllm_nvext/hub/ngc_injector.py\",",[151,5868,5752],{"class":481},[151,5870,5871],{"class":481}," \"106\",",[151,5873,5758],{"class":481},[151,5875,5876],{"class":481}," \"Valid profile: 8835c31752fbc67ef658b20a9f78e056914fdef0660206d82f252d62fd96064d (vllm-fp16-tp1) on GPUs [0]\",",[151,5878,5764],{"class":481},[151,5880,5767],{"class":481},[151,5882,5770],{"class":481},[151,5884,5773],{"class":481},[151,5886,5887,5889,5891,5893,5895,5897,5899,5901,5903,5905,5908,5910,5913,5915,5917,5919],{"class":469,"line":3735},[151,5888,5729],{"class":503},[151,5890,5732],{"class":473},[151,5892,208],{"class":2226},[151,5894,5737],{"class":481},[151,5896,5740],{"class":481},[151,5898,5788],{"class":481},[151,5900,5746],{"class":481},[151,5902,5866],{"class":481},[151,5904,5752],{"class":481},[151,5906,5907],{"class":481}," \"141\",",[151,5909,5758],{"class":481},[151,5911,5912],{"class":481}," \"Selected profile: 8835c31752fbc67ef658b20a9f78e056914fdef0660206d82f252d62fd96064d (vllm-fp16-tp1)\",",[151,5914,5764],{"class":481},[151,5916,5767],{"class":481},[151,5918,5770],{"class":481},[151,5920,5773],{"class":481},[151,5922,5923,5925,5927,5929,5931,5933,5936,5938,5940,5942,5945,5947,5950,5952,5954,5956],{"class":469,"line":3745},[151,5924,5729],{"class":503},[151,5926,5732],{"class":473},[151,5928,208],{"class":2226},[151,5930,5737],{"class":481},[151,5932,5740],{"class":481},[151,5934,5935],{"class":481}," \"02-20 21:46:28.585\",",[151,5937,5746],{"class":481},[151,5939,5866],{"class":481},[151,5941,5752],{"class":481},[151,5943,5944],{"class":481}," \"146\",",[151,5946,5758],{"class":481},[151,5948,5949],{"class":481}," \"Profile metadata: tp: 1\",",[151,5951,5764],{"class":481},[151,5953,5767],{"class":481},[151,5955,5770],{"class":481},[151,5957,5773],{"class":481},[151,5959,5960,5962,5964,5966,5968,5970,5973,5975,5977,5979,5981,5983,5986,5988,5990,5992],{"class":469,"line":3754},[151,5961,5729],{"class":503},[151,5963,5732],{"class":473},[151,5965,208],{"class":2226},[151,5967,5737],{"class":481},[151,5969,5740],{"class":481},[151,5971,5972],{"class":481}," \"02-20 21:46:28.586\",",[151,5974,5746],{"class":481},[151,5976,5866],{"class":481},[151,5978,5752],{"class":481},[151,5980,5944],{"class":481},[151,5982,5758],{"class":481},[151,5984,5985],{"class":481}," \"Profile metadata: llm_engine: vllm\",",[151,5987,5764],{"class":481},[151,5989,5767],{"class":481},[151,5991,5770],{"class":481},[151,5993,5773],{"class":481},[151,5995,5996,5998,6000,6002,6004,6006,6008,6010,6012,6014,6016,6018,6021,6023,6025,6027],{"class":469,"line":3760},[151,5997,5729],{"class":503},[151,5999,5732],{"class":473},[151,6001,208],{"class":2226},[151,6003,5737],{"class":481},[151,6005,5740],{"class":481},[151,6007,5972],{"class":481},[151,6009,5746],{"class":481},[151,6011,5866],{"class":481},[151,6013,5752],{"class":481},[151,6015,5944],{"class":481},[151,6017,5758],{"class":481},[151,6019,6020],{"class":481}," \"Profile metadata: precision: fp16\",",[151,6022,5764],{"class":481},[151,6024,5767],{"class":481},[151,6026,5770],{"class":481},[151,6028,5773],{"class":481},[151,6030,6031,6033,6035,6037,6039,6041,6043,6045,6047,6049,6051,6053,6056,6058,6060,6062],{"class":469,"line":3773},[151,6032,5729],{"class":503},[151,6034,5732],{"class":473},[151,6036,208],{"class":2226},[151,6038,5737],{"class":481},[151,6040,5740],{"class":481},[151,6042,5972],{"class":481},[151,6044,5746],{"class":481},[151,6046,5866],{"class":481},[151,6048,5752],{"class":481},[151,6050,5944],{"class":481},[151,6052,5758],{"class":481},[151,6054,6055],{"class":481}," \"Profile metadata: feat_lora: false\",",[151,6057,5764],{"class":481},[151,6059,5767],{"class":481},[151,6061,5770],{"class":481},[151,6063,5773],{"class":481},[151,6065,6066,6068,6070,6072,6074,6076,6078,6080,6082,6084,6087,6089,6092,6094,6096,6098],{"class":469,"line":3782},[151,6067,5729],{"class":503},[151,6069,5732],{"class":473},[151,6071,208],{"class":2226},[151,6073,5737],{"class":481},[151,6075,5740],{"class":481},[151,6077,5972],{"class":481},[151,6079,5746],{"class":481},[151,6081,5866],{"class":481},[151,6083,5752],{"class":481},[151,6085,6086],{"class":481}," \"166\",",[151,6088,5758],{"class":481},[151,6090,6091],{"class":481}," \"Preparing model workspace. This step might download additional files to run the model.\",",[151,6093,5764],{"class":481},[151,6095,5767],{"class":481},[151,6097,5770],{"class":481},[151,6099,5773],{"class":481},[151,6101,6102,6104,6106,6108,6110,6112,6115,6117,6119,6121,6124,6126,6129,6131,6133,6135],{"class":469,"line":3791},[151,6103,5729],{"class":503},[151,6105,5732],{"class":473},[151,6107,208],{"class":2226},[151,6109,5737],{"class":481},[151,6111,5740],{"class":481},[151,6113,6114],{"class":481}," \"02-20 21:47:14.944\",",[151,6116,5746],{"class":481},[151,6118,5866],{"class":481},[151,6120,5752],{"class":481},[151,6122,6123],{"class":481}," \"172\",",[151,6125,5758],{"class":481},[151,6127,6128],{"class":481}," \"Model workspace is now ready. It took 46.358 seconds\",",[151,6130,5764],{"class":481},[151,6132,5767],{"class":481},[151,6134,5770],{"class":481},[151,6136,5773],{"class":481},[151,6138,6139,6141,6143,6145,6147,6149,6152,6154,6157,6159,6162,6164],{"class":469,"line":3803},[151,6140,5729],{"class":503},[151,6142,5732],{"class":473},[151,6144,208],{"class":2226},[151,6146,5737],{"class":481},[151,6148,5740],{"class":481},[151,6150,6151],{"class":481}," \"02-20 21:47:14.949\",",[151,6153,5746],{"class":481},[151,6155,6156],{"class":481}," \"/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py\",",[151,6158,5752],{"class":481},[151,6160,6161],{"class":481}," \"98\",",[151,6163,5758],{"class":481},[151,6165,6166],{"class":481}," \"Initializing an LLM engine (v0.4.1) with config: model='/tmp/meta--llama3-8b-instruct-tbtjj6ij', speculative_config=None,\n",[11,6168,6169],{},"Under the hood, Cloud Assist is calling:",[459,6171,6173],{"className":461,"code":6172,"language":463,"meta":464,"style":464},"kubectl logs my-nim-nim-llm-0 -n nim\n",[30,6174,6175],{"__ignoreMap":464},[151,6176,6177,6179,6182,6185,6187],{"class":469,"line":470},[151,6178,4624],{"class":473},[151,6180,6181],{"class":481}," logs",[151,6183,6184],{"class":481}," my-nim-nim-llm-0",[151,6186,4696],{"class":477},[151,6188,4632],{"class":481},[11,6190,6191],{},"Running this again once the aWe can see some health checks and usage stats in the logs:",[459,6193,6197],{"className":6194,"code":6195,"language":6196,"meta":464,"style":464},"language-json shiki shiki-themes github-light github-dark monokai","{\"level\": \"INFO\", \"time\": \"02-20 21:54:59.370\", \"file_path\": \"/usr/local/lib/python3.10/dist-packages/vllm/engine/metrics.py\", \"line_number\": \"334\", \"message\": \"Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%\", \"exc_info\": \"None\", \"stack_info\": \"None\"}\n{\"level\": \"INFO\", \"time\": \"02-20 21:55:06.801\", \"file_path\": \"/usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/httptools_impl.py\", \"line_number\": \"481\", \"message\": \"10.28.1.1:49414 - \\\"GET /v1/health/live HTTP/1.1\\\" 200\", \"exc_info\": \"None\", \"stack_info\": \"None\"}\n{\"level\": \"INFO\", \"time\": \"02-20 21:55:06.804\", \"file_path\": \"/usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/httptools_impl.py\", \"line_number\": \"481\", \"message\": \"10.28.1.1:49420 - \\\"GET /v1/health/ready HTTP/1.1\\\" 200\", \"exc_info\": \"None\", \"stack_info\": \"None\"}\n","json",[30,6198,6199,6275,6350],{"__ignoreMap":464},[151,6200,6201,6203,6206,6209,6213,6215,6218,6220,6223,6225,6228,6230,6233,6235,6238,6240,6243,6245,6248,6250,6253,6255,6258,6260,6263,6265,6268,6270,6272],{"class":469,"line":470},[151,6202,5729],{"class":503},[151,6204,5732],{"class":6205},"s-m8C",[151,6207,6208],{"class":503},": ",[151,6210,6212],{"class":6211},"sCZoN","\"INFO\"",[151,6214,106],{"class":503},[151,6216,6217],{"class":6205},"\"time\"",[151,6219,6208],{"class":503},[151,6221,6222],{"class":6211},"\"02-20 21:54:59.370\"",[151,6224,106],{"class":503},[151,6226,6227],{"class":6205},"\"file_path\"",[151,6229,6208],{"class":503},[151,6231,6232],{"class":6211},"\"/usr/local/lib/python3.10/dist-packages/vllm/engine/metrics.py\"",[151,6234,106],{"class":503},[151,6236,6237],{"class":6205},"\"line_number\"",[151,6239,6208],{"class":503},[151,6241,6242],{"class":6211},"\"334\"",[151,6244,106],{"class":503},[151,6246,6247],{"class":6205},"\"message\"",[151,6249,6208],{"class":503},[151,6251,6252],{"class":6211},"\"Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%\"",[151,6254,106],{"class":503},[151,6256,6257],{"class":6205},"\"exc_info\"",[151,6259,6208],{"class":503},[151,6261,6262],{"class":6211},"\"None\"",[151,6264,106],{"class":503},[151,6266,6267],{"class":6205},"\"stack_info\"",[151,6269,6208],{"class":503},[151,6271,6262],{"class":6211},[151,6273,6274],{"class":503},"}\n",[151,6276,6277,6279,6281,6283,6285,6287,6289,6291,6294,6296,6298,6300,6303,6305,6307,6309,6312,6314,6316,6318,6321,6324,6327,6329,6332,6334,6336,6338,6340,6342,6344,6346,6348],{"class":469,"line":488},[151,6278,5729],{"class":503},[151,6280,5732],{"class":6205},[151,6282,6208],{"class":503},[151,6284,6212],{"class":6211},[151,6286,106],{"class":503},[151,6288,6217],{"class":6205},[151,6290,6208],{"class":503},[151,6292,6293],{"class":6211},"\"02-20 21:55:06.801\"",[151,6295,106],{"class":503},[151,6297,6227],{"class":6205},[151,6299,6208],{"class":503},[151,6301,6302],{"class":6211},"\"/usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/httptools_impl.py\"",[151,6304,106],{"class":503},[151,6306,6237],{"class":6205},[151,6308,6208],{"class":503},[151,6310,6311],{"class":6211},"\"481\"",[151,6313,106],{"class":503},[151,6315,6247],{"class":6205},[151,6317,6208],{"class":503},[151,6319,6320],{"class":6211},"\"10.28.1.1:49414 - ",[151,6322,6323],{"class":477},"\\\"",[151,6325,6326],{"class":6211},"GET /v1/health/live HTTP/1.1",[151,6328,6323],{"class":477},[151,6330,6331],{"class":6211}," 200\"",[151,6333,106],{"class":503},[151,6335,6257],{"class":6205},[151,6337,6208],{"class":503},[151,6339,6262],{"class":6211},[151,6341,106],{"class":503},[151,6343,6267],{"class":6205},[151,6345,6208],{"class":503},[151,6347,6262],{"class":6211},[151,6349,6274],{"class":503},[151,6351,6352,6354,6356,6358,6360,6362,6364,6366,6369,6371,6373,6375,6377,6379,6381,6383,6385,6387,6389,6391,6394,6396,6399,6401,6403,6405,6407,6409,6411,6413,6415,6417,6419],{"class":469,"line":500},[151,6353,5729],{"class":503},[151,6355,5732],{"class":6205},[151,6357,6208],{"class":503},[151,6359,6212],{"class":6211},[151,6361,106],{"class":503},[151,6363,6217],{"class":6205},[151,6365,6208],{"class":503},[151,6367,6368],{"class":6211},"\"02-20 21:55:06.804\"",[151,6370,106],{"class":503},[151,6372,6227],{"class":6205},[151,6374,6208],{"class":503},[151,6376,6302],{"class":6211},[151,6378,106],{"class":503},[151,6380,6237],{"class":6205},[151,6382,6208],{"class":503},[151,6384,6311],{"class":6211},[151,6386,106],{"class":503},[151,6388,6247],{"class":6205},[151,6390,6208],{"class":503},[151,6392,6393],{"class":6211},"\"10.28.1.1:49420 - ",[151,6395,6323],{"class":477},[151,6397,6398],{"class":6211},"GET /v1/health/ready HTTP/1.1",[151,6400,6323],{"class":477},[151,6402,6331],{"class":6211},[151,6404,106],{"class":503},[151,6406,6257],{"class":6205},[151,6408,6208],{"class":503},[151,6410,6262],{"class":6211},[151,6412,106],{"class":503},[151,6414,6267],{"class":6205},[151,6416,6208],{"class":503},[151,6418,6262],{"class":6211},[151,6420,6274],{"class":503},[11,6422,6423],{},"Actually, there are a few interesting things I found in the logs since it started up, let’s take a look!",[459,6425,6427],{"className":6194,"code":6426,"language":6196,"meta":464,"style":464},"brian@cloudshell:~ (waywo-487618)$ kubectl logs my-nim-nim-llm-0 -n nim\n\n===========================================\n== NVIDIA Inference Microservice LLM NIM ==\n===========================================\n\nNVIDIA Inference Microservice LLM NIM Version 1.0.0\nModel: nim/meta/llama3-8b-instruct\n\nContainer image Copyright (c) 2016-2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.\n\nThis NIM container is governed by the NVIDIA AI Product Agreement here:\nhttps://www.nvidia.com/en-us/data-center/products/nvidia-ai-enterprise/eula/.\nA copy of this license can be found under /opt/nim/LICENSE.\n\nThe use of this model is governed by the AI Foundation Models Community License\nhere: https://docs.nvidia.com/ai-foundation-models-community-license.pdf.\n\nADDITIONAL INFORMATION: Meta Llama 3 Community License, Built with Meta Llama 3.\nA copy of the Llama 3 license can be found under /opt/nim/MODEL_LICENSE.\n\n2026-02-20 21:46:26,056 [INFO] PyTorch version 2.2.2 available.\n2026-02-20 21:46:26,645 [WARNING] [TRT-LLM] [W] Logger level already set from environment. Discard new verbosity: error\n2026-02-20 21:46:26,645 [INFO] [TRT-LLM] [I] Starting TensorRT-LLM init.\n[TensorRT-LLM][INFO] Set logger level by INFO\n2026-02-20 21:46:26,777 [INFO] [TRT-LLM] [I] TensorRT-LLM inited.\n[TensorRT-LLM] TensorRT-LLM version: 0.10.1.dev2024053000\n{\"level\": \"INFO\", \"time\": \"02-20 21:46:27.896\", \"file_path\": \"/usr/local/lib/python3.10/dist-packages/vllm_nvext/entrypoints/openai/api_server.py\", \"line_number\": \"489\", \"message\": \"NIM LLM API version 1.0.0\", \"exc_info\": \"None\", \"stack_info\": \"None\"}\n{\"level\": \"INFO\", \"time\": \"02-20 21:46:27.898\", \"file_path\": \"/usr/local/lib/python3.10/dist-packages/vllm_nvext/hub/ngc_profile.py\", \"line_number\": \"217\", \"message\": \"Running NIM without LoRA. Only looking for compatible profiles that do not support LoRA.\", \"exc_info\": \"None\", \"stack_info\": \"None\"}\n{\"level\": \"INFO\", \"time\": \"02-20 21:46:27.898\", \"file_path\": \"/usr/local/lib/python3.10/dist-packages/vllm_nvext/hub/ngc_profile.py\", \"line_number\": \"219\", \"message\": \"Detected 1 compatible profile(s).\", \"exc_info\": \"None\", \"stack_info\": \"None\"}\n{\"level\": \"INFO\", \"time\": \"02-20 21:46:27.898\", \"file_path\": \"/usr/local/lib/python3.10/dist-packages/vllm_nvext/hub/ngc_injector.py\", \"line_number\": \"106\", \"message\": \"Valid profile: 8835c31752fbc67ef658b20a9f78e056914fdef0660206d82f252d62fd96064d (vllm-fp16-tp1) on GPUs [0]\", \"exc_info\": \"None\", \"stack_info\": \"None\"}\n{\"level\": \"INFO\", \"time\": \"02-20 21:46:27.898\", \"file_path\": \"/usr/local/lib/python3.10/dist-packages/vllm_nvext/hub/ngc_injector.py\", \"line_number\": \"141\", \"message\": \"Selected profile: 8835c31752fbc67ef658b20a9f78e056914fdef0660206d82f252d62fd96064d (vllm-fp16-tp1)\", \"exc_info\": \"None\", \"stack_info\": \"None\"}\n{\"level\": \"INFO\", \"time\": \"02-20 21:46:28.585\", \"file_path\": \"/usr/local/lib/python3.10/dist-packages/vllm_nvext/hub/ngc_injector.py\", \"line_number\": \"146\", \"message\": \"Profile metadata: tp: 1\", \"exc_info\": \"None\", \"stack_info\": \"None\"}\n{\"level\": \"INFO\", \"time\": \"02-20 21:46:28.586\", \"file_path\": \"/usr/local/lib/python3.10/dist-packages/vllm_nvext/hub/ngc_injector.py\", \"line_number\": \"146\", \"message\": \"Profile metadata: llm_engine: vllm\", \"exc_info\": \"None\", \"stack_info\": \"None\"}\n{\"level\": \"INFO\", \"time\": \"02-20 21:46:28.586\", \"file_path\": \"/usr/local/lib/python3.10/dist-packages/vllm_nvext/hub/ngc_injector.py\", \"line_number\": \"146\", \"message\": \"Profile metadata: precision: fp16\", \"exc_info\": \"None\", \"stack_info\": \"None\"}\n{\"level\": \"INFO\", \"time\": \"02-20 21:46:28.586\", \"file_path\": \"/usr/local/lib/python3.10/dist-packages/vllm_nvext/hub/ngc_injector.py\", \"line_number\": \"146\", \"message\": \"Profile metadata: feat_lora: false\", \"exc_info\": \"None\", \"stack_info\": \"None\"}\n{\"level\": \"INFO\", \"time\": \"02-20 21:46:28.586\", \"file_path\": \"/usr/local/lib/python3.10/dist-packages/vllm_nvext/hub/ngc_injector.py\", \"line_number\": \"166\", \"message\": \"Preparing model workspace. This step might download additional files to run the model.\", \"exc_info\": \"None\", \"stack_info\": \"None\"}\n{\"level\": \"INFO\", \"time\": \"02-20 21:47:14.944\", \"file_path\": \"/usr/local/lib/python3.10/dist-packages/vllm_nvext/hub/ngc_injector.py\", \"line_number\": \"172\", \"message\": \"Model workspace is now ready. It took 46.358 seconds\", \"exc_info\": \"None\", \"stack_info\": \"None\"}\n\n{\"level\": \"INFO\", \"time\": \"02-20 21:47:14.949\", \"file_path\": \"/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py\", \"line_number\": \"98\", \"message\": \"Initializing an LLM engine (v0.4.1) with config: model='/tmp/meta--llama3-8b-instruct-tbtjj6ij', speculative_config=None, tokenizer='/tmp/meta--llama3-8b-instruct-tbtjj6ij', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=auto, tensor_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), seed=0)\", \"exc_info\": \"None\", \"stack_info\": \"None\"}\n\n{\"level\": \"WARNING\", \"time\": \"02-20 21:47:15.373\", \"file_path\": \"/usr/local/lib/python3.10/dist-packages/transformers/utils/logging.py\", \"line_number\": \"314\", \"message\": \"Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\", \"exc_info\": \"None\", \"stack_info\": \"None\"}\n\n{\"level\": \"INFO\", \"time\": \"02-20 21:47:15.758\", \"file_path\": \"/usr/local/lib/python3.10/dist-packages/vllm/utils.py\", \"line_number\": \"609\", \"message\": \"Found nccl from library /usr/local/lib/python3.10/dist-packages/nvidia/nccl/lib/libnccl.so.2\", \"exc_info\": \"None\", \"stack_info\": \"None\"}\n\nINFO 02-20 21:47:18 selector.py:28] Using FlashAttention backend.\n\nINFO 02-20 21:47:24 model_runner.py:173] Loading model weights took 14.9595 GB\n\n{\"level\": \"INFO\", \"time\": \"02-20 21:47:27.028\", \"file_path\": \"/usr/local/lib/python3.10/dist-packages/vllm/executor/gpu_executor.py\", \"line_number\": \"119\", \"message\": \"# GPU blocks: 1504, # CPU blocks: 2048\", \"exc_info\": \"None\", \"stack_info\": \"None\"}\n\nINFO 02-20 21:47:29 model_runner.py:973] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.\n\nINFO 02-20 21:47:29 model_runner.py:977] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.\n\nINFO 02-20 21:47:38 model_runner.py:1054] Graph capturing finished in 9 secs.\n\n{\"level\": \"WARNING\", \"time\": \"02-20 21:47:38.846\", \"file_path\": \"/usr/local/lib/python3.10/dist-packages/transformers/utils/logging.py\", \"line_number\": \"314\", \"message\": \"Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\", \"exc_info\": \"None\", \"stack_info\": \"None\"}\n\n{\"level\": \"INFO\", \"time\": \"02-20 21:47:38.863\", \"file_path\": \"/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/serving_chat.py\", \"line_number\": \"347\", \"message\": \"Using default chat template:\\n{% set loop_messages = messages %}{% for message in loop_messages %}{% set content = '\u003C|start_header_id|>' + message['role'] + '\u003C|end_header_id|>\\n\\n'+ message['content'] | trim + '\u003C|eot_id|>' %}{% if loop.index0 == 0 %}{% set content = bos_token + content %}{% endif %}{{ content }}{% endfor %}{% if add_generation_prompt %}{{ '\u003C|start_header_id|>assistant\u003C|end_header_id|>\\n\\n' }}{% endif %}\", \"exc_info\": \"None\", \"stack_info\": \"None\"}\n{\"level\": \"WARNING\", \"time\": \"02-20 21:47:39.246\", \"file_path\": \"/usr/local/lib/python3.10/dist-packages/transformers/utils/logging.py\", \"line_number\": \"314\", \"message\": \"Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\", \"exc_info\": \"None\", \"stack_info\": \"None\"}\n\n{\"level\": \"INFO\", \"time\": \"02-20 21:47:39.265\", \"file_path\": \"/usr/local/lib/python3.10/dist-packages/vllm_nvext/entrypoints/openai/api_server.py\", \"line_number\": \"456\", \"message\": \"Serving endpoints:\\n  0.0.0.0:8000/openapi.json\\n  0.0.0.0:8000/docs\\n  0.0.0.0:8000/docs/oauth2-redirect\\n  0.0.0.0:8000/metrics\\n  0.0.0.0:8000/v1/health/ready\\n  0.0.0.0:8000/v1/health/live\\n  0.0.0.0:8000/v1/models\\n  0.0.0.0:8000/v1/version\\n  0.0.0.0:8000/v1/chat/completions\\n  0.0.0.0:8000/v1/completions\", \"exc_info\": \"None\", \"stack_info\": \"None\"}\n\n{\"level\": \"INFO\", \"time\": \"02-20 21:47:39.265\", \"file_path\": \"/usr/local/lib/python3.10/dist-packages/vllm_nvext/entrypoints/openai/api_server.py\", \"line_number\": \"460\", \"message\": \"An example cURL request:\\ncurl -X 'POST' \\\\\\n  'http://0.0.0.0:8000/v1/chat/completions' \\\\\\n  -H 'accept: application/json' \\\\\\n  -H 'Content-Type: application/json' \\\\\\n  -d '{\\n    \\\"model\\\": \\\"meta/llama3-8b-instruct\\\",\\n    \\\"messages\\\": [\\n      {\\n        \\\"role\\\":\\\"user\\\",\\n        \\\"content\\\":\\\"Hello! How are you?\\\"\\n      },\\n      {\\n        \\\"role\\\":\\\"assistant\\\",\\n        \\\"content\\\":\\\"Hi! I am quite well, how can I help you today?\\\"\\n      },\\n      {\\n        \\\"role\\\":\\\"user\\\",\\n        \\\"content\\\":\\\"Can you write me a song?\\\"\\n      }\\n    ],\\n    \\\"top_p\\\": 1,\\n    \\\"n\\\": 1,\\n    \\\"max_tokens\\\": 15,\\n    \\\"stream\\\": true,\\n    \\\"frequency_penalty\\\": 1.0,\\n    \\\"stop\\\": [\\\"hello\\\"]\\n  }'\\n\", \"exc_info\": \"None\", \"stack_info\": \"None\"}\n{\"level\": \"INFO\", \"time\": \"02-20 21:47:39.336\", \"file_path\": \"/usr/local/lib/python3.10/dist-packages/uvicorn/server.py\", \"line_number\": \"82\", \"message\": \"Started server process [32]\", \"exc_info\": \"None\", \"stack_info\": \"None\"}\n{\"level\": \"INFO\", \"time\": \"02-20 21:47:39.336\", \"file_path\": \"/usr/local/lib/python3.10/dist-packages/uvicorn/lifespan/on.py\", \"line_number\": \"48\", \"message\": \"Waiting for application startup.\", \"exc_info\": \"None\", \"stack_info\": \"None\"}\n{\"level\": \"INFO\", \"time\": \"02-20 21:47:39.338\", \"file_path\": \"/usr/local/lib/python3.10/dist-packages/uvicorn/lifespan/on.py\", \"line_number\": \"62\", \"message\": \"Application startup complete.\", \"exc_info\": \"None\", \"stack_info\": \"None\"}\n{\"level\": \"INFO\", \"time\": \"02-20 21:47:39.340\", \"file_path\": \"/usr/local/lib/python3.10/dist-packages/uvicorn/server.py\", \"line_number\": \"214\", \"message\": \"Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)\", \"exc_info\": \"None\", \"stack_info\": \"None\"}\n\n// the rest of the logs are just health checks and stats\n\n// health check\n{\"level\": \"INFO\", \"time\": \"02-20 21:54:56.805\", \"file_path\": \"/usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/httptools_impl.py\", \"line_number\": \"481\", \"message\": \"10.28.1.1:35412 - \\\"GET /v1/health/ready HTTP/1.1\\\" 200\", \"exc_info\": \"None\", \"stack_info\": \"None\"}\n// stats\n{\"level\": \"INFO\", \"time\": \"02-20 21:54:59.370\", \"file_path\": \"/usr/local/lib/python3.10/dist-packages/vllm/engine/metrics.py\", \"line_number\": \"334\", \"message\": \"Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%\", \"exc_info\": \"None\", \"stack_info\": \"None\"}\n",[30,6428,6429,6446,6450,6455,6460,6464,6468,6481,6492,6496,6507,6511,6516,6524,6529,6533,6538,6546,6550,6566,6576,6580,6623,6661,6694,6710,6744,6767,6831,6895,6957,7020,7082,7146,7209,7271,7333,7396,7460,7465,7530,7535,7601,7606,7671,7676,7706,7711,7744,7749,7814,7819,7845,7850,7885,7890,7922,7927,7989,7994,8076,8138,8143,8257,8262,8611,8676,8740,8804,8868,8873,8879,8884,8890,8961,8967],{"__ignoreMap":464},[151,6430,6431,6434,6437,6440,6443],{"class":469,"line":470},[151,6432,6433],{"class":503},"brian@cloudshell:~ (waywo",[151,6435,6436],{"class":477},"-487618",[151,6438,6439],{"class":503},")$ kubectl logs my-nim-nim-llm",[151,6441,6442],{"class":477},"-0",[151,6444,6445],{"class":503}," -n nim\n",[151,6447,6448],{"class":469,"line":488},[151,6449,1090],{"emptyLinePlaceholder":609},[151,6451,6452],{"class":469,"line":500},[151,6453,6454],{"class":503},"===========================================\n",[151,6456,6457],{"class":469,"line":509},[151,6458,6459],{"class":503},"== NVIDIA Inference Microservice LLM NIM ==\n",[151,6461,6462],{"class":469,"line":517},[151,6463,6454],{"class":503},[151,6465,6466],{"class":469,"line":534},[151,6467,1090],{"emptyLinePlaceholder":609},[151,6469,6470,6473,6476,6478],{"class":469,"line":1413},[151,6471,6472],{"class":503},"NVIDIA Inference Microservice LLM NIM Version ",[151,6474,6475],{"class":477},"1.0",[151,6477,643],{"class":503},[151,6479,6480],{"class":477},"0\n",[151,6482,6483,6486,6489],{"class":469,"line":1418},[151,6484,6485],{"class":503},"Model: nim/meta/llama",[151,6487,6488],{"class":477},"3-8",[151,6490,6491],{"class":503},"b-instruct\n",[151,6493,6494],{"class":469,"line":2462},[151,6495,1090],{"emptyLinePlaceholder":609},[151,6497,6498,6501,6504],{"class":469,"line":2471},[151,6499,6500],{"class":503},"Container image Copyright (c) ",[151,6502,6503],{"class":477},"2016-2024",[151,6505,6506],{"class":503},", NVIDIA CORPORATION & AFFILIATES. All rights reserved.\n",[151,6508,6509],{"class":469,"line":2480},[151,6510,1090],{"emptyLinePlaceholder":609},[151,6512,6513],{"class":469,"line":2489},[151,6514,6515],{"class":503},"This NIM container is governed by the NVIDIA AI Product Agreement here:\n",[151,6517,6518,6521],{"class":469,"line":2497},[151,6519,6520],{"class":503},"https:",[151,6522,6523],{"class":1527},"//www.nvidia.com/en-us/data-center/products/nvidia-ai-enterprise/eula/.\n",[151,6525,6526],{"class":469,"line":3140},[151,6527,6528],{"class":503},"A copy of this license can be found under /opt/nim/LICENSE.\n",[151,6530,6531],{"class":469,"line":3149},[151,6532,1090],{"emptyLinePlaceholder":609},[151,6534,6535],{"class":469,"line":3158},[151,6536,6537],{"class":503},"The use of this model is governed by the AI Foundation Models Community License\n",[151,6539,6540,6543],{"class":469,"line":3167},[151,6541,6542],{"class":503},"here: https:",[151,6544,6545],{"class":1527},"//docs.nvidia.com/ai-foundation-models-community-license.pdf.\n",[151,6547,6548],{"class":469,"line":3175},[151,6549,1090],{"emptyLinePlaceholder":609},[151,6551,6552,6555,6558,6561,6563],{"class":469,"line":3184},[151,6553,6554],{"class":503},"ADDITIONAL INFORMATION: Meta Llama ",[151,6556,6557],{"class":477},"3",[151,6559,6560],{"class":503}," Community License, Built with Meta Llama ",[151,6562,6557],{"class":477},[151,6564,6565],{"class":503},".\n",[151,6567,6568,6571,6573],{"class":469,"line":3193},[151,6569,6570],{"class":503},"A copy of the Llama ",[151,6572,6557],{"class":477},[151,6574,6575],{"class":503}," license can be found under /opt/nim/MODEL_LICENSE.\n",[151,6577,6578],{"class":469,"line":3720},[151,6579,1090],{"emptyLinePlaceholder":609},[151,6581,6582,6584,6587,6589,6592,6594,6597,6599,6602,6605,6609,6612,6615,6617,6620],{"class":469,"line":3729},[151,6583,5679],{"class":477},[151,6585,6586],{"class":477}," 21",[151,6588,208],{"class":503},[151,6590,6591],{"class":477},"46",[151,6593,208],{"class":503},[151,6595,6596],{"class":477},"26",[151,6598,3634],{"class":503},[151,6600,6601],{"class":477},"056",[151,6603,6604],{"class":503}," [",[151,6606,6608],{"class":6607},"st05x","INFO",[151,6610,6611],{"class":503},"] PyTorch version ",[151,6613,6614],{"class":477},"2.2",[151,6616,643],{"class":503},[151,6618,6619],{"class":477},"2",[151,6621,6622],{"class":503}," available.\n",[151,6624,6625,6627,6629,6631,6633,6635,6637,6639,6642,6644,6647,6650,6653,6655,6658],{"class":469,"line":3735},[151,6626,5679],{"class":477},[151,6628,6586],{"class":477},[151,6630,208],{"class":503},[151,6632,6591],{"class":477},[151,6634,208],{"class":503},[151,6636,6596],{"class":477},[151,6638,3634],{"class":503},[151,6640,6641],{"class":477},"645",[151,6643,6604],{"class":503},[151,6645,6646],{"class":6607},"WARNING",[151,6648,6649],{"class":503},"] [",[151,6651,6652],{"class":6607},"TRT-LLM",[151,6654,6649],{"class":503},[151,6656,6657],{"class":6607},"W",[151,6659,6660],{"class":503},"] Logger level already set from environment. Discard new verbosity: error\n",[151,6662,6663,6665,6667,6669,6671,6673,6675,6677,6679,6681,6683,6685,6687,6689,6691],{"class":469,"line":3745},[151,6664,5679],{"class":477},[151,6666,6586],{"class":477},[151,6668,208],{"class":503},[151,6670,6591],{"class":477},[151,6672,208],{"class":503},[151,6674,6596],{"class":477},[151,6676,3634],{"class":503},[151,6678,6641],{"class":477},[151,6680,6604],{"class":503},[151,6682,6608],{"class":6607},[151,6684,6649],{"class":503},[151,6686,6652],{"class":6607},[151,6688,6649],{"class":503},[151,6690,279],{"class":6607},[151,6692,6693],{"class":503},"] Starting TensorRT-LLM init.\n",[151,6695,6696,6699,6702,6705,6707],{"class":469,"line":3754},[151,6697,6698],{"class":503},"[",[151,6700,6701],{"class":6607},"TensorRT-LLM",[151,6703,6704],{"class":503},"][",[151,6706,6608],{"class":6607},[151,6708,6709],{"class":503},"] Set logger level by INFO\n",[151,6711,6712,6714,6716,6718,6720,6722,6724,6726,6729,6731,6733,6735,6737,6739,6741],{"class":469,"line":3760},[151,6713,5679],{"class":477},[151,6715,6586],{"class":477},[151,6717,208],{"class":503},[151,6719,6591],{"class":477},[151,6721,208],{"class":503},[151,6723,6596],{"class":477},[151,6725,3634],{"class":503},[151,6727,6728],{"class":477},"777",[151,6730,6604],{"class":503},[151,6732,6608],{"class":6607},[151,6734,6649],{"class":503},[151,6736,6652],{"class":6607},[151,6738,6649],{"class":503},[151,6740,279],{"class":6607},[151,6742,6743],{"class":503},"] TensorRT-LLM inited.\n",[151,6745,6746,6748,6750,6753,6756,6758,6761,6764],{"class":469,"line":3773},[151,6747,6698],{"class":503},[151,6749,6701],{"class":6607},[151,6751,6752],{"class":503},"] TensorRT-LLM version: ",[151,6754,6755],{"class":477},"0.10",[151,6757,643],{"class":503},[151,6759,6760],{"class":477},"1",[151,6762,6763],{"class":503},".dev",[151,6765,6766],{"class":477},"2024053000\n",[151,6768,6769,6771,6773,6775,6777,6779,6781,6783,6786,6788,6790,6792,6795,6797,6799,6801,6804,6806,6808,6810,6813,6815,6817,6819,6821,6823,6825,6827,6829],{"class":469,"line":3782},[151,6770,5729],{"class":503},[151,6772,5732],{"class":6205},[151,6774,6208],{"class":503},[151,6776,6212],{"class":6211},[151,6778,106],{"class":503},[151,6780,6217],{"class":6205},[151,6782,6208],{"class":503},[151,6784,6785],{"class":6211},"\"02-20 21:46:27.896\"",[151,6787,106],{"class":503},[151,6789,6227],{"class":6205},[151,6791,6208],{"class":503},[151,6793,6794],{"class":6211},"\"/usr/local/lib/python3.10/dist-packages/vllm_nvext/entrypoints/openai/api_server.py\"",[151,6796,106],{"class":503},[151,6798,6237],{"class":6205},[151,6800,6208],{"class":503},[151,6802,6803],{"class":6211},"\"489\"",[151,6805,106],{"class":503},[151,6807,6247],{"class":6205},[151,6809,6208],{"class":503},[151,6811,6812],{"class":6211},"\"NIM LLM API version 1.0.0\"",[151,6814,106],{"class":503},[151,6816,6257],{"class":6205},[151,6818,6208],{"class":503},[151,6820,6262],{"class":6211},[151,6822,106],{"class":503},[151,6824,6267],{"class":6205},[151,6826,6208],{"class":503},[151,6828,6262],{"class":6211},[151,6830,6274],{"class":503},[151,6832,6833,6835,6837,6839,6841,6843,6845,6847,6850,6852,6854,6856,6859,6861,6863,6865,6868,6870,6872,6874,6877,6879,6881,6883,6885,6887,6889,6891,6893],{"class":469,"line":3791},[151,6834,5729],{"class":503},[151,6836,5732],{"class":6205},[151,6838,6208],{"class":503},[151,6840,6212],{"class":6211},[151,6842,106],{"class":503},[151,6844,6217],{"class":6205},[151,6846,6208],{"class":503},[151,6848,6849],{"class":6211},"\"02-20 21:46:27.898\"",[151,6851,106],{"class":503},[151,6853,6227],{"class":6205},[151,6855,6208],{"class":503},[151,6857,6858],{"class":6211},"\"/usr/local/lib/python3.10/dist-packages/vllm_nvext/hub/ngc_profile.py\"",[151,6860,106],{"class":503},[151,6862,6237],{"class":6205},[151,6864,6208],{"class":503},[151,6866,6867],{"class":6211},"\"217\"",[151,6869,106],{"class":503},[151,6871,6247],{"class":6205},[151,6873,6208],{"class":503},[151,6875,6876],{"class":6211},"\"Running NIM without LoRA. Only looking for compatible profiles that do not support LoRA.\"",[151,6878,106],{"class":503},[151,6880,6257],{"class":6205},[151,6882,6208],{"class":503},[151,6884,6262],{"class":6211},[151,6886,106],{"class":503},[151,6888,6267],{"class":6205},[151,6890,6208],{"class":503},[151,6892,6262],{"class":6211},[151,6894,6274],{"class":503},[151,6896,6897,6899,6901,6903,6905,6907,6909,6911,6913,6915,6917,6919,6921,6923,6925,6927,6930,6932,6934,6936,6939,6941,6943,6945,6947,6949,6951,6953,6955],{"class":469,"line":3803},[151,6898,5729],{"class":503},[151,6900,5732],{"class":6205},[151,6902,6208],{"class":503},[151,6904,6212],{"class":6211},[151,6906,106],{"class":503},[151,6908,6217],{"class":6205},[151,6910,6208],{"class":503},[151,6912,6849],{"class":6211},[151,6914,106],{"class":503},[151,6916,6227],{"class":6205},[151,6918,6208],{"class":503},[151,6920,6858],{"class":6211},[151,6922,106],{"class":503},[151,6924,6237],{"class":6205},[151,6926,6208],{"class":503},[151,6928,6929],{"class":6211},"\"219\"",[151,6931,106],{"class":503},[151,6933,6247],{"class":6205},[151,6935,6208],{"class":503},[151,6937,6938],{"class":6211},"\"Detected 1 compatible profile(s).\"",[151,6940,106],{"class":503},[151,6942,6257],{"class":6205},[151,6944,6208],{"class":503},[151,6946,6262],{"class":6211},[151,6948,106],{"class":503},[151,6950,6267],{"class":6205},[151,6952,6208],{"class":503},[151,6954,6262],{"class":6211},[151,6956,6274],{"class":503},[151,6958,6959,6961,6963,6965,6967,6969,6971,6973,6975,6977,6979,6981,6984,6986,6988,6990,6993,6995,6997,6999,7002,7004,7006,7008,7010,7012,7014,7016,7018],{"class":469,"line":3811},[151,6960,5729],{"class":503},[151,6962,5732],{"class":6205},[151,6964,6208],{"class":503},[151,6966,6212],{"class":6211},[151,6968,106],{"class":503},[151,6970,6217],{"class":6205},[151,6972,6208],{"class":503},[151,6974,6849],{"class":6211},[151,6976,106],{"class":503},[151,6978,6227],{"class":6205},[151,6980,6208],{"class":503},[151,6982,6983],{"class":6211},"\"/usr/local/lib/python3.10/dist-packages/vllm_nvext/hub/ngc_injector.py\"",[151,6985,106],{"class":503},[151,6987,6237],{"class":6205},[151,6989,6208],{"class":503},[151,6991,6992],{"class":6211},"\"106\"",[151,6994,106],{"class":503},[151,6996,6247],{"class":6205},[151,6998,6208],{"class":503},[151,7000,7001],{"class":6211},"\"Valid profile: 8835c31752fbc67ef658b20a9f78e056914fdef0660206d82f252d62fd96064d (vllm-fp16-tp1) on GPUs [0]\"",[151,7003,106],{"class":503},[151,7005,6257],{"class":6205},[151,7007,6208],{"class":503},[151,7009,6262],{"class":6211},[151,7011,106],{"class":503},[151,7013,6267],{"class":6205},[151,7015,6208],{"class":503},[151,7017,6262],{"class":6211},[151,7019,6274],{"class":503},[151,7021,7022,7024,7026,7028,7030,7032,7034,7036,7038,7040,7042,7044,7046,7048,7050,7052,7055,7057,7059,7061,7064,7066,7068,7070,7072,7074,7076,7078,7080],{"class":469,"line":3820},[151,7023,5729],{"class":503},[151,7025,5732],{"class":6205},[151,7027,6208],{"class":503},[151,7029,6212],{"class":6211},[151,7031,106],{"class":503},[151,7033,6217],{"class":6205},[151,7035,6208],{"class":503},[151,7037,6849],{"class":6211},[151,7039,106],{"class":503},[151,7041,6227],{"class":6205},[151,7043,6208],{"class":503},[151,7045,6983],{"class":6211},[151,7047,106],{"class":503},[151,7049,6237],{"class":6205},[151,7051,6208],{"class":503},[151,7053,7054],{"class":6211},"\"141\"",[151,7056,106],{"class":503},[151,7058,6247],{"class":6205},[151,7060,6208],{"class":503},[151,7062,7063],{"class":6211},"\"Selected profile: 8835c31752fbc67ef658b20a9f78e056914fdef0660206d82f252d62fd96064d (vllm-fp16-tp1)\"",[151,7065,106],{"class":503},[151,7067,6257],{"class":6205},[151,7069,6208],{"class":503},[151,7071,6262],{"class":6211},[151,7073,106],{"class":503},[151,7075,6267],{"class":6205},[151,7077,6208],{"class":503},[151,7079,6262],{"class":6211},[151,7081,6274],{"class":503},[151,7083,7085,7087,7089,7091,7093,7095,7097,7099,7102,7104,7106,7108,7110,7112,7114,7116,7119,7121,7123,7125,7128,7130,7132,7134,7136,7138,7140,7142,7144],{"class":469,"line":7084},33,[151,7086,5729],{"class":503},[151,7088,5732],{"class":6205},[151,7090,6208],{"class":503},[151,7092,6212],{"class":6211},[151,7094,106],{"class":503},[151,7096,6217],{"class":6205},[151,7098,6208],{"class":503},[151,7100,7101],{"class":6211},"\"02-20 21:46:28.585\"",[151,7103,106],{"class":503},[151,7105,6227],{"class":6205},[151,7107,6208],{"class":503},[151,7109,6983],{"class":6211},[151,7111,106],{"class":503},[151,7113,6237],{"class":6205},[151,7115,6208],{"class":503},[151,7117,7118],{"class":6211},"\"146\"",[151,7120,106],{"class":503},[151,7122,6247],{"class":6205},[151,7124,6208],{"class":503},[151,7126,7127],{"class":6211},"\"Profile metadata: tp: 1\"",[151,7129,106],{"class":503},[151,7131,6257],{"class":6205},[151,7133,6208],{"class":503},[151,7135,6262],{"class":6211},[151,7137,106],{"class":503},[151,7139,6267],{"class":6205},[151,7141,6208],{"class":503},[151,7143,6262],{"class":6211},[151,7145,6274],{"class":503},[151,7147,7149,7151,7153,7155,7157,7159,7161,7163,7166,7168,7170,7172,7174,7176,7178,7180,7182,7184,7186,7188,7191,7193,7195,7197,7199,7201,7203,7205,7207],{"class":469,"line":7148},34,[151,7150,5729],{"class":503},[151,7152,5732],{"class":6205},[151,7154,6208],{"class":503},[151,7156,6212],{"class":6211},[151,7158,106],{"class":503},[151,7160,6217],{"class":6205},[151,7162,6208],{"class":503},[151,7164,7165],{"class":6211},"\"02-20 21:46:28.586\"",[151,7167,106],{"class":503},[151,7169,6227],{"class":6205},[151,7171,6208],{"class":503},[151,7173,6983],{"class":6211},[151,7175,106],{"class":503},[151,7177,6237],{"class":6205},[151,7179,6208],{"class":503},[151,7181,7118],{"class":6211},[151,7183,106],{"class":503},[151,7185,6247],{"class":6205},[151,7187,6208],{"class":503},[151,7189,7190],{"class":6211},"\"Profile metadata: llm_engine: vllm\"",[151,7192,106],{"class":503},[151,7194,6257],{"class":6205},[151,7196,6208],{"class":503},[151,7198,6262],{"class":6211},[151,7200,106],{"class":503},[151,7202,6267],{"class":6205},[151,7204,6208],{"class":503},[151,7206,6262],{"class":6211},[151,7208,6274],{"class":503},[151,7210,7212,7214,7216,7218,7220,7222,7224,7226,7228,7230,7232,7234,7236,7238,7240,7242,7244,7246,7248,7250,7253,7255,7257,7259,7261,7263,7265,7267,7269],{"class":469,"line":7211},35,[151,7213,5729],{"class":503},[151,7215,5732],{"class":6205},[151,7217,6208],{"class":503},[151,7219,6212],{"class":6211},[151,7221,106],{"class":503},[151,7223,6217],{"class":6205},[151,7225,6208],{"class":503},[151,7227,7165],{"class":6211},[151,7229,106],{"class":503},[151,7231,6227],{"class":6205},[151,7233,6208],{"class":503},[151,7235,6983],{"class":6211},[151,7237,106],{"class":503},[151,7239,6237],{"class":6205},[151,7241,6208],{"class":503},[151,7243,7118],{"class":6211},[151,7245,106],{"class":503},[151,7247,6247],{"class":6205},[151,7249,6208],{"class":503},[151,7251,7252],{"class":6211},"\"Profile metadata: precision: fp16\"",[151,7254,106],{"class":503},[151,7256,6257],{"class":6205},[151,7258,6208],{"class":503},[151,7260,6262],{"class":6211},[151,7262,106],{"class":503},[151,7264,6267],{"class":6205},[151,7266,6208],{"class":503},[151,7268,6262],{"class":6211},[151,7270,6274],{"class":503},[151,7272,7274,7276,7278,7280,7282,7284,7286,7288,7290,7292,7294,7296,7298,7300,7302,7304,7306,7308,7310,7312,7315,7317,7319,7321,7323,7325,7327,7329,7331],{"class":469,"line":7273},36,[151,7275,5729],{"class":503},[151,7277,5732],{"class":6205},[151,7279,6208],{"class":503},[151,7281,6212],{"class":6211},[151,7283,106],{"class":503},[151,7285,6217],{"class":6205},[151,7287,6208],{"class":503},[151,7289,7165],{"class":6211},[151,7291,106],{"class":503},[151,7293,6227],{"class":6205},[151,7295,6208],{"class":503},[151,7297,6983],{"class":6211},[151,7299,106],{"class":503},[151,7301,6237],{"class":6205},[151,7303,6208],{"class":503},[151,7305,7118],{"class":6211},[151,7307,106],{"class":503},[151,7309,6247],{"class":6205},[151,7311,6208],{"class":503},[151,7313,7314],{"class":6211},"\"Profile metadata: feat_lora: false\"",[151,7316,106],{"class":503},[151,7318,6257],{"class":6205},[151,7320,6208],{"class":503},[151,7322,6262],{"class":6211},[151,7324,106],{"class":503},[151,7326,6267],{"class":6205},[151,7328,6208],{"class":503},[151,7330,6262],{"class":6211},[151,7332,6274],{"class":503},[151,7334,7336,7338,7340,7342,7344,7346,7348,7350,7352,7354,7356,7358,7360,7362,7364,7366,7369,7371,7373,7375,7378,7380,7382,7384,7386,7388,7390,7392,7394],{"class":469,"line":7335},37,[151,7337,5729],{"class":503},[151,7339,5732],{"class":6205},[151,7341,6208],{"class":503},[151,7343,6212],{"class":6211},[151,7345,106],{"class":503},[151,7347,6217],{"class":6205},[151,7349,6208],{"class":503},[151,7351,7165],{"class":6211},[151,7353,106],{"class":503},[151,7355,6227],{"class":6205},[151,7357,6208],{"class":503},[151,7359,6983],{"class":6211},[151,7361,106],{"class":503},[151,7363,6237],{"class":6205},[151,7365,6208],{"class":503},[151,7367,7368],{"class":6211},"\"166\"",[151,7370,106],{"class":503},[151,7372,6247],{"class":6205},[151,7374,6208],{"class":503},[151,7376,7377],{"class":6211},"\"Preparing model workspace. This step might download additional files to run the model.\"",[151,7379,106],{"class":503},[151,7381,6257],{"class":6205},[151,7383,6208],{"class":503},[151,7385,6262],{"class":6211},[151,7387,106],{"class":503},[151,7389,6267],{"class":6205},[151,7391,6208],{"class":503},[151,7393,6262],{"class":6211},[151,7395,6274],{"class":503},[151,7397,7399,7401,7403,7405,7407,7409,7411,7413,7416,7418,7420,7422,7424,7426,7428,7430,7433,7435,7437,7439,7442,7444,7446,7448,7450,7452,7454,7456,7458],{"class":469,"line":7398},38,[151,7400,5729],{"class":503},[151,7402,5732],{"class":6205},[151,7404,6208],{"class":503},[151,7406,6212],{"class":6211},[151,7408,106],{"class":503},[151,7410,6217],{"class":6205},[151,7412,6208],{"class":503},[151,7414,7415],{"class":6211},"\"02-20 21:47:14.944\"",[151,7417,106],{"class":503},[151,7419,6227],{"class":6205},[151,7421,6208],{"class":503},[151,7423,6983],{"class":6211},[151,7425,106],{"class":503},[151,7427,6237],{"class":6205},[151,7429,6208],{"class":503},[151,7431,7432],{"class":6211},"\"172\"",[151,7434,106],{"class":503},[151,7436,6247],{"class":6205},[151,7438,6208],{"class":503},[151,7440,7441],{"class":6211},"\"Model workspace is now ready. It took 46.358 seconds\"",[151,7443,106],{"class":503},[151,7445,6257],{"class":6205},[151,7447,6208],{"class":503},[151,7449,6262],{"class":6211},[151,7451,106],{"class":503},[151,7453,6267],{"class":6205},[151,7455,6208],{"class":503},[151,7457,6262],{"class":6211},[151,7459,6274],{"class":503},[151,7461,7463],{"class":469,"line":7462},39,[151,7464,1090],{"emptyLinePlaceholder":609},[151,7466,7468,7470,7472,7474,7476,7478,7480,7482,7485,7487,7489,7491,7494,7496,7498,7500,7503,7505,7507,7509,7512,7514,7516,7518,7520,7522,7524,7526,7528],{"class":469,"line":7467},40,[151,7469,5729],{"class":503},[151,7471,5732],{"class":6205},[151,7473,6208],{"class":503},[151,7475,6212],{"class":6211},[151,7477,106],{"class":503},[151,7479,6217],{"class":6205},[151,7481,6208],{"class":503},[151,7483,7484],{"class":6211},"\"02-20 21:47:14.949\"",[151,7486,106],{"class":503},[151,7488,6227],{"class":6205},[151,7490,6208],{"class":503},[151,7492,7493],{"class":6211},"\"/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py\"",[151,7495,106],{"class":503},[151,7497,6237],{"class":6205},[151,7499,6208],{"class":503},[151,7501,7502],{"class":6211},"\"98\"",[151,7504,106],{"class":503},[151,7506,6247],{"class":6205},[151,7508,6208],{"class":503},[151,7510,7511],{"class":6211},"\"Initializing an LLM engine (v0.4.1) with config: model='/tmp/meta--llama3-8b-instruct-tbtjj6ij', speculative_config=None, tokenizer='/tmp/meta--llama3-8b-instruct-tbtjj6ij', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=auto, tensor_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), seed=0)\"",[151,7513,106],{"class":503},[151,7515,6257],{"class":6205},[151,7517,6208],{"class":503},[151,7519,6262],{"class":6211},[151,7521,106],{"class":503},[151,7523,6267],{"class":6205},[151,7525,6208],{"class":503},[151,7527,6262],{"class":6211},[151,7529,6274],{"class":503},[151,7531,7533],{"class":469,"line":7532},41,[151,7534,1090],{"emptyLinePlaceholder":609},[151,7536,7538,7540,7542,7544,7547,7549,7551,7553,7556,7558,7560,7562,7565,7567,7569,7571,7574,7576,7578,7580,7583,7585,7587,7589,7591,7593,7595,7597,7599],{"class":469,"line":7537},42,[151,7539,5729],{"class":503},[151,7541,5732],{"class":6205},[151,7543,6208],{"class":503},[151,7545,7546],{"class":6211},"\"WARNING\"",[151,7548,106],{"class":503},[151,7550,6217],{"class":6205},[151,7552,6208],{"class":503},[151,7554,7555],{"class":6211},"\"02-20 21:47:15.373\"",[151,7557,106],{"class":503},[151,7559,6227],{"class":6205},[151,7561,6208],{"class":503},[151,7563,7564],{"class":6211},"\"/usr/local/lib/python3.10/dist-packages/transformers/utils/logging.py\"",[151,7566,106],{"class":503},[151,7568,6237],{"class":6205},[151,7570,6208],{"class":503},[151,7572,7573],{"class":6211},"\"314\"",[151,7575,106],{"class":503},[151,7577,6247],{"class":6205},[151,7579,6208],{"class":503},[151,7581,7582],{"class":6211},"\"Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\"",[151,7584,106],{"class":503},[151,7586,6257],{"class":6205},[151,7588,6208],{"class":503},[151,7590,6262],{"class":6211},[151,7592,106],{"class":503},[151,7594,6267],{"class":6205},[151,7596,6208],{"class":503},[151,7598,6262],{"class":6211},[151,7600,6274],{"class":503},[151,7602,7604],{"class":469,"line":7603},43,[151,7605,1090],{"emptyLinePlaceholder":609},[151,7607,7609,7611,7613,7615,7617,7619,7621,7623,7626,7628,7630,7632,7635,7637,7639,7641,7644,7646,7648,7650,7653,7655,7657,7659,7661,7663,7665,7667,7669],{"class":469,"line":7608},44,[151,7610,5729],{"class":503},[151,7612,5732],{"class":6205},[151,7614,6208],{"class":503},[151,7616,6212],{"class":6211},[151,7618,106],{"class":503},[151,7620,6217],{"class":6205},[151,7622,6208],{"class":503},[151,7624,7625],{"class":6211},"\"02-20 21:47:15.758\"",[151,7627,106],{"class":503},[151,7629,6227],{"class":6205},[151,7631,6208],{"class":503},[151,7633,7634],{"class":6211},"\"/usr/local/lib/python3.10/dist-packages/vllm/utils.py\"",[151,7636,106],{"class":503},[151,7638,6237],{"class":6205},[151,7640,6208],{"class":503},[151,7642,7643],{"class":6211},"\"609\"",[151,7645,106],{"class":503},[151,7647,6247],{"class":6205},[151,7649,6208],{"class":503},[151,7651,7652],{"class":6211},"\"Found nccl from library /usr/local/lib/python3.10/dist-packages/nvidia/nccl/lib/libnccl.so.2\"",[151,7654,106],{"class":503},[151,7656,6257],{"class":6205},[151,7658,6208],{"class":503},[151,7660,6262],{"class":6211},[151,7662,106],{"class":503},[151,7664,6267],{"class":6205},[151,7666,6208],{"class":503},[151,7668,6262],{"class":6211},[151,7670,6274],{"class":503},[151,7672,7674],{"class":469,"line":7673},45,[151,7675,1090],{"emptyLinePlaceholder":609},[151,7677,7679,7682,7685,7687,7689,7692,7694,7697,7700,7703],{"class":469,"line":7678},46,[151,7680,7681],{"class":503},"INFO ",[151,7683,7684],{"class":477},"02-20",[151,7686,6586],{"class":477},[151,7688,208],{"class":503},[151,7690,7691],{"class":477},"47",[151,7693,208],{"class":503},[151,7695,7696],{"class":477},"18",[151,7698,7699],{"class":503}," selector.py:",[151,7701,7702],{"class":477},"28",[151,7704,7705],{"class":503},"] Using FlashAttention backend.\n",[151,7707,7709],{"class":469,"line":7708},47,[151,7710,1090],{"emptyLinePlaceholder":609},[151,7712,7714,7716,7718,7720,7722,7724,7726,7729,7732,7735,7738,7741],{"class":469,"line":7713},48,[151,7715,7681],{"class":503},[151,7717,7684],{"class":477},[151,7719,6586],{"class":477},[151,7721,208],{"class":503},[151,7723,7691],{"class":477},[151,7725,208],{"class":503},[151,7727,7728],{"class":477},"24",[151,7730,7731],{"class":503}," model_runner.py:",[151,7733,7734],{"class":477},"173",[151,7736,7737],{"class":503},"] Loading model weights took ",[151,7739,7740],{"class":477},"14.9595",[151,7742,7743],{"class":503}," GB\n",[151,7745,7747],{"class":469,"line":7746},49,[151,7748,1090],{"emptyLinePlaceholder":609},[151,7750,7752,7754,7756,7758,7760,7762,7764,7766,7769,7771,7773,7775,7778,7780,7782,7784,7787,7789,7791,7793,7796,7798,7800,7802,7804,7806,7808,7810,7812],{"class":469,"line":7751},50,[151,7753,5729],{"class":503},[151,7755,5732],{"class":6205},[151,7757,6208],{"class":503},[151,7759,6212],{"class":6211},[151,7761,106],{"class":503},[151,7763,6217],{"class":6205},[151,7765,6208],{"class":503},[151,7767,7768],{"class":6211},"\"02-20 21:47:27.028\"",[151,7770,106],{"class":503},[151,7772,6227],{"class":6205},[151,7774,6208],{"class":503},[151,7776,7777],{"class":6211},"\"/usr/local/lib/python3.10/dist-packages/vllm/executor/gpu_executor.py\"",[151,7779,106],{"class":503},[151,7781,6237],{"class":6205},[151,7783,6208],{"class":503},[151,7785,7786],{"class":6211},"\"119\"",[151,7788,106],{"class":503},[151,7790,6247],{"class":6205},[151,7792,6208],{"class":503},[151,7794,7795],{"class":6211},"\"# GPU blocks: 1504, # CPU blocks: 2048\"",[151,7797,106],{"class":503},[151,7799,6257],{"class":6205},[151,7801,6208],{"class":503},[151,7803,6262],{"class":6211},[151,7805,106],{"class":503},[151,7807,6267],{"class":6205},[151,7809,6208],{"class":503},[151,7811,6262],{"class":6211},[151,7813,6274],{"class":503},[151,7815,7817],{"class":469,"line":7816},51,[151,7818,1090],{"emptyLinePlaceholder":609},[151,7820,7822,7824,7826,7828,7830,7832,7834,7837,7839,7842],{"class":469,"line":7821},52,[151,7823,7681],{"class":503},[151,7825,7684],{"class":477},[151,7827,6586],{"class":477},[151,7829,208],{"class":503},[151,7831,7691],{"class":477},[151,7833,208],{"class":503},[151,7835,7836],{"class":477},"29",[151,7838,7731],{"class":503},[151,7840,7841],{"class":477},"973",[151,7843,7844],{"class":503},"] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.\n",[151,7846,7848],{"class":469,"line":7847},53,[151,7849,1090],{"emptyLinePlaceholder":609},[151,7851,7853,7855,7857,7859,7861,7863,7865,7867,7869,7872,7875,7877,7880,7882],{"class":469,"line":7852},54,[151,7854,7681],{"class":503},[151,7856,7684],{"class":477},[151,7858,6586],{"class":477},[151,7860,208],{"class":503},[151,7862,7691],{"class":477},[151,7864,208],{"class":503},[151,7866,7836],{"class":477},[151,7868,7731],{"class":503},[151,7870,7871],{"class":477},"977",[151,7873,7874],{"class":503},"] CUDA graphs can take additional ",[151,7876,6760],{"class":477},[151,7878,7879],{"class":503},"~",[151,7881,6557],{"class":477},[151,7883,7884],{"class":503}," GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.\n",[151,7886,7888],{"class":469,"line":7887},55,[151,7889,1090],{"emptyLinePlaceholder":609},[151,7891,7893,7895,7897,7899,7901,7903,7905,7908,7910,7913,7916,7919],{"class":469,"line":7892},56,[151,7894,7681],{"class":503},[151,7896,7684],{"class":477},[151,7898,6586],{"class":477},[151,7900,208],{"class":503},[151,7902,7691],{"class":477},[151,7904,208],{"class":503},[151,7906,7907],{"class":477},"38",[151,7909,7731],{"class":503},[151,7911,7912],{"class":477},"1054",[151,7914,7915],{"class":503},"] Graph capturing finished in ",[151,7917,7918],{"class":477},"9",[151,7920,7921],{"class":503}," secs.\n",[151,7923,7925],{"class":469,"line":7924},57,[151,7926,1090],{"emptyLinePlaceholder":609},[151,7928,7930,7932,7934,7936,7938,7940,7942,7944,7947,7949,7951,7953,7955,7957,7959,7961,7963,7965,7967,7969,7971,7973,7975,7977,7979,7981,7983,7985,7987],{"class":469,"line":7929},58,[151,7931,5729],{"class":503},[151,7933,5732],{"class":6205},[151,7935,6208],{"class":503},[151,7937,7546],{"class":6211},[151,7939,106],{"class":503},[151,7941,6217],{"class":6205},[151,7943,6208],{"class":503},[151,7945,7946],{"class":6211},"\"02-20 21:47:38.846\"",[151,7948,106],{"class":503},[151,7950,6227],{"class":6205},[151,7952,6208],{"class":503},[151,7954,7564],{"class":6211},[151,7956,106],{"class":503},[151,7958,6237],{"class":6205},[151,7960,6208],{"class":503},[151,7962,7573],{"class":6211},[151,7964,106],{"class":503},[151,7966,6247],{"class":6205},[151,7968,6208],{"class":503},[151,7970,7582],{"class":6211},[151,7972,106],{"class":503},[151,7974,6257],{"class":6205},[151,7976,6208],{"class":503},[151,7978,6262],{"class":6211},[151,7980,106],{"class":503},[151,7982,6267],{"class":6205},[151,7984,6208],{"class":503},[151,7986,6262],{"class":6211},[151,7988,6274],{"class":503},[151,7990,7992],{"class":469,"line":7991},59,[151,7993,1090],{"emptyLinePlaceholder":609},[151,7995,7997,7999,8001,8003,8005,8007,8009,8011,8014,8016,8018,8020,8023,8025,8027,8029,8032,8034,8036,8038,8041,8044,8047,8050,8053,8055,8058,8060,8062,8064,8066,8068,8070,8072,8074],{"class":469,"line":7996},60,[151,7998,5729],{"class":503},[151,8000,5732],{"class":6205},[151,8002,6208],{"class":503},[151,8004,6212],{"class":6211},[151,8006,106],{"class":503},[151,8008,6217],{"class":6205},[151,8010,6208],{"class":503},[151,8012,8013],{"class":6211},"\"02-20 21:47:38.863\"",[151,8015,106],{"class":503},[151,8017,6227],{"class":6205},[151,8019,6208],{"class":503},[151,8021,8022],{"class":6211},"\"/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/serving_chat.py\"",[151,8024,106],{"class":503},[151,8026,6237],{"class":6205},[151,8028,6208],{"class":503},[151,8030,8031],{"class":6211},"\"347\"",[151,8033,106],{"class":503},[151,8035,6247],{"class":6205},[151,8037,6208],{"class":503},[151,8039,8040],{"class":6211},"\"Using default chat template:",[151,8042,8043],{"class":477},"\\n",[151,8045,8046],{"class":6211},"{% set loop_messages = messages %}{% for message in loop_messages %}{% set content = '\u003C|start_header_id|>' + message['role'] + '\u003C|end_header_id|>",[151,8048,8049],{"class":477},"\\n\\n",[151,8051,8052],{"class":6211},"'+ message['content'] | trim + '\u003C|eot_id|>' %}{% if loop.index0 == 0 %}{% set content = bos_token + content %}{% endif %}{{ content }}{% endfor %}{% if add_generation_prompt %}{{ '\u003C|start_header_id|>assistant\u003C|end_header_id|>",[151,8054,8049],{"class":477},[151,8056,8057],{"class":6211},"' }}{% endif %}\"",[151,8059,106],{"class":503},[151,8061,6257],{"class":6205},[151,8063,6208],{"class":503},[151,8065,6262],{"class":6211},[151,8067,106],{"class":503},[151,8069,6267],{"class":6205},[151,8071,6208],{"class":503},[151,8073,6262],{"class":6211},[151,8075,6274],{"class":503},[151,8077,8079,8081,8083,8085,8087,8089,8091,8093,8096,8098,8100,8102,8104,8106,8108,8110,8112,8114,8116,8118,8120,8122,8124,8126,8128,8130,8132,8134,8136],{"class":469,"line":8078},61,[151,8080,5729],{"class":503},[151,8082,5732],{"class":6205},[151,8084,6208],{"class":503},[151,8086,7546],{"class":6211},[151,8088,106],{"class":503},[151,8090,6217],{"class":6205},[151,8092,6208],{"class":503},[151,8094,8095],{"class":6211},"\"02-20 21:47:39.246\"",[151,8097,106],{"class":503},[151,8099,6227],{"class":6205},[151,8101,6208],{"class":503},[151,8103,7564],{"class":6211},[151,8105,106],{"class":503},[151,8107,6237],{"class":6205},[151,8109,6208],{"class":503},[151,8111,7573],{"class":6211},[151,8113,106],{"class":503},[151,8115,6247],{"class":6205},[151,8117,6208],{"class":503},[151,8119,7582],{"class":6211},[151,8121,106],{"class":503},[151,8123,6257],{"class":6205},[151,8125,6208],{"class":503},[151,8127,6262],{"class":6211},[151,8129,106],{"class":503},[151,8131,6267],{"class":6205},[151,8133,6208],{"class":503},[151,8135,6262],{"class":6211},[151,8137,6274],{"class":503},[151,8139,8141],{"class":469,"line":8140},62,[151,8142,1090],{"emptyLinePlaceholder":609},[151,8144,8146,8148,8150,8152,8154,8156,8158,8160,8163,8165,8167,8169,8171,8173,8175,8177,8180,8182,8184,8186,8189,8191,8194,8196,8199,8201,8204,8206,8209,8211,8214,8216,8219,8221,8224,8226,8229,8231,8234,8236,8239,8241,8243,8245,8247,8249,8251,8253,8255],{"class":469,"line":8145},63,[151,8147,5729],{"class":503},[151,8149,5732],{"class":6205},[151,8151,6208],{"class":503},[151,8153,6212],{"class":6211},[151,8155,106],{"class":503},[151,8157,6217],{"class":6205},[151,8159,6208],{"class":503},[151,8161,8162],{"class":6211},"\"02-20 21:47:39.265\"",[151,8164,106],{"class":503},[151,8166,6227],{"class":6205},[151,8168,6208],{"class":503},[151,8170,6794],{"class":6211},[151,8172,106],{"class":503},[151,8174,6237],{"class":6205},[151,8176,6208],{"class":503},[151,8178,8179],{"class":6211},"\"456\"",[151,8181,106],{"class":503},[151,8183,6247],{"class":6205},[151,8185,6208],{"class":503},[151,8187,8188],{"class":6211},"\"Serving endpoints:",[151,8190,8043],{"class":477},[151,8192,8193],{"class":6211},"  0.0.0.0:8000/openapi.json",[151,8195,8043],{"class":477},[151,8197,8198],{"class":6211},"  0.0.0.0:8000/docs",[151,8200,8043],{"class":477},[151,8202,8203],{"class":6211},"  0.0.0.0:8000/docs/oauth2-redirect",[151,8205,8043],{"class":477},[151,8207,8208],{"class":6211},"  0.0.0.0:8000/metrics",[151,8210,8043],{"class":477},[151,8212,8213],{"class":6211},"  0.0.0.0:8000/v1/health/ready",[151,8215,8043],{"class":477},[151,8217,8218],{"class":6211},"  0.0.0.0:8000/v1/health/live",[151,8220,8043],{"class":477},[151,8222,8223],{"class":6211},"  0.0.0.0:8000/v1/models",[151,8225,8043],{"class":477},[151,8227,8228],{"class":6211},"  0.0.0.0:8000/v1/version",[151,8230,8043],{"class":477},[151,8232,8233],{"class":6211},"  0.0.0.0:8000/v1/chat/completions",[151,8235,8043],{"class":477},[151,8237,8238],{"class":6211},"  0.0.0.0:8000/v1/completions\"",[151,8240,106],{"class":503},[151,8242,6257],{"class":6205},[151,8244,6208],{"class":503},[151,8246,6262],{"class":6211},[151,8248,106],{"class":503},[151,8250,6267],{"class":6205},[151,8252,6208],{"class":503},[151,8254,6262],{"class":6211},[151,8256,6274],{"class":503},[151,8258,8260],{"class":469,"line":8259},64,[151,8261,1090],{"emptyLinePlaceholder":609},[151,8263,8265,8267,8269,8271,8273,8275,8277,8279,8281,8283,8285,8287,8289,8291,8293,8295,8298,8300,8302,8304,8307,8309,8312,8315,8318,8320,8323,8325,8328,8330,8333,8335,8338,8341,8343,8345,8347,8350,8352,8354,8356,8358,8361,8363,8366,8368,8371,8373,8376,8379,8381,8383,8385,8388,8390,8392,8394,8396,8399,8401,8403,8405,8408,8411,8414,8416,8418,8420,8422,8424,8426,8428,8430,8433,8435,8437,8439,8441,8443,8445,8447,8449,8452,8454,8456,8458,8460,8462,8464,8466,8468,8470,8472,8474,8476,8478,8480,8482,8484,8486,8488,8490,8493,8495,8498,8500,8503,8505,8507,8510,8512,8515,8517,8519,8522,8524,8526,8528,8530,8533,8535,8538,8540,8542,8545,8547,8550,8552,8554,8557,8559,8562,8564,8566,8569,8571,8573,8575,8578,8580,8583,8585,8588,8590,8593,8595,8597,8599,8601,8603,8605,8607,8609],{"class":469,"line":8264},65,[151,8266,5729],{"class":503},[151,8268,5732],{"class":6205},[151,8270,6208],{"class":503},[151,8272,6212],{"class":6211},[151,8274,106],{"class":503},[151,8276,6217],{"class":6205},[151,8278,6208],{"class":503},[151,8280,8162],{"class":6211},[151,8282,106],{"class":503},[151,8284,6227],{"class":6205},[151,8286,6208],{"class":503},[151,8288,6794],{"class":6211},[151,8290,106],{"class":503},[151,8292,6237],{"class":6205},[151,8294,6208],{"class":503},[151,8296,8297],{"class":6211},"\"460\"",[151,8299,106],{"class":503},[151,8301,6247],{"class":6205},[151,8303,6208],{"class":503},[151,8305,8306],{"class":6211},"\"An example cURL request:",[151,8308,8043],{"class":477},[151,8310,8311],{"class":6211},"curl -X 'POST' ",[151,8313,8314],{"class":477},"\\\\\\n",[151,8316,8317],{"class":6211},"  'http://0.0.0.0:8000/v1/chat/completions' ",[151,8319,8314],{"class":477},[151,8321,8322],{"class":6211},"  -H 'accept: application/json' ",[151,8324,8314],{"class":477},[151,8326,8327],{"class":6211},"  -H 'Content-Type: application/json' ",[151,8329,8314],{"class":477},[151,8331,8332],{"class":6211},"  -d '{",[151,8334,8043],{"class":477},[151,8336,8337],{"class":477},"    \\\"",[151,8339,8340],{"class":6211},"model",[151,8342,6323],{"class":477},[151,8344,6208],{"class":6211},[151,8346,6323],{"class":477},[151,8348,8349],{"class":6211},"meta/llama3-8b-instruct",[151,8351,6323],{"class":477},[151,8353,3634],{"class":6211},[151,8355,8043],{"class":477},[151,8357,8337],{"class":477},[151,8359,8360],{"class":6211},"messages",[151,8362,6323],{"class":477},[151,8364,8365],{"class":6211},": [",[151,8367,8043],{"class":477},[151,8369,8370],{"class":6211},"      {",[151,8372,8043],{"class":477},[151,8374,8375],{"class":477},"        \\\"",[151,8377,8378],{"class":6211},"role",[151,8380,6323],{"class":477},[151,8382,208],{"class":6211},[151,8384,6323],{"class":477},[151,8386,8387],{"class":6211},"user",[151,8389,6323],{"class":477},[151,8391,3634],{"class":6211},[151,8393,8043],{"class":477},[151,8395,8375],{"class":477},[151,8397,8398],{"class":6211},"content",[151,8400,6323],{"class":477},[151,8402,208],{"class":6211},[151,8404,6323],{"class":477},[151,8406,8407],{"class":6211},"Hello! How are you?",[151,8409,8410],{"class":477},"\\\"\\n",[151,8412,8413],{"class":6211},"      },",[151,8415,8043],{"class":477},[151,8417,8370],{"class":6211},[151,8419,8043],{"class":477},[151,8421,8375],{"class":477},[151,8423,8378],{"class":6211},[151,8425,6323],{"class":477},[151,8427,208],{"class":6211},[151,8429,6323],{"class":477},[151,8431,8432],{"class":6211},"assistant",[151,8434,6323],{"class":477},[151,8436,3634],{"class":6211},[151,8438,8043],{"class":477},[151,8440,8375],{"class":477},[151,8442,8398],{"class":6211},[151,8444,6323],{"class":477},[151,8446,208],{"class":6211},[151,8448,6323],{"class":477},[151,8450,8451],{"class":6211},"Hi! I am quite well, how can I help you today?",[151,8453,8410],{"class":477},[151,8455,8413],{"class":6211},[151,8457,8043],{"class":477},[151,8459,8370],{"class":6211},[151,8461,8043],{"class":477},[151,8463,8375],{"class":477},[151,8465,8378],{"class":6211},[151,8467,6323],{"class":477},[151,8469,208],{"class":6211},[151,8471,6323],{"class":477},[151,8473,8387],{"class":6211},[151,8475,6323],{"class":477},[151,8477,3634],{"class":6211},[151,8479,8043],{"class":477},[151,8481,8375],{"class":477},[151,8483,8398],{"class":6211},[151,8485,6323],{"class":477},[151,8487,208],{"class":6211},[151,8489,6323],{"class":477},[151,8491,8492],{"class":6211},"Can you write me a song?",[151,8494,8410],{"class":477},[151,8496,8497],{"class":6211},"      }",[151,8499,8043],{"class":477},[151,8501,8502],{"class":6211},"    ],",[151,8504,8043],{"class":477},[151,8506,8337],{"class":477},[151,8508,8509],{"class":6211},"top_p",[151,8511,6323],{"class":477},[151,8513,8514],{"class":6211},": 1,",[151,8516,8043],{"class":477},[151,8518,8337],{"class":477},[151,8520,8521],{"class":6211},"n",[151,8523,6323],{"class":477},[151,8525,8514],{"class":6211},[151,8527,8043],{"class":477},[151,8529,8337],{"class":477},[151,8531,8532],{"class":6211},"max_tokens",[151,8534,6323],{"class":477},[151,8536,8537],{"class":6211},": 15,",[151,8539,8043],{"class":477},[151,8541,8337],{"class":477},[151,8543,8544],{"class":6211},"stream",[151,8546,6323],{"class":477},[151,8548,8549],{"class":6211},": true,",[151,8551,8043],{"class":477},[151,8553,8337],{"class":477},[151,8555,8556],{"class":6211},"frequency_penalty",[151,8558,6323],{"class":477},[151,8560,8561],{"class":6211},": 1.0,",[151,8563,8043],{"class":477},[151,8565,8337],{"class":477},[151,8567,8568],{"class":6211},"stop",[151,8570,6323],{"class":477},[151,8572,8365],{"class":6211},[151,8574,6323],{"class":477},[151,8576,8577],{"class":6211},"hello",[151,8579,6323],{"class":477},[151,8581,8582],{"class":6211},"]",[151,8584,8043],{"class":477},[151,8586,8587],{"class":6211},"  }'",[151,8589,8043],{"class":477},[151,8591,8592],{"class":6211},"\"",[151,8594,106],{"class":503},[151,8596,6257],{"class":6205},[151,8598,6208],{"class":503},[151,8600,6262],{"class":6211},[151,8602,106],{"class":503},[151,8604,6267],{"class":6205},[151,8606,6208],{"class":503},[151,8608,6262],{"class":6211},[151,8610,6274],{"class":503},[151,8612,8614,8616,8618,8620,8622,8624,8626,8628,8631,8633,8635,8637,8640,8642,8644,8646,8649,8651,8653,8655,8658,8660,8662,8664,8666,8668,8670,8672,8674],{"class":469,"line":8613},66,[151,8615,5729],{"class":503},[151,8617,5732],{"class":6205},[151,8619,6208],{"class":503},[151,8621,6212],{"class":6211},[151,8623,106],{"class":503},[151,8625,6217],{"class":6205},[151,8627,6208],{"class":503},[151,8629,8630],{"class":6211},"\"02-20 21:47:39.336\"",[151,8632,106],{"class":503},[151,8634,6227],{"class":6205},[151,8636,6208],{"class":503},[151,8638,8639],{"class":6211},"\"/usr/local/lib/python3.10/dist-packages/uvicorn/server.py\"",[151,8641,106],{"class":503},[151,8643,6237],{"class":6205},[151,8645,6208],{"class":503},[151,8647,8648],{"class":6211},"\"82\"",[151,8650,106],{"class":503},[151,8652,6247],{"class":6205},[151,8654,6208],{"class":503},[151,8656,8657],{"class":6211},"\"Started server process [32]\"",[151,8659,106],{"class":503},[151,8661,6257],{"class":6205},[151,8663,6208],{"class":503},[151,8665,6262],{"class":6211},[151,8667,106],{"class":503},[151,8669,6267],{"class":6205},[151,8671,6208],{"class":503},[151,8673,6262],{"class":6211},[151,8675,6274],{"class":503},[151,8677,8679,8681,8683,8685,8687,8689,8691,8693,8695,8697,8699,8701,8704,8706,8708,8710,8713,8715,8717,8719,8722,8724,8726,8728,8730,8732,8734,8736,8738],{"class":469,"line":8678},67,[151,8680,5729],{"class":503},[151,8682,5732],{"class":6205},[151,8684,6208],{"class":503},[151,8686,6212],{"class":6211},[151,8688,106],{"class":503},[151,8690,6217],{"class":6205},[151,8692,6208],{"class":503},[151,8694,8630],{"class":6211},[151,8696,106],{"class":503},[151,8698,6227],{"class":6205},[151,8700,6208],{"class":503},[151,8702,8703],{"class":6211},"\"/usr/local/lib/python3.10/dist-packages/uvicorn/lifespan/on.py\"",[151,8705,106],{"class":503},[151,8707,6237],{"class":6205},[151,8709,6208],{"class":503},[151,8711,8712],{"class":6211},"\"48\"",[151,8714,106],{"class":503},[151,8716,6247],{"class":6205},[151,8718,6208],{"class":503},[151,8720,8721],{"class":6211},"\"Waiting for application startup.\"",[151,8723,106],{"class":503},[151,8725,6257],{"class":6205},[151,8727,6208],{"class":503},[151,8729,6262],{"class":6211},[151,8731,106],{"class":503},[151,8733,6267],{"class":6205},[151,8735,6208],{"class":503},[151,8737,6262],{"class":6211},[151,8739,6274],{"class":503},[151,8741,8743,8745,8747,8749,8751,8753,8755,8757,8760,8762,8764,8766,8768,8770,8772,8774,8777,8779,8781,8783,8786,8788,8790,8792,8794,8796,8798,8800,8802],{"class":469,"line":8742},68,[151,8744,5729],{"class":503},[151,8746,5732],{"class":6205},[151,8748,6208],{"class":503},[151,8750,6212],{"class":6211},[151,8752,106],{"class":503},[151,8754,6217],{"class":6205},[151,8756,6208],{"class":503},[151,8758,8759],{"class":6211},"\"02-20 21:47:39.338\"",[151,8761,106],{"class":503},[151,8763,6227],{"class":6205},[151,8765,6208],{"class":503},[151,8767,8703],{"class":6211},[151,8769,106],{"class":503},[151,8771,6237],{"class":6205},[151,8773,6208],{"class":503},[151,8775,8776],{"class":6211},"\"62\"",[151,8778,106],{"class":503},[151,8780,6247],{"class":6205},[151,8782,6208],{"class":503},[151,8784,8785],{"class":6211},"\"Application startup complete.\"",[151,8787,106],{"class":503},[151,8789,6257],{"class":6205},[151,8791,6208],{"class":503},[151,8793,6262],{"class":6211},[151,8795,106],{"class":503},[151,8797,6267],{"class":6205},[151,8799,6208],{"class":503},[151,8801,6262],{"class":6211},[151,8803,6274],{"class":503},[151,8805,8807,8809,8811,8813,8815,8817,8819,8821,8824,8826,8828,8830,8832,8834,8836,8838,8841,8843,8845,8847,8850,8852,8854,8856,8858,8860,8862,8864,8866],{"class":469,"line":8806},69,[151,8808,5729],{"class":503},[151,8810,5732],{"class":6205},[151,8812,6208],{"class":503},[151,8814,6212],{"class":6211},[151,8816,106],{"class":503},[151,8818,6217],{"class":6205},[151,8820,6208],{"class":503},[151,8822,8823],{"class":6211},"\"02-20 21:47:39.340\"",[151,8825,106],{"class":503},[151,8827,6227],{"class":6205},[151,8829,6208],{"class":503},[151,8831,8639],{"class":6211},[151,8833,106],{"class":503},[151,8835,6237],{"class":6205},[151,8837,6208],{"class":503},[151,8839,8840],{"class":6211},"\"214\"",[151,8842,106],{"class":503},[151,8844,6247],{"class":6205},[151,8846,6208],{"class":503},[151,8848,8849],{"class":6211},"\"Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)\"",[151,8851,106],{"class":503},[151,8853,6257],{"class":6205},[151,8855,6208],{"class":503},[151,8857,6262],{"class":6211},[151,8859,106],{"class":503},[151,8861,6267],{"class":6205},[151,8863,6208],{"class":503},[151,8865,6262],{"class":6211},[151,8867,6274],{"class":503},[151,8869,8871],{"class":469,"line":8870},70,[151,8872,1090],{"emptyLinePlaceholder":609},[151,8874,8876],{"class":469,"line":8875},71,[151,8877,8878],{"class":1527},"// the rest of the logs are just health checks and stats\n",[151,8880,8882],{"class":469,"line":8881},72,[151,8883,1090],{"emptyLinePlaceholder":609},[151,8885,8887],{"class":469,"line":8886},73,[151,8888,8889],{"class":1527},"// health check\n",[151,8891,8893,8895,8897,8899,8901,8903,8905,8907,8910,8912,8914,8916,8918,8920,8922,8924,8926,8928,8930,8932,8935,8937,8939,8941,8943,8945,8947,8949,8951,8953,8955,8957,8959],{"class":469,"line":8892},74,[151,8894,5729],{"class":503},[151,8896,5732],{"class":6205},[151,8898,6208],{"class":503},[151,8900,6212],{"class":6211},[151,8902,106],{"class":503},[151,8904,6217],{"class":6205},[151,8906,6208],{"class":503},[151,8908,8909],{"class":6211},"\"02-20 21:54:56.805\"",[151,8911,106],{"class":503},[151,8913,6227],{"class":6205},[151,8915,6208],{"class":503},[151,8917,6302],{"class":6211},[151,8919,106],{"class":503},[151,8921,6237],{"class":6205},[151,8923,6208],{"class":503},[151,8925,6311],{"class":6211},[151,8927,106],{"class":503},[151,8929,6247],{"class":6205},[151,8931,6208],{"class":503},[151,8933,8934],{"class":6211},"\"10.28.1.1:35412 - ",[151,8936,6323],{"class":477},[151,8938,6398],{"class":6211},[151,8940,6323],{"class":477},[151,8942,6331],{"class":6211},[151,8944,106],{"class":503},[151,8946,6257],{"class":6205},[151,8948,6208],{"class":503},[151,8950,6262],{"class":6211},[151,8952,106],{"class":503},[151,8954,6267],{"class":6205},[151,8956,6208],{"class":503},[151,8958,6262],{"class":6211},[151,8960,6274],{"class":503},[151,8962,8964],{"class":469,"line":8963},75,[151,8965,8966],{"class":1527},"// stats\n",[151,8968,8970,8972,8974,8976,8978,8980,8982,8984,8986,8988,8990,8992,8994,8996,8998,9000,9002,9004,9006,9008,9010,9012,9014,9016,9018,9020,9022,9024,9026],{"class":469,"line":8969},76,[151,8971,5729],{"class":503},[151,8973,5732],{"class":6205},[151,8975,6208],{"class":503},[151,8977,6212],{"class":6211},[151,8979,106],{"class":503},[151,8981,6217],{"class":6205},[151,8983,6208],{"class":503},[151,8985,6222],{"class":6211},[151,8987,106],{"class":503},[151,8989,6227],{"class":6205},[151,8991,6208],{"class":503},[151,8993,6232],{"class":6211},[151,8995,106],{"class":503},[151,8997,6237],{"class":6205},[151,8999,6208],{"class":503},[151,9001,6242],{"class":6211},[151,9003,106],{"class":503},[151,9005,6247],{"class":6205},[151,9007,6208],{"class":503},[151,9009,6252],{"class":6211},[151,9011,106],{"class":503},[151,9013,6257],{"class":6205},[151,9015,6208],{"class":503},[151,9017,6262],{"class":6211},[151,9019,106],{"class":503},[151,9021,6267],{"class":6205},[151,9023,6208],{"class":503},[151,9025,6262],{"class":6211},[151,9027,6274],{"class":503},[11,9029,9030],{},"Now it is running! Yay!",[459,9032,9034],{"className":461,"code":9033,"language":463,"meta":464,"style":464},"brian@cloudshell:~ (waywo-487618)$ kubectl get pods -n nim\nNAME               READY   STATUS    RESTARTS   AGE\nmy-nim-nim-llm-0   1/1     Running   0          6m26s\n",[30,9035,9036,9042,9054],{"__ignoreMap":464},[151,9037,9038,9040],{"class":469,"line":470},[151,9039,2130],{"class":473},[151,9041,5417],{"class":503},[151,9043,9044,9046,9048,9050,9052],{"class":469,"line":488},[151,9045,5422],{"class":473},[151,9047,5425],{"class":481},[151,9049,5428],{"class":481},[151,9051,5431],{"class":481},[151,9053,5434],{"class":481},[151,9055,9056,9058,9061,9063,9065],{"class":469,"line":500},[151,9057,5439],{"class":473},[151,9059,9060],{"class":481},"   1/1",[151,9062,5445],{"class":481},[151,9064,5448],{"class":477},[151,9066,9067],{"class":481},"          6m26s\n",[11,9069,9070],{},"Let’s just poke around a little bit:",[459,9072,9074],{"className":6194,"code":9073,"language":6196,"meta":464,"style":464},"brian@cloudshell:~ (waywo-487618)$ kubectl exec -it my-nim-nim-llm-0 -n nim -- nvidia-smi\nFri Feb 20 22:02:53 2026\n+-----------------------------------------------------------------------------------------+\n| NVIDIA-SMI 580.105.08             Driver Version: 580.105.08     CUDA Version: 13.0     |\n+-----------------------------------------+------------------------+----------------------+\n| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |\n| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |\n|                                         |                        |               MIG M. |\n|=========================================+========================+======================|\n|   0  NVIDIA L4                      Off |   00000000:00:03.0 Off |                    0 |\n| N/A   62C    P0             37W /   72W |   19140MiB /  23034MiB |      0%      Default |\n|                                         |                        |                  N/A |\n+-----------------------------------------+------------------------+----------------------+\n\n+-----------------------------------------------------------------------------------------+\n| Processes:                                                                              |\n|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |\n|        ID   ID                                                               Usage      |\n|=========================================================================================|\n|    0   N/A  N/A              32      C   python3                               19124MiB |\n+-----------------------------------------------------------------------------------------+\nbrian@cloudshell:~ (waywo-487618)$\n",[30,9075,9076,9090,9113,9118,9149,9154,9159,9164,9169,9174,9211,9253,9258,9262,9266,9270,9275,9280,9285,9290,9314,9318],{"__ignoreMap":464},[151,9077,9078,9080,9082,9085,9087],{"class":469,"line":470},[151,9079,6433],{"class":503},[151,9081,6436],{"class":477},[151,9083,9084],{"class":503},")$ kubectl exec -it my-nim-nim-llm",[151,9086,6442],{"class":477},[151,9088,9089],{"class":503}," -n nim -- nvidia-smi\n",[151,9091,9092,9095,9098,9101,9103,9106,9108,9111],{"class":469,"line":488},[151,9093,9094],{"class":503},"Fri Feb ",[151,9096,9097],{"class":477},"20",[151,9099,9100],{"class":477}," 22",[151,9102,208],{"class":503},[151,9104,9105],{"class":477},"02",[151,9107,208],{"class":503},[151,9109,9110],{"class":477},"53",[151,9112,5256],{"class":477},[151,9114,9115],{"class":469,"line":500},[151,9116,9117],{"class":503},"+-----------------------------------------------------------------------------------------+\n",[151,9119,9120,9123,9126,9128,9131,9134,9136,9138,9140,9143,9146],{"class":469,"line":509},[151,9121,9122],{"class":503},"| NVIDIA-SMI ",[151,9124,9125],{"class":477},"580.105",[151,9127,643],{"class":503},[151,9129,9130],{"class":477},"08",[151,9132,9133],{"class":503},"             Driver Version: ",[151,9135,9125],{"class":477},[151,9137,643],{"class":503},[151,9139,9130],{"class":477},[151,9141,9142],{"class":503},"     CUDA Version: ",[151,9144,9145],{"class":477},"13.0",[151,9147,9148],{"class":503},"     |\n",[151,9150,9151],{"class":469,"line":517},[151,9152,9153],{"class":503},"+-----------------------------------------+------------------------+----------------------+\n",[151,9155,9156],{"class":469,"line":534},[151,9157,9158],{"class":503},"| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |\n",[151,9160,9161],{"class":469,"line":1413},[151,9162,9163],{"class":503},"| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |\n",[151,9165,9166],{"class":469,"line":1418},[151,9167,9168],{"class":503},"|                                         |                        |               MIG M. |\n",[151,9170,9171],{"class":469,"line":2462},[151,9172,9173],{"class":503},"|=========================================+========================+======================|\n",[151,9175,9176,9179,9182,9185,9188,9191,9194,9196,9199,9201,9204,9207,9209],{"class":469,"line":2471},[151,9177,9178],{"class":503},"|   ",[151,9180,9181],{"class":477},"0",[151,9183,9184],{"class":503},"  NVIDIA L",[151,9186,9187],{"class":477},"4",[151,9189,9190],{"class":503},"                      Off |   ",[151,9192,9193],{"class":477},"00000000",[151,9195,208],{"class":503},[151,9197,9198],{"class":477},"00",[151,9200,208],{"class":503},[151,9202,9203],{"class":477},"03.0",[151,9205,9206],{"class":503}," Off |                    ",[151,9208,9181],{"class":477},[151,9210,3979],{"class":503},[151,9212,9213,9216,9219,9222,9224,9227,9230,9233,9236,9239,9242,9245,9248,9250],{"class":469,"line":2480},[151,9214,9215],{"class":503},"| N/A   ",[151,9217,9218],{"class":477},"62",[151,9220,9221],{"class":503},"C    P",[151,9223,9181],{"class":477},[151,9225,9226],{"class":477},"             37",[151,9228,9229],{"class":503},"W /   ",[151,9231,9232],{"class":477},"72",[151,9234,9235],{"class":503},"W |   ",[151,9237,9238],{"class":477},"19140",[151,9240,9241],{"class":503},"MiB /  ",[151,9243,9244],{"class":477},"23034",[151,9246,9247],{"class":503},"MiB |      ",[151,9249,9181],{"class":477},[151,9251,9252],{"class":503},"%      Default |\n",[151,9254,9255],{"class":469,"line":2489},[151,9256,9257],{"class":503},"|                                         |                        |                  N/A |\n",[151,9259,9260],{"class":469,"line":2497},[151,9261,9153],{"class":503},[151,9263,9264],{"class":469,"line":3140},[151,9265,1090],{"emptyLinePlaceholder":609},[151,9267,9268],{"class":469,"line":3149},[151,9269,9117],{"class":503},[151,9271,9272],{"class":469,"line":3158},[151,9273,9274],{"class":503},"| Processes:                                                                              |\n",[151,9276,9277],{"class":469,"line":3167},[151,9278,9279],{"class":503},"|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |\n",[151,9281,9282],{"class":469,"line":3175},[151,9283,9284],{"class":503},"|        ID   ID                                                               Usage      |\n",[151,9286,9287],{"class":469,"line":3184},[151,9288,9289],{"class":503},"|=========================================================================================|\n",[151,9291,9292,9295,9297,9300,9303,9306,9308,9311],{"class":469,"line":3193},[151,9293,9294],{"class":503},"|    ",[151,9296,9181],{"class":477},[151,9298,9299],{"class":503},"   N/A  N/A              ",[151,9301,9302],{"class":477},"32",[151,9304,9305],{"class":503},"      C   python",[151,9307,6557],{"class":477},[151,9309,9310],{"class":477},"                               19124",[151,9312,9313],{"class":503},"MiB |\n",[151,9315,9316],{"class":469,"line":3720},[151,9317,9117],{"class":503},[151,9319,9320,9322,9324],{"class":469,"line":3729},[151,9321,6433],{"class":503},[151,9323,6436],{"class":477},[151,9325,9326],{"class":503},")$\n",[11,9328,9329],{},[2718,9330],{"alt":464,"src":464},[11,9332,9333],{},"Let’s query the LLM! Up until now I have been doing everything in the Cloud Shell terminal. Let’s port forward traffic to localhost:8000. Let’s run the following command in one Cloud Shell terminal",[459,9335,9337],{"className":461,"code":9336,"language":463,"meta":464,"style":464},"kubectl port-forward service/my-nim-nim-llm 8000:8000 -n nim\n",[30,9338,9339],{"__ignoreMap":464},[151,9340,9341,9343,9346,9349,9352,9354],{"class":469,"line":470},[151,9342,4624],{"class":473},[151,9344,9345],{"class":481}," port-forward",[151,9347,9348],{"class":481}," service/my-nim-nim-llm",[151,9350,9351],{"class":481}," 8000:8000",[151,9353,4696],{"class":477},[151,9355,4632],{"class":481},[11,9357,9358],{},"Then, in another Cloud Shell terminal, let’s send a request:",[459,9360,9362],{"className":6194,"code":9361,"language":6196,"meta":464,"style":464},"curl -X 'POST' \\\n  'http://localhost:8000/v1/chat/completions' \\\n  -H 'accept: application/json' \\\n  -H 'Content-Type: application/json' \\\n  -d '{\n  \"messages\": [\n    {\n      \"content\": \"You are a polite and respectful chatbot helping people plan a vacation.\",\n      \"role\": \"system\"\n    },\n    {\n      \"content\": \"What should I do for a 4 day vacation in Spain?\",\n      \"role\": \"user\"\n    }\n  ],\n  \"model\": \"meta/llama3-8b-instruct\",\n  \"max_tokens\": 128,\n  \"top_p\": 1,\n  \"n\": 1,\n  \"stream\": false,\n  \"stop\": \"\\n\",\n  \"frequency_penalty\": 0.0\n}'\n",[30,9363,9364,9369,9377,9382,9387,9392,9400,9405,9418,9428,9433,9437,9448,9457,9462,9467,9479,9491,9502,9513,9525,9540,9550],{"__ignoreMap":464},[151,9365,9366],{"class":469,"line":470},[151,9367,9368],{"class":503},"curl -X 'POST' \\\n",[151,9370,9371,9374],{"class":469,"line":488},[151,9372,9373],{"class":503},"  'http:",[151,9375,9376],{"class":1527},"//localhost:8000/v1/chat/completions' \\\n",[151,9378,9379],{"class":469,"line":500},[151,9380,9381],{"class":503},"  -H 'accept: application/json' \\\n",[151,9383,9384],{"class":469,"line":509},[151,9385,9386],{"class":503},"  -H 'Content-Type: application/json' \\\n",[151,9388,9389],{"class":469,"line":517},[151,9390,9391],{"class":503},"  -d '{\n",[151,9393,9394,9397],{"class":469,"line":534},[151,9395,9396],{"class":6205},"  \"messages\"",[151,9398,9399],{"class":503},": [\n",[151,9401,9402],{"class":469,"line":1413},[151,9403,9404],{"class":503},"    {\n",[151,9406,9407,9410,9412,9415],{"class":469,"line":1418},[151,9408,9409],{"class":6205},"      \"content\"",[151,9411,6208],{"class":503},[151,9413,9414],{"class":6211},"\"You are a polite and respectful chatbot helping people plan a vacation.\"",[151,9416,9417],{"class":503},",\n",[151,9419,9420,9423,9425],{"class":469,"line":2462},[151,9421,9422],{"class":6205},"      \"role\"",[151,9424,6208],{"class":503},[151,9426,9427],{"class":6211},"\"system\"\n",[151,9429,9430],{"class":469,"line":2471},[151,9431,9432],{"class":503},"    },\n",[151,9434,9435],{"class":469,"line":2480},[151,9436,9404],{"class":503},[151,9438,9439,9441,9443,9446],{"class":469,"line":2489},[151,9440,9409],{"class":6205},[151,9442,6208],{"class":503},[151,9444,9445],{"class":6211},"\"What should I do for a 4 day vacation in Spain?\"",[151,9447,9417],{"class":503},[151,9449,9450,9452,9454],{"class":469,"line":2497},[151,9451,9422],{"class":6205},[151,9453,6208],{"class":503},[151,9455,9456],{"class":6211},"\"user\"\n",[151,9458,9459],{"class":469,"line":3140},[151,9460,9461],{"class":503},"    }\n",[151,9463,9464],{"class":469,"line":3149},[151,9465,9466],{"class":503},"  ],\n",[151,9468,9469,9472,9474,9477],{"class":469,"line":3158},[151,9470,9471],{"class":6205},"  \"model\"",[151,9473,6208],{"class":503},[151,9475,9476],{"class":6211},"\"meta/llama3-8b-instruct\"",[151,9478,9417],{"class":503},[151,9480,9481,9484,9486,9489],{"class":469,"line":3167},[151,9482,9483],{"class":6205},"  \"max_tokens\"",[151,9485,6208],{"class":503},[151,9487,9488],{"class":477},"128",[151,9490,9417],{"class":503},[151,9492,9493,9496,9498,9500],{"class":469,"line":3175},[151,9494,9495],{"class":6205},"  \"top_p\"",[151,9497,6208],{"class":503},[151,9499,6760],{"class":477},[151,9501,9417],{"class":503},[151,9503,9504,9507,9509,9511],{"class":469,"line":3184},[151,9505,9506],{"class":6205},"  \"n\"",[151,9508,6208],{"class":503},[151,9510,6760],{"class":477},[151,9512,9417],{"class":503},[151,9514,9515,9518,9520,9523],{"class":469,"line":3193},[151,9516,9517],{"class":6205},"  \"stream\"",[151,9519,6208],{"class":503},[151,9521,9522],{"class":477},"false",[151,9524,9417],{"class":503},[151,9526,9527,9530,9532,9534,9536,9538],{"class":469,"line":3720},[151,9528,9529],{"class":6205},"  \"stop\"",[151,9531,6208],{"class":503},[151,9533,8592],{"class":6211},[151,9535,8043],{"class":477},[151,9537,8592],{"class":6211},[151,9539,9417],{"class":503},[151,9541,9542,9545,9547],{"class":469,"line":3729},[151,9543,9544],{"class":6205},"  \"frequency_penalty\"",[151,9546,6208],{"class":503},[151,9548,9549],{"class":477},"0.0\n",[151,9551,9552],{"class":469,"line":3735},[151,9553,9554],{"class":503},"}'\n",[11,9556,9557],{},"Nice! We got a response from Llama 3 8B Instruct LLM served by TensorRT-LLM running in an NVIDIA NIM container orchestrated by GKE on GCP!!",[11,9559,9560],{},[2718,9561],{"alt":464,"src":9562},"/static/codelab/nvgcp_inference.png",[459,9564,9566],{"className":6194,"code":9565,"language":6196,"meta":464,"style":464},"brian@cloudshell:~ (waywo-487618)$ curl -X 'POST' \\\n  'http://localhost:8000/v1/chat/completions' \\\n  -H 'accept: application/json' \\\n  -H 'Content-Type: application/json' \\\n  -d '{\n  \"messages\": [\n    {\n      \"content\": \"You are a polite and respectful chatbot helping people plan a vacation.\",\n      \"role\": \"system\"\n    },\n    {\n      \"content\": \"What should I do for a 4 day vacation in Spain?\",\n      \"role\": \"user\"\n    }\n  ],\n  \"model\": \"meta/llama3-8b-instruct\",\n  \"max_tokens\": 128,\n  \"top_p\": 1,\n  \"n\": 1,\n  \"stream\": false,\n  \"stop\": \"\\n\",\n  \"frequency_penalty\": 0.0\n}'\n{\"id\":\"cmpl-d3296c2507f54673a01c715b58a6d414\",\"object\":\"chat.completion\",\"created\":1771625503,\"model\":\"meta/llama3-8b-instruct\",\"choices\":[{\"index\":0,\"message\":{\"role\":\"assistant\",\"content\":\"Spain! What a fantastic destination for a 4-day vacation! With so much to see and do, I'd be happy to help you plan your trip.\"},\"logprobs\":null,\"finish_reason\":\"stobrian@cloudshel\nbrian@cloudshell:~ (waywo-487618)$\n",[30,9567,9568,9577,9583,9587,9591,9595,9601,9605,9615,9623,9627,9631,9641,9649,9653,9657,9667,9677,9687,9697,9707,9721,9729,9733,9835],{"__ignoreMap":464},[151,9569,9570,9572,9574],{"class":469,"line":470},[151,9571,6433],{"class":503},[151,9573,6436],{"class":477},[151,9575,9576],{"class":503},")$ curl -X 'POST' \\\n",[151,9578,9579,9581],{"class":469,"line":488},[151,9580,9373],{"class":503},[151,9582,9376],{"class":1527},[151,9584,9585],{"class":469,"line":500},[151,9586,9381],{"class":503},[151,9588,9589],{"class":469,"line":509},[151,9590,9386],{"class":503},[151,9592,9593],{"class":469,"line":517},[151,9594,9391],{"class":503},[151,9596,9597,9599],{"class":469,"line":534},[151,9598,9396],{"class":6205},[151,9600,9399],{"class":503},[151,9602,9603],{"class":469,"line":1413},[151,9604,9404],{"class":503},[151,9606,9607,9609,9611,9613],{"class":469,"line":1418},[151,9608,9409],{"class":6205},[151,9610,6208],{"class":503},[151,9612,9414],{"class":6211},[151,9614,9417],{"class":503},[151,9616,9617,9619,9621],{"class":469,"line":2462},[151,9618,9422],{"class":6205},[151,9620,6208],{"class":503},[151,9622,9427],{"class":6211},[151,9624,9625],{"class":469,"line":2471},[151,9626,9432],{"class":503},[151,9628,9629],{"class":469,"line":2480},[151,9630,9404],{"class":503},[151,9632,9633,9635,9637,9639],{"class":469,"line":2489},[151,9634,9409],{"class":6205},[151,9636,6208],{"class":503},[151,9638,9445],{"class":6211},[151,9640,9417],{"class":503},[151,9642,9643,9645,9647],{"class":469,"line":2497},[151,9644,9422],{"class":6205},[151,9646,6208],{"class":503},[151,9648,9456],{"class":6211},[151,9650,9651],{"class":469,"line":3140},[151,9652,9461],{"class":503},[151,9654,9655],{"class":469,"line":3149},[151,9656,9466],{"class":503},[151,9658,9659,9661,9663,9665],{"class":469,"line":3158},[151,9660,9471],{"class":6205},[151,9662,6208],{"class":503},[151,9664,9476],{"class":6211},[151,9666,9417],{"class":503},[151,9668,9669,9671,9673,9675],{"class":469,"line":3167},[151,9670,9483],{"class":6205},[151,9672,6208],{"class":503},[151,9674,9488],{"class":477},[151,9676,9417],{"class":503},[151,9678,9679,9681,9683,9685],{"class":469,"line":3175},[151,9680,9495],{"class":6205},[151,9682,6208],{"class":503},[151,9684,6760],{"class":477},[151,9686,9417],{"class":503},[151,9688,9689,9691,9693,9695],{"class":469,"line":3184},[151,9690,9506],{"class":6205},[151,9692,6208],{"class":503},[151,9694,6760],{"class":477},[151,9696,9417],{"class":503},[151,9698,9699,9701,9703,9705],{"class":469,"line":3193},[151,9700,9517],{"class":6205},[151,9702,6208],{"class":503},[151,9704,9522],{"class":477},[151,9706,9417],{"class":503},[151,9708,9709,9711,9713,9715,9717,9719],{"class":469,"line":3720},[151,9710,9529],{"class":6205},[151,9712,6208],{"class":503},[151,9714,8592],{"class":6211},[151,9716,8043],{"class":477},[151,9718,8592],{"class":6211},[151,9720,9417],{"class":503},[151,9722,9723,9725,9727],{"class":469,"line":3729},[151,9724,9544],{"class":6205},[151,9726,6208],{"class":503},[151,9728,9549],{"class":477},[151,9730,9731],{"class":469,"line":3735},[151,9732,9554],{"class":503},[151,9734,9735,9737,9740,9742,9745,9747,9750,9752,9755,9757,9760,9762,9765,9767,9770,9772,9774,9776,9779,9782,9785,9787,9789,9791,9793,9796,9799,9801,9804,9806,9809,9811,9814,9817,9820,9822,9825,9827,9830,9832],{"class":469,"line":3745},[151,9736,5729],{"class":503},[151,9738,9739],{"class":6205},"\"id\"",[151,9741,208],{"class":503},[151,9743,9744],{"class":6211},"\"cmpl-d3296c2507f54673a01c715b58a6d414\"",[151,9746,3634],{"class":503},[151,9748,9749],{"class":6205},"\"object\"",[151,9751,208],{"class":503},[151,9753,9754],{"class":6211},"\"chat.completion\"",[151,9756,3634],{"class":503},[151,9758,9759],{"class":6205},"\"created\"",[151,9761,208],{"class":503},[151,9763,9764],{"class":477},"1771625503",[151,9766,3634],{"class":503},[151,9768,9769],{"class":6205},"\"model\"",[151,9771,208],{"class":503},[151,9773,9476],{"class":6211},[151,9775,3634],{"class":503},[151,9777,9778],{"class":6205},"\"choices\"",[151,9780,9781],{"class":503},":[{",[151,9783,9784],{"class":6205},"\"index\"",[151,9786,208],{"class":503},[151,9788,9181],{"class":477},[151,9790,3634],{"class":503},[151,9792,6247],{"class":6205},[151,9794,9795],{"class":503},":{",[151,9797,9798],{"class":6205},"\"role\"",[151,9800,208],{"class":503},[151,9802,9803],{"class":6211},"\"assistant\"",[151,9805,3634],{"class":503},[151,9807,9808],{"class":6205},"\"content\"",[151,9810,208],{"class":503},[151,9812,9813],{"class":6211},"\"Spain! What a fantastic destination for a 4-day vacation! With so much to see and do, I'd be happy to help you plan your trip.\"",[151,9815,9816],{"class":503},"},",[151,9818,9819],{"class":6205},"\"logprobs\"",[151,9821,208],{"class":503},[151,9823,9824],{"class":477},"null",[151,9826,3634],{"class":503},[151,9828,9829],{"class":6205},"\"finish_reason\"",[151,9831,208],{"class":503},[151,9833,9834],{"class":6211},"\"stobrian@cloudshel\n",[151,9836,9837],{"class":469,"line":3754},[151,9838,9839],{"class":6211},"brian@cloudshell:~ (waywo-487618)$\n",[11,9841,9842,9843,9847,9848,9852],{},"That was fun. Using the Cloud Terminal shell and Cloud Assist super convenient! Gemini CLI is probably pretty amazing to use in Cloud Terminal, too. There’s also a gcloud-mcp server from Google, I wonder why this isn’t installed by default? ",[20,9844,9845],{"href":9845,"rel":9846},"https://github.com/googleapis/gcloud-mcp",[24],". The gke-mcp MCP server also looks interesting: ",[20,9849,9850],{"href":9850,"rel":9851},"https://github.com/GoogleCloudPlatform/gke-mcp",[24],". Gemini + MCP servers in Cloud Shell seems like a really powerful way to both build and debug!",[11,9854,9855],{},[2718,9856],{"alt":2718,"src":9857},"/static/codelab/nvgcp_gemini.png",[736,9859,9861],{"id":9860},"infrastructure-as-code","Infrastructure as Code",[11,9863,9864,9865,9867,9868,9871],{},"I found ",[30,9866,1842],{}," to be a really informative way to take first steps with NIMs on GKE, it was awesome! Building everything logically step by step helped me map cloud concepts to GKE from my understanding of AWS with EKS and ECS for container orchestration. However, the automation engineer in me wanted to know how I could take this great NIM deployment example and fully automate with Infrastructure as Code! Getting this working would be a helpful stepping stone to getting my ",[30,9869,9870],{},"waywo"," app deployed on GKE, with multiple NIMs, nodes, bells, and whistles!",[736,9873,9875],{"id":9874},"cleanup","Cleanup",[11,9877,9878,9879,9881],{},"Don’t forget to clean up the resources! That’s the next part of the ",[30,9880,1842],{}," tutorial.",[11,9883,9884],{},"This can be done with just one simple command:",[459,9886,9888],{"className":461,"code":9887,"language":463,"meta":464,"style":464},"gcloud container clusters delete $CLUSTER_NAME --zone=$ZONE\n",[30,9889,9890],{"__ignoreMap":464},[151,9891,9892,9894,9896,9898,9901,9904,9907],{"class":469,"line":470},[151,9893,1977],{"class":473},[151,9895,1980],{"class":481},[151,9897,1983],{"class":481},[151,9899,9900],{"class":481}," delete",[151,9902,9903],{"class":503}," $CLUSTER_NAME ",[151,9905,9906],{"class":477},"--zone=",[151,9908,9909],{"class":503},"$ZONE\n",[11,9911,9912],{},"Good night, gcloud! ☁️ 🌝",[459,9914,9916],{"className":461,"code":9915,"language":463,"meta":464,"style":464},"brian@cloudshell:~ (waywo-487618)$ gcloud container clusters delete $CLUSTER_NAME --zone=$ZONE\nThe following clusters will be deleted.\n - [nim-demo] in [us-east4-a]\n\nDo you want to continue (Y/n)?\n\nDeleting cluster nim-demo...done.\nDeleted [https://container.googleapis.com/v1/projects/waywo-487618/zones/us-east4-a/clusters/nim-demo].\n",[30,9917,9918,9929,9945,9953,9957,9977,9981,9991],{"__ignoreMap":464},[151,9919,9920,9922,9925,9927],{"class":469,"line":470},[151,9921,2130],{"class":473},[151,9923,9924],{"class":503}," (waywo-487618)$ gcloud container clusters delete $CLUSTER_NAME --zone",[151,9926,1876],{"class":1869},[151,9928,9909],{"class":503},[151,9930,9931,9933,9935,9937,9940,9942],{"class":469,"line":488},[151,9932,5029],{"class":473},[151,9934,3927],{"class":481},[151,9936,1983],{"class":481},[151,9938,9939],{"class":481}," will",[151,9941,4051],{"class":481},[151,9943,9944],{"class":481}," deleted.\n",[151,9946,9947,9950],{"class":469,"line":500},[151,9948,9949],{"class":473}," -",[151,9951,9952],{"class":503}," [nim-demo] in [us-east4-a]\n",[151,9954,9955],{"class":469,"line":509},[151,9956,1090],{"emptyLinePlaceholder":609},[151,9958,9959,9962,9964,9966,9968,9971,9974],{"class":469,"line":517},[151,9960,9961],{"class":473},"Do",[151,9963,3438],{"class":481},[151,9965,4214],{"class":481},[151,9967,2312],{"class":481},[151,9969,9970],{"class":481}," continue",[151,9972,9973],{"class":503}," (Y/n)",[151,9975,9976],{"class":1869},"?\n",[151,9978,9979],{"class":469,"line":534},[151,9980,1090],{"emptyLinePlaceholder":609},[151,9982,9983,9986,9988],{"class":469,"line":1413},[151,9984,9985],{"class":473},"Deleting",[151,9987,2814],{"class":481},[151,9989,9990],{"class":481}," nim-demo...done.\n",[151,9992,9993,9996],{"class":469,"line":1418},[151,9994,9995],{"class":473},"Deleted",[151,9997,3073],{"class":503},[56,9999,10000],{"id":1267},"Learning!",[11,10002,10003],{},"Sweet! New learning officially unlocked!",[11,10005,10006],{},[2718,10007],{"alt":2718,"src":10008},"/static/codelab/nvgcp_learning.png",[56,10010,10012],{"id":10011},"whats-next-nim-deployment-on-gke-with-iac-feat-pulumi","What’s Next? 👉 NIM deployment on GKE with IaC (feat. Pulumi!)",[11,10014,10015],{},"That was a fun tutorial! It adds up to a lot of one-off commands that can all be run through Cloud Terminal. Knowing how everything works, spinning everything up would be tedious! For a next step, I want to see if I can spin up this same demo using Infrastructure as Code. I’ll choose Pulumi, it is basically like Terraform, but it can be written in any language which gives lots of advantages over HCL (Hashicorp Configuration Language).",[11,10017,10018,10019,10021],{},"Currently OpenAI’s Codex is free, and I have been using it heavily, in addition to lots of open source coding agent CLI tools like Claude Code, Qwen Code, OpenCode, etc. I’m basically going to prompt Codex with all of the text of the ",[30,10020,1842],{}," tutorial, examples of Pulumi with GCP and GKE, and instructions about what I want. I’m pretty confident that Codex will be able to one-shot this. I suppose I could also use the Gemini CLI!",[11,10023,10024],{},"My goal is basically to:",[76,10026,10027,10034,10040,10043],{},[79,10028,10029,10030,10033],{},"run ",[30,10031,10032],{},"pulumi up"," from my laptop,",[79,10035,10036,10037,10039],{},"configure ",[30,10038,4624],{}," to point to the cluster using outputs,",[79,10041,10042],{},"port-forward and then",[79,10044,10045],{},"make requests to the LLM NIM",[736,10047,10049],{"id":10048},"goal-setting-up-nim-on-gke-with-pulumi","Goal: setting up NIM on GKE with Pulumi",[11,10051,10052],{},"I would like to deploy an NVIDIA NIM on Google Cloud Platform (GCP) with Google Kubernetes Engine (GKE), and I would like to do this using Pulumi.",[11,10054,10055,10056],{},"Here is a Google Kubernetes Engine (GKE) Tutorial: ",[20,10057,10058],{"href":10058,"rel":10059},"https://www.pulumi.com/registry/packages/kubernetes/how-to-guides/gke/",[24],[11,10061,10062,10063,10066],{},"I recently did a tutorial that shows how to set up a NIM on GKE using gcloud commands and kubectl commands, and I would like to automate this as much as possible with Infrastructure as Code. I would like you to use Pulumi, written in TypeScript, and using the pulumi cli. Let's add all of the Pulumi TypeScript code in a top level directory called ",[30,10064,10065],{},"nim-on-gke/"," (please create this directory if it doesn't exist).",[11,10068,10069,10070,10073,10074,10077],{},"The goals are to keep this simple and for demonstration purposes. The primary goal is to demonstrate setting up the infrastructure such as node and gpu pools, deploying the containers and other resources in the GKE cluster and then finally exposing a port with port forwarding and performing LLM inference with a ",[30,10071,10072],{},"curl"," request that makes and OpenAI API call for basic chat completion. Otherwise please use clean, well-written and well-documented code. And include a comprehensive ",[30,10075,10076],{},"nim-on-gke/README.md"," file that documents what the Pulumi code does.",[11,10079,10080],{},"Below I will provide the content of the tutorial and you will make a detailed plan for building a Pulumi stack that can deploy everything. We do not need to follow rigourous cloud engineering practices, we just want to focus on the  essential components that are covered by the below tutorial.",[11,10082,10083],{},"Here are the steps from the tutorial (Steps 5, 6 and 7 from the tutorial are the only steps you need to see):",[700,10085,10086],{"start":517},[79,10087,10088],{},"Create a GKE cluster with GPUs\nOpen Cloud Shell or your terminal.\nSpecify the following parameters:",[11,10090,10091,10092],{},"export PROJECT_ID=",[10093,10094,10095,10096],"your",{"project":464,"id":464},"\nexport REGION=",[10093,10097,10098,10099],{"region":464},"\nexport ZONE=",[10093,10100,10101],{"zone":464},"\nexport CLUSTER_NAME=nim-demo\nexport NODE_POOL_MACHINE_TYPE=g2-standard-16\nexport CLUSTER_MACHINE_TYPE=e2-standard-4\nexport GPU_TYPE=nvidia-l4\nexport GPU_COUNT=1\nPlease note you may have to change the values for NODE_POOL_MACHINE_TYPE, CLUSTER_MACHINE_TYPE and GPU_TYPE based on what type of Compute Instance and GPUs you are using.",[11,10103,10104],{},"Create GKE Cluster:",[11,10106,10107,10108,10110,10111,10113,10114,10116,10117,10119,10120,10122],{},"gcloud container clusters create ${CLUSTER_NAME} ",[1205,10109],{},"\n--project=${PROJECT_ID} ",[1205,10112],{},"\n--location=${ZONE} ",[1205,10115],{},"\n--release-channel=rapid ",[1205,10118],{},"\n--machine-type=${CLUSTER_MACHINE_TYPE} ",[1205,10121],{},"\n--num-nodes=1\nCreate GPU node pool:",[11,10124,10125,10126,10128,10129,10110,10131,10113,10133,10135,10136,10138,10139,10141],{},"gcloud container node-pools create gpupool ",[1205,10127],{},"\n--accelerator type=${GPU_TYPE},count=${GPU_COUNT},gpu-driver-version=latest ",[1205,10130],{},[1205,10132],{},[1205,10134],{},"\n--cluster=${CLUSTER_NAME} ",[1205,10137],{},"\n--machine-type=${NODE_POOL_MACHINE_TYPE} ",[1205,10140],{},"\n--num-nodes=1",[700,10143,10144],{"start":534},[79,10145,10146],{},"Configure NVIDIA NGC API Key\nThe NGC API key allows you to pull custom images from NVIDIA NGC. To specify your key:",[11,10148,10149,10150],{},"export NGC_CLI_API_KEY=\"",[10093,10151,10152],{"ngc":464,"api":464,"key":464},"\"\nThis is the key that was generated, as part of the Prerequisites.",[700,10154,10155],{"start":1413},[79,10156,10157],{},"Deploy and test NVIDIA NIM\nFetch NIM LLM Helm Chart:",[11,10159,10160,10161,10165],{},"helm fetch ",[20,10162,10163],{"href":10163,"rel":10164},"https://helm.ngc.nvidia.com/nim/charts/nim-llm-1.3.0.tgz",[24]," --username='$oauthtoken' --password=$NGC_CLI_API_KEY\nCreate a NIM Namespace:",[11,10167,10168],{},"kubectl create namespace nim\nConfigure secrets:",[11,10170,10171],{},"kubectl create secret docker-registry registry-secret --docker-server=nvcr.io --docker-username='$oauthtoken'     --docker-password=$NGC_CLI_API_KEY -n nim",[11,10173,10174],{},"kubectl create secret generic ngc-api --from-literal=NGC_API_KEY=$NGC_CLI_API_KEY -n nim\nSetup NIM Configuration:",[11,10176,10177,10178],{},"cat \u003C",[10179,10180,10181],"eof",{}," nim_custom_value.yaml\nimage:\nrepository: \"nvcr.io/nim/meta/llama3-8b-instruct\" # container location\ntag: 1.0.0 # NIM version you want to deploy\nmodel:\nngcAPISecret: ngc-api  # name of a secret in the cluster that includes a key named NGC_CLI_API_KEY and is an NGC API key\npersistence:\nenabled: true\nimagePullSecrets:",[76,10183,10184],{},[79,10185,10186,10187,10191],{},"name: registry-secret # name of a secret used to pull nvcr.io images, see ",[20,10188,10189],{"href":10189,"rel":10190},"https://kubernetes.io/docs/tasks/",[24],"    configure-pod-container/pull-image-private-registry/\nEOF\nLaunching NIM deployment:",[11,10193,10194],{},"helm install my-nim nim-llm-1.3.0.tgz -f nim_custom_value.yaml --namespace nim\nVerify NIM pod is running:",[11,10196,10197],{},"kubectl get pods -n nim\nTesting NIM deployment:\nOnce we've verified that our NIM service was deployed successfully, we can make inference requests to see what type of feedback we'll receive from the NIM service. In order to do this, we enable port forwarding on the service to be able to access the NIM from our localhost on port 8000:",[11,10199,10200],{},"kubectl port-forward service/my-nim-nim-llm 8000:8000 -n nim\nNext, we can open another terminal or tab in the cloud shell and try the following request:",[11,10202,8311,10203,10205,10206,10210,10211,10213,10214,10216,10217,10219,10220,10223],{},[1205,10204],{},"\n'",[20,10207,10208],{"href":10208,"rel":10209},"http://localhost:8000/v1/chat/completions",[24],"' ",[1205,10212],{},"\n-H 'accept: application/json' ",[1205,10215],{},"\n-H 'Content-Type: application/json' ",[1205,10218],{},"\n-d '{\n\"messages\": ",[151,10221,10222],{},"\n{\n\"content\": \"You are a polite and respectful chatbot helping people plan a vacation.\",\n\"role\": \"system\"\n},\n{\n\"content\": \"What should I do for a 4 day vacation in Spain?\",\n\"role\": \"user\"\n}\n",",\n\"model\": \"meta/llama3-8b-instruct\",\n\"max_tokens\": 128,\n\"top_p\": 1,\n\"n\": 1,\n\"stream\": false,\n\"stop\": \"\\n\",\n\"frequency_penalty\": 0.0\n}'\nIf you get a chat completion from the NIM service, that means the service is working as expected!",[651,10225],{},[11,10227,10228],{},"Here are some things you might want to consider:",[76,10230,10231,10234,10237,10240,10243],{},[79,10232,10233],{},"We may need to provide environment variables for the project ID in GCP and NGC_API_KEY, otherwise default to what the tutorial uses",[79,10235,10236],{},"For the region and zone please use  REGION=us-east4 and ZONE=us-east4-a",[79,10238,10239],{},"we can just store our Pulumi state in the local file system, we don't need to use Pulumi Cloud",[79,10241,10242],{},"use Pulumi outputs for helpful things like kubectl config that we can use locally for configuring our local kubectl for accessing our GKE Cluster",[79,10244,10245],{},"For Pulumi, we might want to consider splitting things into two stacks: one stack for the cluster and node pools, and another stack for the kubernetes resources",[11,10247,10248],{},"Let me know if you have any other questions or if you need clarification on anything here. Please feel free to look up Pulumi and GKE documentation from the internet.",[11,10250,10251],{},"First let's make a plan, then I'll review and answer any questions, then we will proceed with implementation and code when we are both ready.",[11,10253,10254],{},"You will then write the code, and then finally you will give me precise constructions for the pulumi commands needed for deploying the infrastructure and kubernetes resources, kubectl configuration and port forwarding command, and then LLM inference test commands, finally followed by cleanup commands using pulumi destroy and any other commands needed to ensure things are cleaned up properly.",[11,10256,10257],{},"Please use available CLI tools and Pulumi commands to verify your work at each step if and when possible, such as checking Pulumi outputs. You can also tell me to run commands if that is easier.",[11,10259,10260,10261,10264],{},"Also, by the way, you will be building this as part of an application that has already been written. For now you don't need to reference any other code in the repository, so don't bother exploring other files, just focus on the ",[30,10262,10263],{},"nim-on-gke"," directory in which you are working.",[56,10266,10268],{"id":10267},"openai-codex","OpenAI Codex",[11,10270,10271],{},"I made a new thread in Codex with the above section as the prompt.",[11,10273,10274],{},[2718,10275],{"alt":2718,"src":10276},"/static/codelab/nvgcp_codex.png",[11,10278,10279],{},"Here is what Codex responded with:",[11,10281,10282,10285,10286,10289,10290,10293],{},[15,10283,10284],{},"Skill usage:"," no skill invoked (",[30,10287,10288],{},"skill-creator"," / ",[30,10291,10292],{},"skill-installer"," are not applicable to this task).",[11,10295,10296],{},[15,10297,10298],{},"Proposed Plan",[700,10300,10301],{},[79,10302,10303,10304,10307],{},"Create a clean demo layout under ",[30,10305,10306],{},"/Users/brian/git/waywo/nim-on-gke/"," with two Pulumi TypeScript projects.",[76,10309,10310,10316,10322,10328],{},[79,10311,10312,10315],{},[30,10313,10314],{},"/Users/brian/git/waywo/nim-on-gke/infra/",": GKE cluster + GPU node pool.",[79,10317,10318,10321],{},[30,10319,10320],{},"/Users/brian/git/waywo/nim-on-gke/workloads/",": namespace, secrets, NIM Helm release.",[79,10323,10324,10327],{},[30,10325,10326],{},"/Users/brian/git/waywo/nim-on-gke/README.md",": full setup/deploy/test/cleanup guide.",[79,10329,10330,10331,10334],{},"Optional convenience scripts in ",[30,10332,10333],{},"/Users/brian/git/waywo/nim-on-gke/scripts/"," for command wrappers.",[700,10336,10337],{"start":488},[79,10338,10339,10340,129,10343,10345],{},"Build the ",[15,10341,10342],{},"infra Pulumi project",[30,10344,10314],{},") for GKE.",[76,10347,10348,10394,10413],{},[79,10349,10350,10351],{},"Configurable inputs (with defaults from your tutorial):\n",[76,10352,10353,10359,10364,10369,10374,10379,10384,10389],{},[79,10354,10355,10358],{},[30,10356,10357],{},"projectId"," (required)",[79,10360,10361],{},[30,10362,10363],{},"region=us-east4",[79,10365,10366],{},[30,10367,10368],{},"zone=us-east4-a",[79,10370,10371],{},[30,10372,10373],{},"clusterName=nim-demo",[79,10375,10376],{},[30,10377,10378],{},"clusterMachineType=e2-standard-4",[79,10380,10381],{},[30,10382,10383],{},"gpuNodePoolMachineType=g2-standard-16",[79,10385,10386],{},[30,10387,10388],{},"gpuType=nvidia-l4",[79,10390,10391],{},[30,10392,10393],{},"gpuCount=1",[79,10395,10396,10397],{},"Resources:\n",[76,10398,10399,10406],{},[79,10400,10401,10402,10405],{},"GKE cluster (Rapid release channel, zonal cluster in ",[30,10403,10404],{},"us-east4-a",", 1 CPU node).",[79,10407,10408,10409,10412],{},"Separate GPU node pool (",[30,10410,10411],{},"gpupool",") with L4 GPU config and 1 node.",[79,10414,10415,10416],{},"Outputs:\n",[76,10417,10418,10421,10424,10427],{},[79,10419,10420],{},"Cluster name/zone.",[79,10422,10423],{},"API endpoint and CA.",[79,10425,10426],{},"Generated kubeconfig (Pulumi output).",[79,10428,10429,10432],{},[30,10430,10431],{},"gcloud container clusters get-credentials ..."," helper command string.",[700,10434,10435],{"start":500},[79,10436,10339,10437,129,10440,10442],{},[15,10438,10439],{},"workloads Pulumi project",[30,10441,10320],{},") for Kubernetes resources.",[76,10444,10445,10465,10539],{},[79,10446,10447,10448],{},"Inputs:\n",[76,10449,10450,10456,10462],{},[79,10451,10452,10455],{},[30,10453,10454],{},"infraStackRef"," (cross-stack reference to infra stack).",[79,10457,10458,10461],{},[30,10459,10460],{},"ngcApiKey"," (Pulumi secret).",[79,10463,10464],{},"Optional model/image/chart overrides.",[79,10466,10396,10467],{},[76,10468,10469,10474,10487,10501],{},[79,10470,10471,10472,643],{},"Namespace ",[30,10473,5404],{},[79,10475,10476,10477,10480,10481,10483,10484,643],{},"Docker registry secret ",[30,10478,10479],{},"registry-secret"," for ",[30,10482,4742],{}," with username ",[30,10485,10486],{},"$oauthtoken",[79,10488,10489,10490,10493,10494,10496,10497,10500],{},"Generic secret ",[30,10491,10492],{},"ngc-api"," containing ",[30,10495,4786],{}," (and optionally ",[30,10498,10499],{},"NGC_CLI_API_KEY"," alias for compatibility).",[79,10502,10503,10504,10507,10508,10511,10512],{},"Helm release for ",[30,10505,10506],{},"nim-llm"," chart (version ",[30,10509,10510],{},"1.3.0",") with:\n",[76,10513,10514,10519,10524,10529,10534],{},[79,10515,10516],{},[30,10517,10518],{},"image.repository=nvcr.io/nim/meta/llama3-8b-instruct",[79,10520,10521],{},[30,10522,10523],{},"image.tag=1.0.0",[79,10525,10526],{},[30,10527,10528],{},"model.ngcAPISecret=ngc-api",[79,10530,10531],{},[30,10532,10533],{},"persistence.enabled=true",[79,10535,10536],{},[30,10537,10538],{},"imagePullSecrets=[{name: registry-secret}]",[79,10540,10415,10541],{},[76,10542,10543,10546,10553],{},[79,10544,10545],{},"Namespace, release name, expected service name.",[79,10547,10548,10549,10552],{},"Ready-to-run ",[30,10550,10551],{},"kubectl port-forward ..."," command.",[79,10554,10548,10555,10557],{},[30,10556,10072],{}," inference command template.",[700,10559,10560],{"start":509},[79,10561,10562,10563,643],{},"Add comprehensive docs in ",[30,10564,10326],{},[76,10566,10567,10578,10589,10612],{},[79,10568,10569,10570],{},"Prerequisites:\n",[76,10571,10572],{},[79,10573,10574,10575,643],{},"Pulumi CLI, Node.js, npm, kubectl, gcloud + ",[30,10576,10577],{},"gke-gcloud-auth-plugin",[79,10579,10580,10581],{},"Local state setup:\n",[76,10582,10583],{},[79,10584,10585,10588],{},[30,10586,10587],{},"pulumi login file:///..."," (local backend only).",[79,10590,10591,10592],{},"End-to-end command sequence:\n",[76,10593,10594,10600,10609],{},[79,10595,10596,10597,10599],{},"Install deps, init/select stacks, set config, ",[30,10598,10032],{}," infra then workloads.",[79,10601,10602,10603,187,10606,643],{},"Verify with ",[30,10604,10605],{},"pulumi stack output",[30,10607,10608],{},"kubectl get pods -n nim",[79,10610,10611],{},"Port-forward and run OpenAI-compatible chat completion curl.",[79,10613,10614,10615],{},"Cleanup:\n",[76,10616,10617,10620],{},[79,10618,10619],{},"Destroy workloads first, then infra.",[79,10621,10622],{},"Remove local kube context and optional local state directory.",[700,10624,10625],{"start":517},[79,10626,10627],{},"Verification strategy during implementation.",[76,10629,10630,10633,10640,10643],{},[79,10631,10632],{},"Run TypeScript build checks in each project.",[79,10634,10635,10636,10639],{},"Run ",[30,10637,10638],{},"pulumi preview"," for both projects.",[79,10641,10642],{},"If cloud access isn’t available in this environment, I’ll still validate code structure/types and provide exact commands for you to run on a machine with GCP access.",[79,10644,10645,10646,187,10649,10651,10652,10654],{},"I already checked this environment: ",[30,10647,10648],{},"pulumi",[30,10650,4624],{}," exist, but ",[30,10653,1977],{}," is not installed here, so live GKE deployment verification will need your local machine or Cloud Shell.",[11,10656,10657],{},[15,10658,10659],{},"Key Implementation Decisions (for your approval)",[700,10661,10662,10676,10683,10694],{},[79,10663,10664,10665,129,10668,10671,10672,10675],{},"Use ",[15,10666,10667],{},"two Pulumi projects/states",[30,10669,10670],{},"infra"," then ",[30,10673,10674],{},"workloads",") with cross-stack references.",[79,10677,10678,10679,10682],{},"Keep to ",[15,10680,10681],{},"tutorial-equivalent minimal scope"," (no extra VPC/IAM hardening beyond what’s necessary).",[79,10684,10685,10686,10689,10690,10693],{},"Install NIM via Pulumi Helm ",[30,10687,10688],{},"Release"," directly from NVIDIA Helm repo using NGC key auth (no manual ",[30,10691,10692],{},"helm fetch"," step).",[79,10695,10696,10697,413,10700,10702],{},"Keep runtime access testing as explicit imperative commands (",[30,10698,10699],{},"kubectl port-forward",[30,10701,10072],{},") documented in README.",[11,10704,10705],{},[15,10706,10707],{},"Questions Before I Implement",[700,10709,10710,10728,10738],{},[79,10711,10712,10713,10716,10717,187,10720,10723,10724,10727],{},"Do you want stack names as ",[30,10714,10715],{},"dev"," for both projects (",[30,10718,10719],{},"nim-gke-infra/dev",[30,10721,10722],{},"nim-gke-workloads/dev","), or do you prefer ",[30,10725,10726],{},"demo","?",[79,10729,10730,10731,413,10734,10737],{},"Should I pin chart/model exactly as ",[30,10732,10733],{},"nim-llm 1.3.0",[30,10735,10736],{},"meta/llama3-8b-instruct:1.0.0",", or make these defaults but configurable?",[79,10739,10740,10741,10744,10745,10747],{},"Do you want me to include a small ",[30,10742,10743],{},"Makefile"," in ",[30,10746,10306],{}," for one-command deploy/test/destroy helpers?",[11,10749,10750],{},"If this plan looks right, I’ll proceed with full implementation next.",[651,10752],{},[11,10754,10755],{},"Here's my repsonse:",[11,10757,10758],{},"The key implementation decisions all sound great! Keep it simple, no VPC, no IAM. We might add these later as part of additional work, but don't worry about it for the current implementation.",[11,10760,10761],{},"For your questions:",[700,10763,10764,10770,10773],{},[79,10765,10766,10767,10769],{},"Yes you can use ",[30,10768,10715],{}," as the name for both stacks",[79,10771,10772],{},"Yes, you can hardcode the pinned versions",[79,10774,10775],{},"Sure, you can include a Makefile in the nim-on-gke directory",[651,10777],{},[11,10779,10780,10783],{},[15,10781,10782],{},"Yep! Codex nailed it. Really nailed it."," At least for the specs that I gave it!",[11,10785,10786,10787,106,10789,187,10791,10793],{},"I have ",[30,10788,10648],{},[30,10790,1977],{},[30,10792,4624],{}," installed on my laptop.",[11,10795,10796,10797,208],{},"Before running the commands that Codex prepared, I had a warning from ",[30,10798,1977],{},[459,10800,10803],{"className":10801,"code":10802,"language":997},[995],"~/git/waywo/nim-on-gke$ gcloud auth application-default login\nYour browser has been opened to visit:\n\n\u003Cauth link>\n\n\nCredentials saved to file: [/Users/brian/.config/gcloud/application_default_credentials.json]\n\nThese credentials will be used by any library that requests Application Default Credentials (ADC).\nWARNING:\nCannot find a quota project to add to ADC. You might receive a \"quota exceeded\" or \"API not enabled\" error. Run $ gcloud auth application-default set-quota-project to add a quota project.\n",[30,10804,10802],{"__ignoreMap":464},[11,10806,10807],{},"Gemini was able to help with this:",[459,10809,10812],{"className":10810,"code":10811,"language":997},[995],"▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀\n > OK, my project ID is waywo-487618. Can you help me run that command with this project ID\n▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄\n✦ I will configure your Application Default Credentials (ADC) to use waywo-487618 as the quota project, ensuring Google Cloud\n  knows which project's limits to apply to your API requests.\n\n\n╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮\n│ Action Required                                                                                                             │\n│                                                                                                                             │\n│ ?  Shell gcloud auth application-default set-quota-project waywo-487618 [current working directory /Users/brian/git/waywo/… │\n│                                                                                                                             │\n│ gcloud auth application-default set-quota-project waywo-487618                                                              │\n│ Allow execution of: 'gcloud'?                                                                                               │\n│                                                                                                                             │\n│ ● 1. Allow once                                                                                                             │\n│   2. Allow for this session                                                                                                 │\n│   3. No, suggest changes (esc)                                                                                              │\n│                                                                                                                             │\n╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n",[30,10813,10811],{"__ignoreMap":464},[11,10815,10816,10817,10819],{},"Now I can set up the ",[30,10818,10670],{}," stack:",[459,10821,10824],{"className":10822,"code":10823,"language":997},[995],"~/git/waywo/nim-on-gke$ make infra-up PROJECT_ID=\"$PROJECT_ID\" PULUMI_SECRETS_PASSPHRASE=\"$PULUMI_SECRETS_PASSPHRASE\"\n\nPreviewing update (dev):\n     Type                       Name               Plan       Info\n +   pulumi:pulumi:Stack        nim-gke-infra-dev  create\n     ├─ pulumi:providers:gcp    default_8_41_1                1 warning\n +   ├─ gcp:container:Cluster   nim-demo           create\n +   └─ gcp:container:NodePool  gpupool            create\n\nDiagnostics:\n  pulumi:providers:gcp (default_8_41_1):\n    warning: unable to detect a global setting for GCP Project.\n    Pulumi will rely on per-resource settings for this operation.\n    Set the GCP Project by using:\n        `pulumi config set gcp:project \u003Cproject>`\n    If you would like to disable this warning use:\n        `pulumi config set gcp:disableGlobalProjectWarning true`\n\nOutputs:\n    clusterEndpoint            : output\u003Cstring>\n    clusterName                : \"nim-demo\"\n    clusterNameOutput          : \"nim-demo\"\n    gcpProjectId               : \"waywo-487618\"\n    getCredentialsCommand      : \"gcloud container clusters get-credentials nim-demo --zone us-east4-a --project waywo-487618\"\n    getCredentialsCommandOutput: \"gcloud container clusters get-credentials nim-demo --zone us-east4-a --project waywo-487618\"\n    gpuNodePoolNameOutput      : \"gpupool\"\n    kubeconfig                 : output\u003Cstring>\n    kubeconfigOutput           : output\u003Cstring>\n    regionOutput               : \"us-east4\"\n    zone                       : \"us-east4-a\"\n    zoneOutput                 : \"us-east4-a\"\n\nResources:\n    + 3 to create\n\nDo you want to perform this update? yes\nUpdating (dev):\n     Type                       Name               Status             Info\n +   pulumi:pulumi:Stack        nim-gke-infra-dev  created (399s)\n     ├─ pulumi:providers:gcp    default_8_41_1                        1 warning\n +   ├─ gcp:container:Cluster   nim-demo           created (315s)\n +   └─ gcp:container:NodePool  gpupool            created (81s)\n\nDiagnostics:\n  pulumi:providers:gcp (default_8_41_1):\n    warning: unable to detect a global setting for GCP Project.\n    Pulumi will rely on per-resource settings for this operation.\n    Set the GCP Project by using:\n        `pulumi config set gcp:project \u003Cproject>`\n    If you would like to disable this warning use:\n        `pulumi config set gcp:disableGlobalProjectWarning true`\n\nOutputs:\n    clusterEndpoint            : \"34.186.97.5\"\n    clusterName                : \"nim-demo\"\n    clusterNameOutput          : \"nim-demo\"\n    gcpProjectId               : \"waywo-487618\"\n    getCredentialsCommand      : \"gcloud container clusters get-credentials nim-demo --zone us-east4-a --project waywo-487618\"\n    getCredentialsCommandOutput: \"gcloud container clusters get-credentials nim-demo --zone us-east4-a --project waywo-487618\"\n    gpuNodePoolNameOutput      : \"gpupool\"\n    kubeconfig                 : [secret]\n    kubeconfigOutput           : [secret]\n    regionOutput               : \"us-east4\"\n    zone                       : \"us-east4-a\"\n    zoneOutput                 : \"us-east4-a\"\n\nResources:\n    + 3 created\n\nDuration: 6m40s\n",[30,10825,10823],{"__ignoreMap":464},[11,10827,10828],{},[2718,10829],{"alt":10830,"src":10831},"GKE Cluster","/static/codelab/nvgcp_gke_clusters_dashboard.png",[11,10833,10834,10835,10837],{},"Next step is to configure ",[30,10836,4624],{},", I had a few small hiccups:",[459,10839,10842],{"className":10840,"code":10841,"language":997},[995],"~/git/waywo/nim-on-gke/infra$ gcloud container clusters get-credentials nim-demo --zone us-east4-a --project waywo-487618\nFetching cluster endpoint and auth data.\nCRITICAL: ACTION REQUIRED: gke-gcloud-auth-plugin, which is needed for continued use of kubectl, was not found or is not executable. Install gke-gcloud-auth-plugin for use with kubectl by following https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl#install_plugin\nkubeconfig entry generated for nim-demo.\n~/git/waywo/nim-on-gke/infra$\n",[30,10843,10841],{"__ignoreMap":464},[11,10845,10846],{},"That sounds important!",[459,10848,10851],{"className":10849,"code":10850,"language":997},[995],"~/git/waywo/nim-on-gke/infra$   gcloud components install gke-gcloud-auth-plugin\n\n\nYour current Google Cloud CLI version is: 557.0.0\nInstalling components from version: 557.0.0\n\n┌────────────────────────────────────────────────────────────────┐\n│              These components will be installed.               │\n├────────────────────────────────────────────┬─────────┬─────────┤\n│                    Name                    │ Version │   Size  │\n├────────────────────────────────────────────┼─────────┼─────────┤\n│ gke-gcloud-auth-plugin (Platform Specific) │  0.5.11 │ 3.6 MiB │\n└────────────────────────────────────────────┴─────────┴─────────┘\n\n\nFor the latest full release notes, please visit:\n  https://cloud.google.com/sdk/release_notes\n\nOnce started, canceling this operation may leave your SDK installation in an inconsistent state.\n\nDo you want to continue (Y/n)?\nPerforming in place update...\n\n╔════════════════════════════════════════════════════════════╗\n╠═ Downloading: gke-gcloud-auth-plugin                      ═╣\n╠════════════════════════════════════════════════════════════╣\n╠═ Downloading: gke-gcloud-auth-plugin (Platform Specific)  ═╣\n╠════════════════════════════════════════════════════════════╣\n╠═ Installing: gke-gcloud-auth-plugin                       ═╣\n╠════════════════════════════════════════════════════════════╣\n╠═ Installing: gke-gcloud-auth-plugin (Platform Specific)   ═╣\n╚════════════════════════════════════════════════════════════╝\n\n\nGoogle Cloud CLI works best with Python 3.13 and certain modules.\n\nSetting up virtual environment\nUpdating modules...\nCollecting cryptography==42.0.7\n  Downloading https://github.com/googleapis/enterprise-certificate-proxy/releases/download/v0.3.6/cryptography-42.0.7-cp39-abi3-macosx_10_12_universal2.whl (5.6 MB)\n     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.6/5.6 MB 31.4 MB/s  0:00:00\nRequirement already satisfied: crcmod in /Users/brian/.config/gcloud/virtenv/lib/python3.13/site-packages (1.7)\nRequirement already satisfied: grpcio in /Users/brian/.config/gcloud/virtenv/lib/python3.13/site-packages (1.78.1)\nRequirement already satisfied: pyopenssl==24.2.1 in /Users/brian/.config/gcloud/virtenv/lib/python3.13/site-packages (24.2.1)\nRequirement already satisfied: google_crc32c in /Users/brian/.config/gcloud/virtenv/lib/python3.13/site-packages (1.8.0)\nRequirement already satisfied: certifi in /Users/brian/.config/gcloud/virtenv/lib/python3.13/site-packages (2026.1.4)\nRequirement already satisfied: setuptools in /Users/brian/.config/gcloud/virtenv/lib/python3.13/site-packages (82.0.0)\nRequirement already satisfied: cffi>=1.12 in /Users/brian/.config/gcloud/virtenv/lib/python3.13/site-packages (from cryptography==42.0.7) (2.0.0)\nRequirement already satisfied: typing-extensions~=4.12 in /Users/brian/.config/gcloud/virtenv/lib/python3.13/site-packages (from grpcio) (4.15.0)\nRequirement already satisfied: pycparser in /Users/brian/.config/gcloud/virtenv/lib/python3.13/site-packages (from cffi>=1.12->cryptography==42.0.7) (3.0)\nModules updated.\nVirtual env enabled.\n\nPerforming post processing steps...done.\n\nUpdate done!\n",[30,10852,10850],{"__ignoreMap":464},[11,10854,10855,10856,10858],{},"Now I can run the above ",[30,10857,1977],{}," command again with no critical errors:",[459,10860,10863],{"className":10861,"code":10862,"language":997},[995],"~/git/waywo/nim-on-gke/infra$ gcloud container clusters get-credentials nim-demo --zone us-east4-a --project waywo-487618\nFetching cluster endpoint and auth data.\nkubeconfig entry generated for nim-demo.\n",[30,10864,10862],{"__ignoreMap":464},[11,10866,10867],{},"Nice!",[459,10869,10872],{"className":10870,"code":10871,"language":997},[995],"~/git/waywo/nim-on-gke/infra$ kubectl get no\nNAME                                      STATUS   ROLES    AGE   VERSION\ngke-nim-demo-default-pool-415965ed-823l   Ready    \u003Cnone>   13m   v1.35.0-gke.2398000\ngke-nim-demo-gpupool-9920f098-kz28        Ready    \u003Cnone>   11m   v1.35.0-gke.2398000\n",[30,10873,10871],{"__ignoreMap":464},[11,10875,10876],{},"Next, we can go ahead and deploy our next stack that will create a deployment for our NIM in our GKE cluster:",[459,10878,10881],{"className":10879,"code":10880,"language":997},[995],"~/git/waywo/nim-on-gke$ make workloads-up NGC_API_KEY=\"$NGC_API_KEY\"\n\nPreviewing update (dev):\n     Type                             Name                   Plan       Info\n +   pulumi:pulumi:Stack              nim-gke-workloads-dev  create     1 error\n     └─ pulumi:pulumi:StackReference  nim-gke-infra/dev                 1 error\n\nDiagnostics:\n  pulumi:pulumi:StackReference (nim-gke-infra/dev):\n    error: Preview failed: organization name must be 'organization'\n\n  pulumi:pulumi:Stack (nim-gke-workloads-dev):\n    error: preview failed\n\nResources:\n    + 1 to create\n\nmake: *** [workloads-up] Error 255\n",[30,10882,10880],{"__ignoreMap":464},[11,10884,10885],{},"This resulted in an error!",[11,10887,10888],{},"Codex made a quick for for that, it was related to some naming issue.",[11,10890,10891,10892,10894],{},"Now the preview works for the ",[30,10893,10674],{}," stack!",[459,10896,10899],{"className":10897,"code":10898,"language":997},[995],"Previewing update (dev):\n     Type                              Name                   Plan\n +   pulumi:pulumi:Stack               nim-gke-workloads-dev  create\n +   ├─ pulumi:providers:kubernetes    gke-provider           create\n +   ├─ kubernetes:core/v1:Namespace   nim-namespace          create\n +   ├─ kubernetes:core/v1:Secret      registry-secret        create\n +   ├─ kubernetes:core/v1:Secret      ngc-api                create\n +   └─ kubernetes:helm.sh/v3:Release  nim-llm-release        create\n\nOutputs:\n    curlCommand             : \"curl -X POST \\\\\\n  \\\"http://localhost:8000/v1/chat/completions\\\" \\\\\\n  -H \\\"accept: application/json\\\" \\\\\\n  -H \\\"Content-Type: application/json\\\" \\\\\\n  -d '{\\n  \\\"messages\\\": [\\n    {\\n      \\\"content\\\": \\\"You are a polite and respectful chatbot helping people plan a vacation.\\\",\\n      \\\"role\\\": \\\"system\\\"\\n    },\\n    {\\n      \\\"content\\\": \\\"What should I do for a 4 day vacation in Spain?\\\",\\n      \\\"role\\\": \\\"user\\\"\\n    }\\n  ],\\n  \\\"model\\\": \\\"meta/llama3-8b-instruct\\\",\\n  \\\"max_tokens\\\": 128,\\n  \\\"top_p\\\": 1,\\n  \\\"n\\\": 1,\\n  \\\"stream\\\": false,\\n  \\\"stop\\\": \\\"\\\\n\\\",\\n  \\\"frequency_penalty\\\": 0.0\\n}'\"\n    curlCommandOutput       : \"curl -X POST \\\\\\n  \\\"http://localhost:8000/v1/chat/completions\\\" \\\\\\n  -H \\\"accept: application/json\\\" \\\\\\n  -H \\\"Content-Type: application/json\\\" \\\\\\n  -d '{\\n  \\\"messages\\\": [\\n    {\\n      \\\"content\\\": \\\"You are a polite and respectful chatbot helping people plan a vacation.\\\",\\n      \\\"role\\\": \\\"system\\\"\\n    },\\n    {\\n      \\\"content\\\": \\\"What should I do for a 4 day vacation in Spain?\\\",\\n      \\\"role\\\": \\\"user\\\"\\n    }\\n  ],\\n  \\\"model\\\": \\\"meta/llama3-8b-instruct\\\",\\n  \\\"max_tokens\\\": 128,\\n  \\\"top_p\\\": 1,\\n  \\\"n\\\": 1,\\n  \\\"stream\\\": false,\\n  \\\"stop\\\": \\\"\\\\n\\\",\\n  \\\"frequency_penalty\\\": 0.0\\n}'\"\n    helmReleaseName         : \"my-nim\"\n    infraStackReference     : \"organization/nim-gke-infra/dev\"\n    listPodsCommand         : \"kubectl get pods -n nim\"\n    listServicesCommand     : \"kubectl get svc -n nim\"\n    namespaceName           : \"nim\"\n    nimServiceName          : \"my-nim-nim-llm\"\n    portForwardCommand      : \"kubectl port-forward service/my-nim-nim-llm 8000:8000 -n nim\"\n    portForwardCommandOutput: \"kubectl port-forward service/my-nim-nim-llm 8000:8000 -n nim\"\n\nResources:\n    + 6 to create\n\nDo you want to perform this update?  [Use arrows to move, type to filter]\n  yes\n> no\n  details\n",[30,10900,10898],{"__ignoreMap":464},[11,10902,10903],{},"I got another error trying to deploy this stack:",[459,10905,10908],{"className":10906,"code":10907,"language":997},[995],"Do you want to perform this update? yes\nUpdating (dev):\n     Type                             Name                   Status                  Info\n +   pulumi:pulumi:Stack              nim-gke-workloads-dev  **creating failed**     1 error\n +   ├─ pulumi:providers:kubernetes   gke-provider           created (0.00s)\n +   └─ kubernetes:core/v1:Namespace  nim-namespace          **creating failed**     1 error\n\nDiagnostics:\n  pulumi:pulumi:Stack (nim-gke-workloads-dev):\n    error: update failed\n\n  kubernetes:core/v1:Namespace (nim-namespace):\n    error: configured Kubernetes cluster is unreachable: unable to load schema information from the API server: Get \"https://34.186.97.5/openapi/v2?timeout=32s\": getting credentials: exec: executable gke-gcloud-auth-plugin not found\n\n    It looks like you are trying to use a client-go credential plugin that is not installed.\n\n    To learn more about this feature, consult the documentation available at:\n          https://kubernetes.io/docs/reference/access-authn-authz/authentication/#client-go-credential-plugins\n\n    Install gke-gcloud-auth-plugin and configure gcloud credentials.\n\nResources:\n    + 2 created\n\nDuration: 2s\n\nmake: *** [workloads-up] Error 255\n",[30,10909,10907],{"__ignoreMap":464},[11,10911,10912,10913,10916],{},"Codex made a few changes to help resolve my gke auth plugin issue. Then I tried deploying the ",[30,10914,10915],{},"workload"," stack again:",[459,10918,10921],{"className":10919,"code":10920,"language":997},[995],"Previewing update (dev):\n     Type                              Name                   Plan\n     pulumi:pulumi:Stack               nim-gke-workloads-dev\n ~   ├─ pulumi:providers:kubernetes    gke-provider           update\n +   ├─ kubernetes:core/v1:Namespace   nim-namespace          create\n +   ├─ kubernetes:core/v1:Secret      ngc-api                create\n +   ├─ kubernetes:core/v1:Secret      registry-secret        create\n +   └─ kubernetes:helm.sh/v3:Release  nim-llm-release        create\n\nOutputs:\n  + curlCommand             : \"curl -X POST \\\\\\n  \\\"http://localhost:8000/v1/chat/completions\\\" \\\\\\n  -H \\\"accept: application/json\\\" \\\\\\n  -H \\\"Content-Type: application/json\\\" \\\\\\n  -d '{\\n  \\\"messages\\\": [\\n    {\\n      \\\"content\\\": \\\"You are a polite and respectful chatbot helping people plan a vacation.\\\",\\n      \\\"role\\\": \\\"system\\\"\\n    },\\n    {\\n      \\\"content\\\": \\\"What should I do for a 4 day vacation in Spain?\\\",\\n      \\\"role\\\": \\\"user\\\"\\n    }\\n  ],\\n  \\\"model\\\": \\\"meta/llama3-8b-instruct\\\",\\n  \\\"max_tokens\\\": 128,\\n  \\\"top_p\\\": 1,\\n  \\\"n\\\": 1,\\n  \\\"stream\\\": false,\\n  \\\"stop\\\": \\\"\\\\n\\\",\\n  \\\"frequency_penalty\\\": 0.0\\n}'\"\n  + curlCommandOutput       : \"curl -X POST \\\\\\n  \\\"http://localhost:8000/v1/chat/completions\\\" \\\\\\n  -H \\\"accept: application/json\\\" \\\\\\n  -H \\\"Content-Type: application/json\\\" \\\\\\n  -d '{\\n  \\\"messages\\\": [\\n    {\\n      \\\"content\\\": \\\"You are a polite and respectful chatbot helping people plan a vacation.\\\",\\n      \\\"role\\\": \\\"system\\\"\\n    },\\n    {\\n      \\\"content\\\": \\\"What should I do for a 4 day vacation in Spain?\\\",\\n      \\\"role\\\": \\\"user\\\"\\n    }\\n  ],\\n  \\\"model\\\": \\\"meta/llama3-8b-instruct\\\",\\n  \\\"max_tokens\\\": 128,\\n  \\\"top_p\\\": 1,\\n  \\\"n\\\": 1,\\n  \\\"stream\\\": false,\\n  \\\"stop\\\": \\\"\\\\n\\\",\\n  \\\"frequency_penalty\\\": 0.0\\n}'\"\n  + helmReleaseName         : \"my-nim\"\n  + infraStackReference     : \"organization/nim-gke-infra/dev\"\n  + listPodsCommand         : \"kubectl get pods -n nim\"\n  + listServicesCommand     : \"kubectl get svc -n nim\"\n  + namespaceName           : \"nim\"\n  + nimServiceName          : \"my-nim-nim-llm\"\n  + portForwardCommand      : \"kubectl port-forward service/my-nim-nim-llm 8000:8000 -n nim\"\n  + portForwardCommandOutput: \"kubectl port-forward service/my-nim-nim-llm 8000:8000 -n nim\"\n\nResources:\n    + 4 to create\n    ~ 1 to update\n    5 changes. 1 unchanged\n\nDo you want to perform this update? yes\nUpdating (dev):\n     Type                              Name                   Status\n     pulumi:pulumi:Stack               nim-gke-workloads-dev\n ~   ├─ pulumi:providers:kubernetes    gke-provider           updated (0.00s)\n +   ├─ kubernetes:core/v1:Namespace   nim-namespace          created (0.08s)\n +   ├─ kubernetes:core/v1:Secret      ngc-api                created (0.06s)\n +   ├─ kubernetes:core/v1:Secret      registry-secret        created (0.06s)\n +   └─ kubernetes:helm.sh/v3:Release  nim-llm-release        created (244s)\n\nOutputs:\n  + curlCommand             : \"curl -X POST \\\\\\n  \\\"http://localhost:8000/v1/chat/completions\\\" \\\\\\n  -H \\\"accept: application/json\\\" \\\\\\n  -H \\\"Content-Type: application/json\\\" \\\\\\n  -d '{\\n  \\\"messages\\\": [\\n    {\\n      \\\"content\\\": \\\"You are a polite and respectful chatbot helping people plan a vacation.\\\",\\n      \\\"role\\\": \\\"system\\\"\\n    },\\n    {\\n      \\\"content\\\": \\\"What should I do for a 4 day vacation in Spain?\\\",\\n      \\\"role\\\": \\\"user\\\"\\n    }\\n  ],\\n  \\\"model\\\": \\\"meta/llama3-8b-instruct\\\",\\n  \\\"max_tokens\\\": 128,\\n  \\\"top_p\\\": 1,\\n  \\\"n\\\": 1,\\n  \\\"stream\\\": false,\\n  \\\"stop\\\": \\\"\\\\n\\\",\\n  \\\"frequency_penalty\\\": 0.0\\n}'\"\n  + curlCommandOutput       : \"curl -X POST \\\\\\n  \\\"http://localhost:8000/v1/chat/completions\\\" \\\\\\n  -H \\\"accept: application/json\\\" \\\\\\n  -H \\\"Content-Type: application/json\\\" \\\\\\n  -d '{\\n  \\\"messages\\\": [\\n    {\\n      \\\"content\\\": \\\"You are a polite and respectful chatbot helping people plan a vacation.\\\",\\n      \\\"role\\\": \\\"system\\\"\\n    },\\n    {\\n      \\\"content\\\": \\\"What should I do for a 4 day vacation in Spain?\\\",\\n      \\\"role\\\": \\\"user\\\"\\n    }\\n  ],\\n  \\\"model\\\": \\\"meta/llama3-8b-instruct\\\",\\n  \\\"max_tokens\\\": 128,\\n  \\\"top_p\\\": 1,\\n  \\\"n\\\": 1,\\n  \\\"stream\\\": false,\\n  \\\"stop\\\": \\\"\\\\n\\\",\\n  \\\"frequency_penalty\\\": 0.0\\n}'\"\n  + helmReleaseName         : \"my-nim\"\n  + infraStackReference     : \"organization/nim-gke-infra/dev\"\n  + listPodsCommand         : \"kubectl get pods -n nim\"\n  + listServicesCommand     : \"kubectl get svc -n nim\"\n  + namespaceName           : \"nim\"\n  + nimServiceName          : \"my-nim-nim-llm\"\n  + portForwardCommand      : \"kubectl port-forward service/my-nim-nim-llm 8000:8000 -n nim\"\n  + portForwardCommandOutput: \"kubectl port-forward service/my-nim-nim-llm 8000:8000 -n nim\"\n\nResources:\n    + 4 created\n    ~ 1 updated\n    5 changes. 1 unchanged\n\nDuration: 4m8s\n",[30,10922,10920],{"__ignoreMap":464},[11,10924,10925],{},"Great, I was able to see the pods coming up while Pulumi was deploying the stack:",[459,10927,10930],{"className":10928,"code":10929,"language":997},[995],"~$ kubectl get pods -n nim\nNAME               READY   STATUS    RESTARTS   AGE\nmy-nim-nim-llm-0   0/1     Running   0          3m9s\n",[30,10931,10929],{"__ignoreMap":464},[11,10933,10934],{},"Now I can port forward and then try running inference on the NIM!",[459,10936,10939],{"className":10937,"code":10938,"language":997},[995],"~/git/waywo/nim-on-gke$ kubectl port-forward service/my-nim-nim-llm 8000:8000 -n nim\nForwarding from 127.0.0.1:8000 -> 8000\nForwarding from [::1]:8000 -> 8000\n",[30,10940,10938],{"__ignoreMap":464},[11,10942,10943],{},"This time I'll configure my local instance of Open WebUI to point to the NIM:",[11,10945,10946],{},[2718,10947],{"alt":10948,"src":10949},"Open WebUI Config","/static/codelab/nvgcp_openwebui.png",[11,10951,10952],{},"Now let's ask it for a random fact about the Roman Empire:",[11,10954,10955],{},[2718,10956],{"alt":10948,"src":10957},"/static/codelab/nvgcp_openwebui_1.png",[459,10959,10962],{"className":10960,"code":10961,"language":997},[995],"~/git/waywo/nim-on-gke$ kubectl port-forward service/my-nim-nim-llm 8000:8000 -n nim\nForwarding from 127.0.0.1:8000 -> 8000\nForwarding from [::1]:8000 -> 8000\nHandling connection for 8000\nHandling connection for 8000\nHandling connection for 8000\nHandling connection for 8000\nHandling connection for 8000\n",[30,10963,10961],{"__ignoreMap":464},[11,10965,10966],{},"Great, everything works. Let's not forget to clean up. We don't want to leave a cloud GPU instance running! And we need to make sure that our commands for destroying the stacks and associated infrastructure don't have any issues.",[11,10968,10969],{},[2718,10970],{"alt":10971,"src":10972},"Pulumi destroy","/static/codelab/nvgcp_pulumi_destroy.png",[459,10974,10977],{"className":10975,"code":10976,"language":997},[995],"Resources:\n    - 3 to delete\n\nDestroying (dev):\n     Type                       Name               Status\n -   pulumi:pulumi:Stack        nim-gke-infra-dev  deleted (0.00s)\n -   ├─ gcp:container:NodePool  gpupool            deleted (71s)\n -   └─ gcp:container:Cluster   nim-demo           deleted (232s)\n\nOutputs:\n  - clusterEndpoint            : \"34.186.97.5\"\n  - clusterName                : \"nim-demo\"\n  - clusterNameOutput          : \"nim-demo\"\n  - gcpProjectId               : \"waywo-487618\"\n  - getCredentialsCommand      : \"gcloud container clusters get-credentials nim-demo --zone us-east4-a --project waywo-487618\"\n  - getCredentialsCommandOutput: \"gcloud container clusters get-credentials nim-demo --zone us-east4-a --project waywo-487618\"\n  - gpuNodePoolNameOutput      : \"gpupool\"\n  - kubeconfig                 : [secret]\n  - kubeconfigOutput           : [secret]\n  - regionOutput               : \"us-east4\"\n  - zone                       : \"us-east4-a\"\n  - zoneOutput                 : \"us-east4-a\"\n\nResources:\n    - 3 deleted\n\nDuration: 5m4s\n\nThe resources in the stack have been deleted, but the history and configuration associated with the stack are still maintained.\nIf you want to remove the stack completely, run `pulumi stack rm dev`.\n~/git/waywo/nim-on-gke/infra$ pulumi stack rm dev\nThis will permanently remove the 'dev' stack!\nPlease confirm that this is what you'd like to do by typing `dev`: dev\nStack 'dev' has been removed!\n~/git/waywo/nim-on-gke/infra$\n",[30,10978,10976],{"__ignoreMap":464},[11,10980,10981],{},"Awesome! We now have a way to spin up NIMs on GKE using Pulumi with just a few commands that run in under 10 minutes! This keeps the custom CLI commands to a minimum and captures all of the dependencies in code while still being easily configurable via environment variables.",[11,10983,10984,10985,10987,10988,643],{},"The code for this can be found in my ",[30,10986,9870],{}," repo: ",[20,10989,10990],{"href":10990,"rel":10991},"https://github.com/briancaffey/waywo",[24],[56,10993,10995],{"id":10994},"security-best-practices-disclaimers","Security Best Practices & Disclaimers",[11,10997,10998,11001,11002,11005],{},[15,10999,11000],{},"Security Note:"," This tutorial uses environment variables for configuration, which is good practice. Never commit ",[30,11003,11004],{},".env"," files with real credentials to version control. Use GCP IAM roles and service accounts for production deployments.",[11,11007,11008,11011,11012,11015],{},[15,11009,11010],{},"Version Disclaimer:"," The Kubernetes version shown (1.35.0) is current as of February 2026. Check ",[20,11013,1117],{"href":1115,"rel":11014},[24]," for the latest versions. For production environments, consider using the \"stable\" or \"regular\" release channels instead of \"rapid\" for better stability.",[11,11017,11018,11021,11022,11027],{},[15,11019,11020],{},"Project-Specific Details:"," The project ID and project number shown in this tutorial are from my test environment. You'll see your own values when running these commands in your GCP account. Create a new project in the ",[20,11023,11026],{"href":11024,"rel":11025},"https://console.cloud.google.com",[24],"Google Cloud Console"," before following this tutorial.",[56,11029,11031],{"id":11030},"now-whats-next","Now what's next?",[11,11033,11034,11035,11037],{},"Pulumi and other \"infrastructure as code\" (IaC) tools are really great for automating repeatable, predictable and configurable cloud resources. Fronteir AI coding agents are incredibly good at using IaC tools to implement cloud architecture, and it was a great learning experience for me to learn about running NIMs on GCP/GKE by first doing the ",[30,11036,1842],{}," CodeLab. Implementing the same simple concept with IaC is a good stepping stone for deploying more complex applications.",[11,11039,11040,11041,11043],{},"About one month ago I was gifted a Claude Code Max 20 subscription. This is the top tier plan from Anthropic with usage limits that are hard to hit. One of the things I built with Claude code is an app called ",[30,11042,9870],{}," that uses AI to explore \"What are you working on?\" posts from Hacker News. Claude Code's planning mode for some reason allowed me to become a lot more ambitious when making plans for what to develop. I previously used Cursor and I thought it was amazing, but like a lot of people, something about Claude Code unlocked something in how I can build apps with AI coding agents that I wasn't getting with cursor. I still have a lot to learn about being more productive, you just have to embrace the exponentials! I'm still pretty hands on, and I like tight feedback loops between me and the agents that write the code. waywo used NVIDIA Nemotron models, most of which are available as NIMs, that I ran across a few different computers with NVIDIA RTX cards, and a DGX Spark for the Nemotron 3 Nano reasoning LLM. The models included:",[76,11045,11046,11052,11058,11064,11070,11076],{},[79,11047,11048,11051],{},[30,11049,11050],{},"nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16"," (LLM)",[79,11053,11054,11057],{},[30,11055,11056],{},"nvidia/llama-embed-nemotron-8b"," (for RAG/semantic search)",[79,11059,11060,11063],{},[30,11061,11062],{},"nvidia/llama-nemotron-rerank-1b-v2"," (also for RAG/semantic search)",[79,11065,11066,11069],{},[30,11067,11068],{},"nvidia/Nemotron-Content-Safety-Reasoning-4B"," (for filtering unrelated/inappropriate messages)",[79,11071,11072,11075],{},[30,11073,11074],{},"nvidia/nemotron-speech-streaming-en-0.6b"," (automated speech recognition)",[79,11077,11078,11081],{},[30,11079,11080],{},"nvidia/magpie_tts_multilingual_357m"," (text-to-speech)",[11,11083,11084],{},"Waywo also has a FastAPI server, celery worker and redis server. It uses SQLite for the database, where it stores text embeddings created with the Nemotron embedding models.",[11,11086,11087],{},"Since my Claude Code Max 20 gifted subscription ended I have been testing out Codex (it is free until March 2!) and this model too has a different feel that I'm coming to like a lot. It takes longer for tasks to finish, but I really like the results! I also really like Qwen Code and Gemini CLI with their respective models that are currently offered for free. I use these for simpler tasks that are not as long-horizon as what Codex or Claude Code can handle.",[11,11089,11090],{},"With the remaining time on my free Codex access I would love to have it build out a full GKE-based deployment for Waywo using Pulumi that can handle multiple NIMs for inference as well as API and worker servers, and other services like redis. It is fun to think about how I could build this out. Writing a detailed description of the application architecture in testable Milestones is a great way to develop with the models. The main thing I have to do is get out of the way of the coding agents. Let them build and run GitHub Actions for deploying infrastructure and application changes. Maybe I'll be doing this with a crustacean-themed Telegram bot from my Meta Ray-Ban Glasses?",[589,11092,11093],{},"html pre.shiki code .sC2Qs, html code.shiki .sC2Qs{--shiki-default:#D73A49;--shiki-dark:#F97583;--shiki-sepia:#F92672}html pre.shiki code .sMOD_, html code.shiki .sMOD_{--shiki-default:#24292E;--shiki-dark:#E1E4E8;--shiki-sepia:#F8F8F2}html pre.shiki code .s7F3e, html code.shiki .s7F3e{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#AE81FF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html .sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html.sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html pre.shiki code .srTi1, html code.shiki .srTi1{--shiki-default:#6F42C1;--shiki-dark:#B392F0;--shiki-sepia:#A6E22E}html pre.shiki code .sstjo, html code.shiki .sstjo{--shiki-default:#032F62;--shiki-dark:#9ECBFF;--shiki-sepia:#E6DB74}html pre.shiki code .sTrkL, html code.shiki .sTrkL{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#66D9EF}html pre.shiki code .sinWB, html code.shiki .sinWB{--shiki-default:#032F62;--shiki-dark:#9ECBFF;--shiki-sepia:#F8F8F2}html pre.shiki code .s-m8C, html code.shiki .s-m8C{--shiki-default:#005CC5;--shiki-default-font-style:inherit;--shiki-dark:#79B8FF;--shiki-dark-font-style:inherit;--shiki-sepia:#66D9EF;--shiki-sepia-font-style:italic}html pre.shiki code .sCZoN, html code.shiki .sCZoN{--shiki-default:#032F62;--shiki-dark:#9ECBFF;--shiki-sepia:#CFCFC2}html pre.shiki code .s8-w5, html code.shiki .s8-w5{--shiki-default:#6A737D;--shiki-dark:#6A737D;--shiki-sepia:#88846F}html pre.shiki code .st05x, html code.shiki .st05x{--shiki-default:#B31D28;--shiki-default-font-style:italic;--shiki-dark:#FDAEB7;--shiki-dark-font-style:italic;--shiki-sepia:#F44747;--shiki-sepia-font-style:inherit}",{"title":464,"searchDepth":488,"depth":488,"links":11095},[11096,11115,11116,11119,11120,11121],{"id":1855,"depth":488,"text":1856,"children":11097},[11098,11099,11100,11101,11102,11103,11104,11105,11106,11107,11108,11109,11110,11111,11112,11113,11114],{"id":1966,"depth":500,"text":1967},{"id":2042,"depth":500,"text":2043},{"id":2376,"depth":500,"text":2377},{"id":2505,"depth":500,"text":2506},{"id":2656,"depth":500,"text":2657},{"id":2953,"depth":500,"text":2954},{"id":3202,"depth":500,"text":3203},{"id":3846,"depth":500,"text":3847},{"id":4534,"depth":500,"text":4535},{"id":4555,"depth":500,"text":4556},{"id":4613,"depth":500,"text":4614},{"id":4635,"depth":500,"text":2954},{"id":4658,"depth":500,"text":4659},{"id":4811,"depth":500,"text":4812},{"id":4965,"depth":500,"text":4966},{"id":9860,"depth":500,"text":9861},{"id":9874,"depth":500,"text":9875},{"id":1267,"depth":488,"text":10000},{"id":10011,"depth":488,"text":10012,"children":11117},[11118],{"id":10048,"depth":500,"text":10049},{"id":10267,"depth":488,"text":10268},{"id":10994,"depth":488,"text":10995},{"id":11030,"depth":488,"text":11031},"An overview of my experience with the Google CodeLabs tutorial for deploying an AI model on GKE with NVIDIA NIM",[11124],{"link":11125,"site":11126},"https://x.com/briancaffey/status/2025106078543888633","x","/static/codelab/nvidia_google.png",{},"/2026/02/20/nvidia-nim-on-google-cloud-gcp",{"title":1829,"description":11122},"2026/02/20/nvidia-nim-on-google-cloud-gcp",[11133,614,615,5404,11134,11135,621],"nvidia","gcp","gke","gnqKToOFEwiAG2yTQPe9aDCtqCPQG_fLm_6yBKX97NE",{"id":11138,"title":11139,"body":11140,"comments":609,"date":11793,"description":11794,"draft":602,"extension":605,"external":606,"image":11795,"meta":11796,"navigation":609,"path":11797,"seo":11798,"stem":11799,"tags":11800,"__hash__":11806},"blog/2025/09/09/hnfm-openai-gpt-oss-hackathon-project-hacker-news-ai-podcast-app.md","hnfm: Building a Local-First AI Podcast Generator for Hacker News",{"type":8,"value":11141,"toc":11773},[11142,11144,11147,11158,11160,11164,11167,11174,11177,11215,11218,11220,11224,11227,11342,11345,11347,11351,11358,11436,11443,11445,11449,11455,11487,11490,11501,11511,11513,11517,11520,11524,11535,11539,11542,11620,11623,11625,11629,11661,11663,11667,11670,11673,11677,11685,11689,11700,11704,11715,11719,11722,11726,11733,11747,11750,11753,11760,11763,11766,11770],[56,11143,632],{"id":631},[11,11145,11146],{},"hnfm is my hackathon project for the OpenAI Open Model Hackathon. It's an application that takes Hacker News articles and automatically transforms them into podcast-style videos using a full suite of AI models.",[11,11148,11149,11150,11153,11154,11157],{},"The idea is simple: turn text-heavy news into a multimedia experience with ",[15,11151,11152],{},"summarization, narration, imagery, and subtitles",". But behind the scenes, hnfm is also a deep experiment in ",[15,11155,11156],{},"local AI inference"," - running everything on my own hardware instead of relying on cloud services. I wanted to see how far I could push open-source models for end-to-end content creation.",[651,11159],{},[56,11161,11163],{"id":11162},"motivation","Motivation",[11,11165,11166],{},"I spend a lot of time on Hacker News. The community constantly surfaces interesting articles, clever projects, and thoughtful discussions. But it's not always easy to keep up.",[11,11168,11169,11170,11173],{},"I also enjoy short-form podcasts and clips. They're easier to consume while multitasking, and they bring a different kind of engagement compared to reading. That's where hnfm came in. I wanted to ",[15,11171,11172],{},"reimagine the Hacker News experience"," by creating a kind of radio show or podcast that could automatically generate short episodes, complete with voices, visuals, and subtitles.",[11,11175,11176],{},"Some guiding motivations for hnfm:",[76,11178,11179,11191,11197,11203,11209],{},[79,11180,11181,11184,11185,187,11188,643],{},[15,11182,11183],{},"Richer formats",": Instead of just reading, I can ",[51,11186,11187],{},"listen",[51,11189,11190],{},"watch",[79,11192,11193,11196],{},[15,11194,11195],{},"Local-first AI",": Everything runs on my own hardware, with no reliance on cloud APIs.",[79,11198,11199,11202],{},[15,11200,11201],{},"State-of-the-art tools",": I wanted to benchmark what's possible today with free and open-source models for language, speech, and images.",[79,11204,11205,11208],{},[15,11206,11207],{},"Personalization",": Hacker News does a good job surfacing popular stories, but hnfm goes deeper - indexing articles from the \"new\" feed and sorting them by relevance to my own interests.",[79,11210,11211,11214],{},[15,11212,11213],{},"Prosumer ethos",": Not just consuming content, but producing something new from it that I can share and monetize.",[11,11216,11217],{},"The name \"hnfm\" plays on the idea of a radio show for Hacker News - a personalized channel that broadcasts the latest in tech and AI.",[651,11219],{},[56,11221,11223],{"id":11222},"the-pipeline","The Pipeline",[11,11225,11226],{},"hnfm stitches together a chain of AI tasks into a single automated pipeline:",[700,11228,11229,11245,11268,11285,11313,11329],{},[79,11230,11231,11234],{},[15,11232,11233],{},"Fetch and scrape",[76,11235,11236,11239],{},[79,11237,11238],{},"Grab post metadata via the free and public Hacker News Firebase API.",[79,11240,10664,11241,11244],{},[15,11242,11243],{},"Firecrawl"," (powered by a gpt-oss-20b) to scrape and clean article content, returning Markdown.",[79,11246,11247,11250],{},[15,11248,11249],{},"Summarization + script generation",[76,11251,11252,11259,11265],{},[79,11253,11254,11255,11258],{},"Send content to ",[15,11256,11257],{},"gpt-oss",", my local large language model.",[79,11260,11261,11262,643],{},"Summarize the article and generate a ",[15,11263,11264],{},"two-speaker podcast script",[79,11266,11267],{},"Adjust reasoning levels (low/medium/high) depending on content complexity.",[79,11269,11270,11273],{},[15,11271,11272],{},"Text-to-speech narration",[76,11274,11275,11282],{},[79,11276,11277,11278,11281],{},"Pass script segments to ",[15,11279,11280],{},"nari-labs/dia",", an ultra-realistic TTS model.",[79,11283,11284],{},"Save each narration as a WAV file.",[79,11286,11287,11290],{},[15,11288,11289],{},"Image generation",[76,11291,11292,11299,11310],{},[79,11293,11294,11295,11298],{},"Use gpt-oss to create ",[15,11296,11297],{},"visual descriptions"," for each narration segment.",[79,11300,11301,11302,11305,11306,11309],{},"Feed these prompts to ",[15,11303,11304],{},"InvokeAI",", running the ",[15,11307,11308],{},"Flux Krea (dev)"," image model.",[79,11311,11312],{},"Save the generated images alongside the audio.",[79,11314,11315,11318],{},[15,11316,11317],{},"Speech recognition and subtitles",[76,11319,11320,11326],{},[79,11321,10635,11322,11325],{},[15,11323,11324],{},"WhisperX"," for word-level timestamps and speaker diarization.",[79,11327,11328],{},"Output JSON data for subtitle alignment.",[79,11330,11331,11334],{},[15,11332,11333],{},"Assembly",[76,11335,11336,11339],{},[79,11337,11338],{},"Combine narration, images, and subtitles into a polished video using ffmpeg.",[79,11340,11341],{},"Each segment has a matched voice, image, and subtitle track.",[11,11343,11344],{},"The result: a mini-episode podcast complete with narration, visuals, and synced captions.",[651,11346],{},[56,11348,11350],{"id":11349},"architecture-and-tools","Architecture and Tools",[11,11352,11353,11354,11357],{},"At its core, hnfm runs on ",[15,11355,11356],{},"Python"," with a modular architecture. Here's the breakdown:",[76,11359,11360,11379,11407,11420],{},[79,11361,11362,11365],{},[15,11363,11364],{},"Backend Frameworks",[76,11366,11367,11370,11373,11376],{},[79,11368,11369],{},"FastAPI for the API server.",[79,11371,11372],{},"Celery for asynchronous task execution.",[79,11374,11375],{},"Redis as a general-purpose database, message broker and vector database.",[79,11377,11378],{},"LangChain for orchestration and chaining model calls.",[79,11380,11381,11384],{},[15,11382,11383],{},"External Services (running on my home network)",[76,11385,11386,11391,11397,11402],{},[79,11387,11388,11390],{},[15,11389,1628],{}," with gpt-oss-20b (LLM inference, embeddings, reasoning).",[79,11392,11393,11396],{},[15,11394,11395],{},"dia"," for text-to-speech (wrapped with a custom FastAPI server to manage VRAM usage).",[79,11398,11399,11401],{},[15,11400,11304],{}," for image generation (Flux Krea (dev)).",[79,11403,11404,11406],{},[15,11405,11324],{}," for speech recognition (ASR).",[79,11408,11409,11412],{},[15,11410,11411],{},"Frontend",[76,11413,11414,11417],{},[79,11415,11416],{},"Built with Nuxt.js and shadcn components.",[79,11418,11419],{},"Lets me scrape Hacker News, browse articles, trigger generation, and review audio, image and video outputs.",[79,11421,11422,11425],{},[15,11423,11424],{},"Deployment",[76,11426,11427,11430,11433],{},[79,11428,11429],{},"Backend services run in Docker.",[79,11431,11432],{},"Redis handles both task queues and embeddings.",[79,11434,11435],{},"Flower provides monitoring for Celery.",[11,11437,11438,11439,11442],{},"I also created a ",[15,11440,11441],{},"status page"," and service status API to confirm all services are online before generating content.",[651,11444],{},[56,11446,11448],{"id":11447},"why-gpt-oss","Why gpt-oss?",[11,11450,11451,11452,11454],{},"The star of the show is ",[15,11453,11257],{},", a reasoning-first open-source model. What makes it special:",[76,11456,11457,11471,11477],{},[79,11458,11459,11462,11463],{},[15,11460,11461],{},"Reasoning control",": I can tune how much \"thinking\" the model does before responding.",[76,11464,11465,11468],{},[79,11466,11467],{},"Low for quick summarization.",[79,11469,11470],{},"High for creative, nuanced scriptwriting or image scene planning.",[79,11472,11473,11476],{},[15,11474,11475],{},"Large context window",": Configurable up to tens of thousands of tokens - enough to handle long-form articles (I configured a context window of about 30k tokens).",[79,11478,11479,11482,11483,11486],{},[15,11480,11481],{},"Embeddings",": I use gpt-oss's embeddings with Redis to build a ",[15,11484,11485],{},"vector index"," of scraped articles.",[11,11488,11489],{},"This indexing step allows hnfm to:",[76,11491,11492,11495,11498],{},[79,11493,11494],{},"Pull from the high-volume \"new\" feed on Hacker News.",[79,11496,11497],{},"Sort articles based on alignment with my personal interests.",[79,11499,11500],{},"Surface content that might never reach the front page.",[11,11502,11503,11504,187,11507,11510],{},"In other words, gpt-oss powers both ",[15,11505,11506],{},"content creation",[15,11508,11509],{},"content discovery"," in hnfm.",[651,11512],{},[56,11514,11516],{"id":11515},"development-journey","Development Journey",[11,11518,11519],{},"The build process had two phases: quick MVP, then structured rebuild.",[736,11521,11523],{"id":11522},"phase-1-rapid-prototyping","Phase 1: Rapid prototyping",[76,11525,11526,11529,11532],{},[79,11527,11528],{},"Started by wiring services together in Cursor with \"vibe coding.\"",[79,11530,11531],{},"Focused on minimal workflow: scrape → summarize → narrate → image → assemble.",[79,11533,11534],{},"Stored outputs in JSON and local directories.",[736,11536,11538],{"id":11537},"phase-2-structured-architecture","Phase 2: Structured architecture",[11,11540,11541],{},"Once I proved the concept worked, I gutted the initial code and rebuilt with a stronger spec:",[76,11543,11544,11557,11570,11583,11607],{},[79,11545,11546,208,11549],{},[15,11547,11548],{},"Custom APIs",[76,11550,11551,11554],{},[79,11552,11553],{},"Wrapped WhisperX and DIA in FastAPI servers for reliable inference.",[79,11555,11556],{},"Added endpoints for managing VRAM by unloading TTS models between runs.",[79,11558,11559,208,11562],{},[15,11560,11561],{},"Celery tasks",[76,11563,11564,11567],{},[79,11565,11566],{},"Split long-running jobs into queued tasks.",[79,11568,11569],{},"Centralized Redis client logic to avoid duplication across tasks.",[79,11571,11572,208,11575],{},[15,11573,11574],{},"File + Redis dual storage",[76,11576,11577,11580],{},[79,11578,11579],{},"Used Redis for fast data structures.",[79,11581,11582],{},"Stored images, audio, and metadata as files.",[79,11584,11585,208,11588],{},[15,11586,11587],{},"Frontend interface",[76,11589,11590,11593,11596],{},[79,11591,11592],{},"Allowed browsing of ~50 scraped stories at once.",[79,11594,11595],{},"Supported A/B testing multiple scripts for the same article.",[79,11597,11598,11599,11602,11603,11606],{},"Structured data into ",[30,11600,11601],{},"item"," (Hacker News post) and ",[30,11604,11605],{},"segment"," (podcast episode unit).",[79,11608,11609,208,11612],{},[15,11610,11611],{},"Testing & linting",[76,11613,11614,11617],{},[79,11615,11616],{},"Added unit tests and linters for stability.",[79,11618,11619],{},"Iterated with Cursor and Gemini CLI for faster development cycles.",[11,11621,11622],{},"I also Dockerized the backend, which made running multiple services far more manageable.",[651,11624],{},[56,11626,11628],{"id":11627},"lessons-learned","Lessons Learned",[76,11630,11631,11637,11643,11649,11655],{},[79,11632,11633,11636],{},[15,11634,11635],{},"Service health is critical",": With multiple AI services, one offline process can derail the pipeline. My health-check API saved hours of frustration.",[79,11638,11639,11642],{},[15,11640,11641],{},"Reasoning tuning matters",": Some steps benefit from deep thinking (scriptwriting), while others only need quick output (summaries). gpt-oss's reasoning control was invaluable, and no other model gives you this type of control.",[79,11644,11645,11648],{},[15,11646,11647],{},"Prompt iteration is the fun part",": Most improvements came from refining prompts, not debugging code.",[79,11650,11651,11654],{},[15,11652,11653],{},"Keep it simple",": Every time the architecture grew too complex, progress stalled. The best results came from aggressively simplifying, and telling code agents to write simple code that sticks to your requirements. Coding agents like Cursor will try to over-optimize features that add unnecessary complexity.",[79,11656,11657,11660],{},[15,11658,11659],{},"Cursor rules and constraints help",": Encoding best practices into coding agents prevented regressions and kept the codebase clean.",[651,11662],{},[56,11664,11666],{"id":11665},"similar-projects","Similar Projects",[11,11668,11669],{},"There are several other projects attempting to generate automated podcasts from Hacker News. Google's NotebookLM popularized the idea of creating podcasts of your documents, but it is not open source. NVIDIA has an open-source Blueprint called \"pdf-to-podcast\" that was a helpful reference, but it uses Eleven Labs for audio generation.",[11,11671,11672],{},"The AI podcast generation landscape offers several distinct approaches:",[736,11674,11676],{"id":11675},"open-source-solutions","Open Source Solutions:",[76,11678,11679,11682],{},[79,11680,11681],{},"Podcastfy.ai - Open-source Python tool for multi-modal content (websites, PDFs, videos) to multilingual audio conversations",[79,11683,11684],{},"TwoCast - One-click 3-5 minute two-person podcast generator with multiple TTS platform support",[736,11686,11688],{"id":11687},"hacker-news-specialists","Hacker News Specialists:",[76,11690,11691,11694,11697],{},[79,11692,11693],{},"Hackercast - Scrapes and summarizes Hacker Newsletter content using Langchain/GPT-4",[79,11695,11696],{},"Hacker News Recap - Daily AI-generated recaps of top HN posts",[79,11698,11699],{},"Hacker News 每日播报 - Chinese-language daily HN summaries with Edge TTS",[736,11701,11703],{"id":11702},"personalized-automated","Personalized & Automated:",[76,11705,11706,11709,11712],{},[79,11707,11708],{},"Personalized News Generator - Custom podcasts based on user interests and news using Gemini/ElevenLabs",[79,11710,11711],{},"AutoPod - Converts reading lists to podcasts via n8n workflows (as low as $0.20/20min)\nVideo Content:",[79,11713,11714],{},"HN to Video Content - n8n workflow converting HN articles to videos using AI image/video generation",[56,11716,11718],{"id":11717},"generative-video","Generative Video",[11,11720,11721],{},"There are some amazing options for doing local generative video. Wan 2.2 is one such model. This would be interesting to explore, but I decided not to include any video generation in this project. Video generation takes a lot of time compared to image generation and TTS.",[56,11723,11725],{"id":11724},"final-thoughts","Final Thoughts",[11,11727,11728,11729,11732],{},"hnfm is more than a neat hackathon project. It's a ",[15,11730,11731],{},"proof of concept for local-first AI media generation",". On a single RTX 4090, I can run:",[76,11734,11735,11738,11741,11744],{},[79,11736,11737],{},"A reasoning LLM (gpt-oss).",[79,11739,11740],{},"An ultra-realistic TTS model (nari-labs/dia).",[79,11742,11743],{},"A high-quality diffusion model (Flux Krea).",[79,11745,11746],{},"A state-of-the-art ASR system (WhisperX).",[11,11748,11749],{},"All locally. No cloud APIs. No pay-per-call. Just consumer hardware and open-source models.",[11,11751,11752],{},"For me, hnfm represents a \"prosumer\" approach to AI - consuming content from Hacker News, but also producing something new with it. It's commentary, transformation, and reimagining, all in one.",[11,11754,11755,11756,11759],{},"I've made the project open source because I believe experiments like this should be shared. The closed-source AI ecosystem is growing rapidly, but it's vital that we also push forward what's possible with ",[15,11757,11758],{},"open models"," running on personal hardware. It's not just about cost or performance - it's about ownership, freedom, and creative expression.",[11,11761,11762],{},"Building hnfm was challenging, rewarding, and a lot of fun. It's a small glimpse into what's possible with state-of-the-art open source AI running on consumer hardware.",[11,11764,11765],{},"I made hnfm with AI assisted coding, but I think it would be interesting to try to replicate this project with n8n, a low-code/no-code workflow tool that is very popular for AI-powered content creation.",[56,11767,11769],{"id":11768},"video","Video",[11,11771,11772],{},"For the OpenAI hackathon I needed to create a short video that introduces the project. I wrote this article on my blog, submitted it to Hacker News and then used hnfm to generate my submission video.",{"title":464,"searchDepth":488,"depth":488,"links":11774},[11775,11776,11777,11778,11779,11780,11784,11785,11790,11791,11792],{"id":631,"depth":488,"text":632},{"id":11162,"depth":488,"text":11163},{"id":11222,"depth":488,"text":11223},{"id":11349,"depth":488,"text":11350},{"id":11447,"depth":488,"text":11448},{"id":11515,"depth":488,"text":11516,"children":11781},[11782,11783],{"id":11522,"depth":500,"text":11523},{"id":11537,"depth":500,"text":11538},{"id":11627,"depth":488,"text":11628},{"id":11665,"depth":488,"text":11666,"children":11786},[11787,11788,11789],{"id":11675,"depth":500,"text":11676},{"id":11687,"depth":500,"text":11688},{"id":11702,"depth":500,"text":11703},{"id":11717,"depth":488,"text":11718},{"id":11724,"depth":488,"text":11725},{"id":11768,"depth":488,"text":11769},"2025-09-09","hnfm transforms the Hacker News feed into your personalized AI-powered podcast","/static/hnfm/hnfm_cover.png",{},"/2025/09/09/hnfm-openai-gpt-oss-hackathon-project-hacker-news-ai-podcast-app",{"title":11139,"description":11794},"2025/09/09/hnfm-openai-gpt-oss-hackathon-project-hacker-news-ai-podcast-app",[614,615,11801,11802,11803,11804,11805,11257],"tts","flux","nuxt","fastapi","openai","0ESUhjVrIdFVOp_PiJwggySoQFyOAn6Q1fk57492fRs",{"id":11808,"title":11809,"body":11810,"comments":609,"date":12638,"description":12639,"draft":602,"extension":605,"external":606,"image":12640,"meta":12641,"navigation":609,"path":12642,"seo":12643,"stem":12644,"tags":12645,"__hash__":12649},"blog/2025/07/21/upgrading-my-static-nuxt-blog-from-nuxt-3-to-nuxt-4.md","Upgrading my Blog to Nuxt 4",{"type":8,"value":11811,"toc":12625},[11812,11821,11827,11830,11836,11839,11845,11848,11854,11857,11863,11869,11876,11881,11887,11890,11896,11899,11906,11911,11917,11923,11930,11936,11942,11948,11954,11956,11965,11969,11976,11991,11995,11998,12014,12018,12042,12046,12062,12066,12073,12114,12125,12128,12130,12135,12140,12143,12149,12152,12158,12164,12170,12176,12182,12185,12194,12197,12203,12210,12216,12227,12233,12236,12242,12248,12257,12263,12270,12276,12283,12289,12295,12300,12306,12309,12316,12319,12325,12328,12333,12336,12466,12472,12488,12494,12500,12504,12517,12523,12532,12538,12551,12557,12563,12569,12580,12588,12591,12597,12600,12606,12615,12619,12622],[11,11813,11814,11815,11820],{},"This article will share my experience upgrading my Nuxt blog from Nuxt 3 to Nuxt 4. I'm following the official ",[20,11816,11819],{"href":11817,"rel":11818},"https://nuxt.com/docs/4.x/getting-started/upgrade",[24],"Upgrade Guide"," from Nuxt. It will be mostly unfiltered and I'll try to document things as they happen. Let's go!",[11,11822,11823,11826],{},[15,11824,11825],{},"Version Note:"," This article was published in July 2025 (~8 months old) and references specific versions (Nuxt 4.0.0, Node 22.17.1, @nuxtjs/i18n v10, etc.). These packages evolve quickly; check official documentation for current best practices, breaking changes, and updated migration steps before implementing similar upgrades today.",[11,11828,11829],{},"First, I'll do this on a new branch:",[459,11831,11834],{"className":11832,"code":11833,"language":997},[995],"git branch -b nuxt4-upgrade\n",[30,11835,11833],{"__ignoreMap":464},[11,11837,11838],{},"The first step in the migration guide says to run:",[459,11840,11843],{"className":11841,"code":11842,"language":997},[995],"yarn nuxt upgrade\n",[30,11844,11842],{"__ignoreMap":464},[11,11846,11847],{},"This gave me an error.",[459,11849,11852],{"className":11850,"code":11851,"language":997},[995],"error @nuxt/vite-builder@4.0.0: The engine \"node\" is incompatible with this module. Expected version \"^20.19.0 || >=22.12.0\". Got \"20.18.0\"\nerror Found incompatible module.\ninfo Visit https://yarnpkg.com/en/docs/cli/add for documentation about this command.\n\n ERROR  Command failed: yarn add -D nuxt                                                  3:41:22 PM\n\n  at genericNodeError (node:internal/errors:984:15)\n  at wrappedFn (node:internal/errors:538:14)\n  at checkExecSyncError (node:child_process:891:11)\n  at execSync (node:child_process:963:15)\n  at Object.run (node_modules/nuxi/dist/chunks/upgrade.mjs:99:5)\n  at async runCommand$1 (node_modules/nuxi/dist/shared/nuxi.6aad497e.mjs:1648:16)\n  at async runCommand$1 (node_modules/nuxi/dist/shared/nuxi.6aad497e.mjs:1639:11)\n  at async runMain$1 (node_modules/nuxi/dist/shared/nuxi.6aad497e.mjs:1777:7)\n\n\n\n ERROR  Command failed: yarn add -D nuxt                                                  3:41:22 PM\n\nerror Command failed with exit code 1.\ninfo Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.\n",[30,11853,11851],{"__ignoreMap":464},[11,11855,11856],{},"First, I need to upgrade node. I use nvm, so this is easy. Let's see what node versions I have installed:",[459,11858,11861],{"className":11859,"code":11860,"language":997},[995],"~/git/briancaffey.github.io$ nvm list\n->     v20.18.0\n       v22.15.0\ndefault -> 20 (-> v20.18.0)\niojs -> N/A (default)\nunstable -> N/A (default)\nnode -> stable (-> v22.15.0) (default)\nstable -> 22.15 (-> v22.15.0) (default)\nlts/* -> lts/jod (-> v22.15.0)\nlts/argon -> v4.9.1 (-> N/A)\nlts/boron -> v6.17.1 (-> N/A)\nlts/carbon -> v8.17.0 (-> N/A)\nlts/dubnium -> v10.24.1 (-> N/A)\nlts/erbium -> v12.22.12 (-> N/A)\nlts/fermium -> v14.21.3 (-> N/A)\nlts/gallium -> v16.20.2 (-> N/A)\nlts/hydrogen -> v18.20.8 (-> N/A)\nlts/iron -> v20.19.1 (-> N/A)\nlts/jod -> v22.15.0\n",[30,11862,11860],{"__ignoreMap":464},[459,11864,11867],{"className":11865,"code":11866,"language":997},[995],"~/git/briancaffey.github.io$ nvm install 22.17.1\nDownloading and installing node v22.17.1...\nDownloading https://nodejs.org/dist/v22.17.1/node-v22.17.1-darwin-arm64.tar.xz...\n###################################################################################################################################################################################### 100.0%\nComputing checksum with sha256sum\nChecksums matched!\nNow using node v22.17.1 (npm v10.9.2)\n",[30,11868,11866],{"__ignoreMap":464},[11,11870,11871,11872,11875],{},"OK, great, now let's run that ",[30,11873,11874],{},"yarn nuxt upgrade"," again:",[459,11877,11879],{"className":11878,"code":11842,"language":997},[995],[30,11880,11842],{"__ignoreMap":464},[459,11882,11885],{"className":11883,"code":11884,"language":997},[995],"~/git/briancaffey.github.io$ yarn nuxt upgrade\nyarn run v1.22.22\nerror Command \"nuxt\" not found.\ninfo Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.\n",[30,11886,11884],{"__ignoreMap":464},[11,11888,11889],{},"Hmm, I'll try adding nuxt again:",[459,11891,11894],{"className":11892,"code":11893,"language":997},[995],"yarn add nuxt@^4.0.0\n\n...\n✨  Done in 36.03s.\n",[30,11895,11893],{"__ignoreMap":464},[11,11897,11898],{},"OK, great! Now I have Nuxt 4 installed.",[11,11900,11901,11902,11905],{},"The next section is about ",[30,11903,11904],{},"Migrating Using Codemods",". I'll try running this:",[210,11907,11908],{},[11,11909,11910],{},"This command will execute all codemods in sequence, with the option to deselect any that you do not wish to run. Each codemod is also listed below alongside its respective change and can be executed independently.",[459,11912,11915],{"className":11913,"code":11914,"language":997},[995],"yarn dlx codemod@latest nuxt/4/migration-recipe\n",[30,11916,11914],{"__ignoreMap":464},[459,11918,11921],{"className":11919,"code":11920,"language":997},[995],"~/git/briancaffey.github.io$ yarn dlx codemod@latest nuxt/4/migration-recipe\nyarn run v1.22.22\nerror Command \"dlx\" not found.\ninfo Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.\n",[30,11922,11920],{"__ignoreMap":464},[11,11924,11925,11926,11929],{},"OK, I guess I need to add ",[30,11927,11928],{},"dlx"," with yarn first:",[459,11931,11934],{"className":11932,"code":11933,"language":997},[995],"yarn add dlx\n",[30,11935,11933],{"__ignoreMap":464},[459,11937,11940],{"className":11938,"code":11939,"language":997},[995],"success Saved lockfile.\nsuccess Saved 2 new dependencies.\ninfo Direct dependencies\n└─ dlx@0.2.1\ninfo All dependencies\n├─ dlx@0.2.1\n└─ underscore@1.13.7\n✨  Done in 2.52s.\n",[30,11941,11939],{"__ignoreMap":464},[11,11943,11944,11945,11947],{},"But I still got the same error about ",[30,11946,11928],{},", so I asked cursor:",[459,11949,11952],{"className":11950,"code":11951,"language":997},[995],"I'm trying to run this yarn command:\n\nyarn dlx codemod@latest nuxt/4/migration-recipe\n\n\nBut I'm getting an error:\n\nyarn dlx codemod@latest nuxt/4/migration-recipe\nyarn run v1.22.22\nerror Command \"dlx\" not found.\ninfo Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.\n\nHow can I fix this?\n",[30,11953,11951],{"__ignoreMap":464},[651,11955],{},[11,11957,11958,11959,11961,11962,11964],{},"The issue is that you're using Yarn v1.22.22, which doesn't have the ",[30,11960,11928],{}," command. The ",[30,11963,11928],{}," command was introduced in Yarn v2+ (Berry). Here are several ways to fix this:",[56,11966,11968],{"id":11967},"option-1-use-npx-recommended","Option 1: Use npx (Recommended)",[11,11970,11971,11972,11975],{},"Since you're using Yarn v1, you can use ",[30,11973,11974],{},"npx"," instead:",[459,11977,11979],{"className":461,"code":11978,"language":463,"meta":464,"style":464},"npx codemod@latest nuxt/4/migration-recipe\n",[30,11980,11981],{"__ignoreMap":464},[151,11982,11983,11985,11988],{"class":469,"line":470},[151,11984,11974],{"class":473},[151,11986,11987],{"class":481}," codemod@latest",[151,11989,11990],{"class":481}," nuxt/4/migration-recipe\n",[56,11992,11994],{"id":11993},"option-2-use-yarn-create-alternative","Option 2: Use yarn create (Alternative)",[11,11996,11997],{},"You can try using yarn's create command:",[459,11999,12001],{"className":461,"code":12000,"language":463,"meta":464,"style":464},"yarn create codemod@latest nuxt/4/migration-recipe\n",[30,12002,12003],{"__ignoreMap":464},[151,12004,12005,12008,12010,12012],{"class":469,"line":470},[151,12006,12007],{"class":473},"yarn",[151,12009,1550],{"class":481},[151,12011,11987],{"class":481},[151,12013,11990],{"class":481},[56,12015,12017],{"id":12016},"option-3-install-codemod-globally-first","Option 3: Install codemod globally first",[459,12019,12021],{"className":461,"code":12020,"language":463,"meta":464,"style":464},"yarn global add codemod\ncodemod@latest nuxt/4/migration-recipe\n",[30,12022,12023,12035],{"__ignoreMap":464},[151,12024,12025,12027,12030,12032],{"class":469,"line":470},[151,12026,12007],{"class":473},[151,12028,12029],{"class":481}," global",[151,12031,2164],{"class":481},[151,12033,12034],{"class":481}," codemod\n",[151,12036,12037,12040],{"class":469,"line":488},[151,12038,12039],{"class":473},"codemod@latest",[151,12041,11990],{"class":481},[56,12043,12045],{"id":12044},"option-4-use-the-full-path","Option 4: Use the full path",[459,12047,12049],{"className":461,"code":12048,"language":463,"meta":464,"style":464},"yarn exec codemod@latest nuxt/4/migration-recipe\n",[30,12050,12051],{"__ignoreMap":464},[151,12052,12053,12055,12058,12060],{"class":469,"line":470},[151,12054,12007],{"class":473},[151,12056,12057],{"class":481}," exec",[151,12059,11987],{"class":481},[151,12061,11990],{"class":481},[56,12063,12065],{"id":12064},"option-5-upgrade-to-yarn-v2-optional","Option 5: Upgrade to Yarn v2+ (Optional)",[11,12067,12068,12069,12072],{},"If you want to use ",[30,12070,12071],{},"yarn dlx",", you could upgrade to Yarn v2+:",[459,12074,12076],{"className":461,"code":12075,"language":463,"meta":464,"style":464},"# Enable Yarn v2+ for this project\nyarn set version berry\n\n# Then you can use dlx\nyarn dlx codemod@latest nuxt/4/migration-recipe\n",[30,12077,12078,12083,12094,12098,12103],{"__ignoreMap":464},[151,12079,12080],{"class":469,"line":470},[151,12081,12082],{"class":1527},"# Enable Yarn v2+ for this project\n",[151,12084,12085,12087,12089,12091],{"class":469,"line":488},[151,12086,12007],{"class":473},[151,12088,2309],{"class":481},[151,12090,5386],{"class":481},[151,12092,12093],{"class":481}," berry\n",[151,12095,12096],{"class":469,"line":500},[151,12097,1090],{"emptyLinePlaceholder":609},[151,12099,12100],{"class":469,"line":509},[151,12101,12102],{"class":1527},"# Then you can use dlx\n",[151,12104,12105,12107,12110,12112],{"class":469,"line":517},[151,12106,12007],{"class":473},[151,12108,12109],{"class":481}," dlx",[151,12111,11987],{"class":481},[151,12113,11990],{"class":481},[11,12115,12116,12121,12122,12124],{},[15,12117,12118,12119,748],{},"I recommend using Option 1 (",[30,12120,11974],{}," since it's the most straightforward and doesn't require changing your package manager setup. The ",[30,12123,11974],{}," command is available by default with Node.js and will work perfectly for running the codemod.",[11,12126,12127],{},"After running the codemod, you can continue with the manual upgrade steps I outlined in the previous plan.",[651,12129],{},[11,12131,12132,12133,208],{},"OK, I'll try upgrading yarn later, and for now I'll go with using ",[30,12134,11974],{},[459,12136,12138],{"className":12137,"code":11978,"language":997},[995],[30,12139,11978],{"__ignoreMap":464},[11,12141,12142],{},"OK, that worked, now I got the list of options to choose from for my Nuxt 4 upgrade:",[459,12144,12147],{"className":12145,"code":12146,"language":997},[995],"✔ Successfully downloaded \"nuxt/4/migration-recipe\" from the registry.\n? Press Enter to run the selected codemods in order. You can deselect anything you don’t want. (Press\n\u003Cspace> to select, \u003Ca> to toggle all, \u003Ci> to invert selection, and \u003Center> to proceed)\n❯◉ nuxt/4/absolute-watch-path\n ◉ nuxt/4/default-data-error-value\n ◉ nuxt/4/deprecated-dedupe-value\n ◉ nuxt/4/file-structure\n ◉ nuxt/4/shallow-function-reactivity\n ◉ nuxt/4/template-compilation-changes\n",[30,12148,12146],{"__ignoreMap":464},[11,12150,12151],{},"OK! That seems to work! It ran each codemod, for example:",[459,12153,12156],{"className":12154,"code":12155,"language":997},[995],"╭──────────────────────────────────────────────────────────────────────────────────────────────────╮\n│                                                                                                  │\n│                                                                                                  │\n│      Codemod: nuxt/4/absolute-watch-path@1.0.3                                                   │\n│      Target: /Users/brian/git/briancaffey.github.io                                              │\n│                                                                                                  │\n│      Using paths provided by codemod settings                                                    │\n│      Included patterns: **/*.js, **/*.jsx, **/*.ts, **/*.tsx, **/*.vue                           │\n│      Patterns excluded by default: **/*.d.ts, **/node_modules/**/*.*, **/.next/**/*.*,           │\n│      **/dist/**/*.*, **/build/**/*.*, **/.git/**/*.*,                                            │\n│      **/.svn/**/*.*, **/.hg/**/*.*, **/.bzr/**/*.*,                                              │\n│      **/_darcs/**/*.*, **/_MTN/**/*.*, **/_FOSSIL_, **/.fslckout,                                │\n│      **/.view/**/*.*                                                                             │\n│      Patterns excluded from gitignore: **/.vscode/**/*.*, **/node_modules, /logs, **/*.log,      │\n│      **/npm-debug.log*, **/yarn-debug.log*, **/yarn-error.log*,                                  │\n│      **/pids, **/*.pid, **/*.seed, **/*.pid.lock, **/lib-cov,                                    │\n│      **/coverage, **/.nyc_output, **/.grunt, **/bower_components,                                │\n│      **/.lock-wscript, **/build/Release, **/node_modules/**/*.*,                                 │\n│      **/jspm_packages/**/*.*, **/typings/**/*.*, **/.npm,                                        │\n│      **/.eslintcache, **/.node_repl_history, **/*.tgz,                                           │\n│      **/.yarn-integrity, **/.env, **/.cache, **/.next, **/.nuxt,                                 │\n│      **/dist, **/.vuepress/dist, **/.serverless, **/.idea,                                       │\n│      **/sw.*, **/.DS_Store, **/*.swp, **/notes, **/.output                                       │\n│                                                                                                  │\n│      Running in 4 threads                                                                        │\n│      File formatting disabled                                                                    │\n│                                                                                                  │\n│                                                                                                  │\n╰──────────────────────────────────────────────────────────────────────────────────────────────────╯\n",[30,12157,12155],{"__ignoreMap":464},[11,12159,12160,12161,12163],{},"The big change at this point is that most of the code was moved to the ",[30,12162,1050],{}," directory:",[459,12165,12168],{"className":12166,"code":12167,"language":997},[995],"Untracked files:\n  (use \"git add \u003Cfile>...\" to include in what will be committed)\n        app/\n        content/2025/07/21/\n\nno changes added to commit (use \"git add\" and/or \"git commit -a\")\n",[30,12169,12167],{"__ignoreMap":464},[11,12171,12172,12173],{},"Is that it? Let's try running ",[30,12174,12175],{},"yarn dev",[459,12177,12180],{"className":12178,"code":12179,"language":997},[995],"yarn dev\n...\n ERROR  Cannot start nuxt:  ENOENT: no such file or directory, open '/Users/brian/git/briancaffey.github.io/app/i18n/en-US.js'                           4:12:36 PM\n",[30,12181,12179],{"__ignoreMap":464},[11,12183,12184],{},"OK, we have an error related to i18n. I have struggled with i18n a fair bit when doing major changes to my site.",[11,12186,12187,12188,12190,12191,12193],{},"It looks like Nuxt 4 expects the i18n folder to be under the new ",[30,12189,1050],{}," directory, but it is still in the root directory, so let's move that and try running ",[30,12192,12175],{}," again.",[11,12195,12196],{},"Cool, no errors:",[459,12198,12201],{"className":12199,"code":12200,"language":997},[995],"~/git/briancaffey.github.io$ yarn dev\nyarn run v1.22.22\n$ nuxi dev --host\nNuxt 4.0.0 with Nitro 2.12.0                                              nuxi 4:16:03 PM\n                                                                               4:16:03 PM\n\n              █▀▀▀▀▀▀▀██▀▀▀█▀██▀█▀▀▀▀▀▀▀█\n              █ █▀▀▀█ ██▀▀ ▄▄██▀█ █▀▀▀█ █\n              █ █   █ █ █ ▄▀▄ ▄▄█ █   █ █\n              █ ▀▀▀▀▀ █▀▄ █▀▄▀█ █ ▀▀▀▀▀ █\n              █▀▀███▀▀▀█ ▀█ ▀█▀ ███▀▀████\n              ██ ▄ ▄▀▀▀█▀▄▀▀▀▀ ▄▄▀▀▄ ▄ ▀█\n              █▀█▄▀▀ ▀▀ ▀ ▄▀█▀▄ ▀▀█▄▄██ █\n              █ ▄██ █▀█▀▀▄█▄ ▄ █▀█▀█▀█ ▀█\n              █ █▀█  ▀▄▄█▄ ▀▀▄█ ▀▀▀ █ █▄█\n              █▀▀▀▀▀▀▀█ ▀▄  █ ▄ █▀█ ▀▄█▀█\n              █ █▀▀▀█ █▄ ▀█▄▀▄▀ ▀▀▀ ▀▀███\n              █ █   █ ██▄▀▄ ████▀▄▄█▄▀▄ █\n              █ ▀▀▀▀▀ █ ▀ ▀▄█▀  █ ▄▄▀██ █\n              ▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀\n\n  ➜ Local:    http://localhost:3000/\n  ➜ Network:  http://192.168.5.226:3000/ [QR code]\n\nℹ Using default Tailwind CSS file                            nuxt:tailwindcss 4:16:04 PM\n\n[4:16:04 PM]  WARN  Locales en-US, fr-FR, zh-ZH, ru-RU, ja-JP, hi-IN uses deprecated iso property, this will be replaced with language in v9\n\n  ➜ DevTools: press Shift + Option + D in the browser (v2.6.2)                 4:16:04 PM\n\n✔ Vite client built in 45ms                                                   4:16:05 PM\n✔ Vite server built in 30ms                                                   4:16:05 PM\n✔ Nuxt Nitro server built in 1071ms                                     nitro 4:16:07 PM\nℹ Vite client warmed up in 1ms                                                4:16:07 PM\nℹ Vite server warmed up in 63ms                                               4:16:07 PM\n[4:16:25 PM] ℹ ✨ new dependencies optimized: @vue/devtools-core, @vue/devtools-kit, vue-disqus, vue3-apexcharts, pinia, @intlify/shared, is-https, @intlify/core-base\nℹ ✨ optimized dependencies changed. reloading                                4:16:25 PM\n",[30,12202,12200],{"__ignoreMap":464},[11,12204,12205,12206,12209],{},"But I am getting a big ",[30,12207,12208],{},"500"," error in the browser:",[459,12211,12214],{"className":12212,"code":12213,"language":997},[995],"500\norgTransform.apply is not a function\n\nCustomize this page\nat createError (/Users/brian/git/briancaffey.github.io/node_modules/h3/dist/index.mjs:71:15)\nat /Users/brian/git/briancaffey.github.io/node_modules/@nuxt/vite-builder/dist/index.mjs:403:21)\nat async processMessage (/Users/brian/git/briancaffey.github.io/node_modules/@nuxt/vite-builder/dist/index.mjs:386:30)\n",[30,12215,12213],{"__ignoreMap":464},[11,12217,12218,12219,12222,12223,12226],{},"Searching GitHub's code, I it looks like ",[30,12220,12221],{},"orgTransform.apply"," comes from ",[30,12224,12225],{},"rdhainaut/unplugin-vue-i18n · lib/index.mjs",", so this is likely another i18n issue.",[459,12228,12231],{"className":12229,"code":12230,"language":997},[995],"\"@nuxtjs/i18n@^8.3.3\":\n  version \"8.5.6\"\n  resolved \"https://registry.yarnpkg.com/@nuxtjs/i18n/-/i18n-8.5.6.tgz#d65b375fba5244d83fc6833a604e2b532a64bef2\"\n  integrity sha512-L+g+LygKNoaS/AXExk7tzS9wSNn9QdP1T9VdTjjEGYftpeFgv2U8AQsY0dQAhgPIbXXhIAkNYxTk4YcINj9CfA==\n  dependencies:\n    \"@intlify/h3\" \"^0.5.0\"\n    \"@intlify/shared\" \"^9.14.1\"\n    \"@intlify/unplugin-vue-i18n\" \"^3.0.1\"\n",[30,12232,12230],{"__ignoreMap":464},[11,12234,12235],{},"Let's make sure that I'm on the latest version of Nuxt's i18n:",[459,12237,12240],{"className":12238,"code":12239,"language":997},[995],"npx nuxi module add i18n\n",[30,12241,12239],{"__ignoreMap":464},[459,12243,12246],{"className":12244,"code":12245,"language":997},[995],"yarn add @nuxtjs/i18n@^10.0.0\n",[30,12247,12245],{"__ignoreMap":464},[11,12249,12250,12251,12254,12255,208],{},"After upgrading ",[30,12252,12253],{},"@nuxtjs/i18n"," to v10.0.0 I got another i18n error when running ",[30,12256,12175],{},[459,12258,12261],{"className":12259,"code":12260,"language":997},[995],"[4:26:12 PM]  ERROR  Cannot start nuxt:  ENOENT: no such file or directory, open '/Users/brian/git/briancaffey.github.io/i18n/locales/i18n/en-US.js'\n\n    at readFileSync (node:fs:441:20)\n    at getLocaleType (node_modules/@nuxtjs/i18n/dist/module.mjs:143:34)\n    at resolveLocales (node_modules/@nuxtjs/i18n/dist/module.mjs:128:20)\n    at resolveLocaleInfo (node_modules/@nuxtjs/i18n/dist/module.mjs:1511:20)\n    at async node_modules/@nuxtjs/i18n/dist/module.mjs:1926:7\n    at async initNuxt (node_modules/nuxt/dist/index.mjs:5613:3)\n    at async NuxtDevServer._load (node_modules/@nuxt/cli/dist/chunks/index.mjs:211:5)\n    at async NuxtDevServer.load (node_modules/@nuxt/cli/dist/chunks/index.mjs:139:7)\n    at async NuxtDevServer.init (node_modules/@nuxt/cli/dist/chunks/index.mjs:130:5)\n    at async initialize (node_modules/@nuxt/cli/dist/chunks/index.mjs:426:3)\n",[30,12262,12260],{"__ignoreMap":464},[11,12264,12265,12266,12269],{},"OK! So I rearranged my ",[30,12267,12268],{},"i18n"," folder to match what it was looking for:",[459,12271,12274],{"className":12272,"code":12273,"language":997},[995],"~/git/briancaffey.github.io$ tree i18n -L 3\ni18n\n├── i18n.config.js\n└── locales\n    └── i18n\n        ├── en-US.js\n        ├── fr-FR.js\n        ├── hi-IN.js\n        ├── jp-JP.js\n        ├── ru-RU.js\n        └── zh-ZH.js\n",[30,12275,12273],{"__ignoreMap":464},[11,12277,12278,12279,12282],{},"Now I have my language files duplicated, however. But at least I'm able to visit my site in the browser! And I can see in the NuxtDevTools window that I am using ",[30,12280,12281],{},"v4.0.0","!",[11,12284,12285],{},[2718,12286],{"alt":12287,"src":12288},"NuxtDevTools","/static/nuxt4/nuxt-dev-tools.png",[459,12290,12293],{"className":12291,"code":12292,"language":997},[995],"  ➜ Local:    http://localhost:3000/\n  ➜ Network:  http://192.168.5.226:3000/ [QR code]\n\nℹ Using default Tailwind CSS file                                                    nuxt:tailwindcss 4:31:24 PM\n  ➜ DevTools: press Shift + Option + D in the browser (v2.6.2)                                         4:31:24 PM\n\nℹ Re-optimizing dependencies because lockfile has changed                                             4:31:25 PM\n✔ Vite client built in 61ms                                                                           4:31:25 PM\n✔ Vite server built in 42ms                                                                           4:31:26 PM\n✔ Nuxt Nitro server built in 1422ms                                                             nitro 4:31:27 PM\nℹ Vite client warmed up in 1ms                                                                        4:31:27 PM\nℹ Vite server warmed up in 164ms                                                                      4:31:27 PM\n[4:31:38 PM]  WARN  [@nuxtjs/mdc] Language \"Dockerfile\" is not loaded to the Shiki highlighter, fallback to plain text. Add the language to \"mdc.highlight.langs\" to fix this.\n[4:31:38 PM]  WARN  [@nuxtjs/mdc] Language \"Dockerfile\" is not loaded to the Shiki highlighter, fallback to plain text. Add the language to \"mdc.highlight.langs\" to fix this.\n[4:31:38 PM]  WARN  [@nuxtjs/mdc] Language \"vue\" is not loaded to the Shiki highlighter, fallback to plain text. Add the language to \"mdc.highlight.langs\" to fix this.\n[4:31:48 PM] ℹ ✨ new dependencies optimized: @vue/devtools-core, @vue/devtools-kit, vue-disqus, vue3-apexcharts, pinia\nℹ ✨ optimized dependencies changed. reloading                                                        4:31:48 PM\nℹ ✨ new dependencies optimized: @vueuse/components                                                   4:31:50 PM\nℹ ✨ optimized dependencies changed. reloading                                                        4:31:50 PM\n",[30,12294,12292],{"__ignoreMap":464},[11,12296,12297,12298,208],{},"I went through each of my project dependencies and updated them to the latest versions. After doing this and resolving some small issues, I got the following when running ",[30,12299,12175],{},[459,12301,12304],{"className":12302,"code":12303,"language":997},[995],"[@nuxt/content 7:10:17 PM]  WARN  No content configuration found, falling back to default collection. In order to have full control over your collections, create the config file in project root. See: https://content.nuxt.com/docs/getting-started/installation\n\nℹ Using default Tailwind CSS file                                                                 nuxt:tailwindcss 7:10:17 PM\n  ➜ DevTools: press Shift + Option + D in the browser (v2.6.2)                                                      7:10:17 PM\n\n\n ERROR  Nuxt Content requires better-sqlite3 module to operate.                                       @nuxt/content 7:10:18 PM\n\n                                                                                                                    7:10:18 PM\n                                                                                                                    7:10:18 PM\n❯ Do you want to install better-sqlite3 package?\n● Yes / ○ No\n",[30,12305,12303],{"__ignoreMap":464},[11,12307,12308],{},"After installing this I got another error!",[11,12310,12311,12312],{},"Upgrading from Nuxt Content v2 to v3 introduced some breaking changes, so I had to work through those. ",[20,12313,12314],{"href":12314,"rel":12315},"https://content.nuxt.com/docs/getting-started/migration",[24],[11,12317,12318],{},"This is one of the errors I got after upgrading Nuxt Content from v2 to v3.",[459,12320,12323],{"className":12321,"code":12322,"language":997},[995]," ERROR  [request error] [unhandled] [GET] http://localhost:3000/blog/1                                                                                                                      7:14:09 PM\n\n\nℹ Error: Cannot read properties of undefined (reading 'length')\n\n ⁃ at _sfc_ssrRender (app/pages/blog/[number].vue:78:62)\n\n   37 ┃\n   38 ┃  const { data: paginatedItems } = await useAsyncData(route.path, () =>\n   39 ┃    queryCollection(\"/\")\n   40 ┃      .where({ draft: { $ne: true } })\n   41 ┃      .sort({'date': -1})\n   42 ┃      .limit(10)\n   43 ┃      .skip(9 * (pageNo - 1))\n   44 ┃      .find()\n   45 ┃  )\n   46 ┃  \u003C/script>\n   47 ┃\n",[30,12324,12322],{"__ignoreMap":464},[11,12326,12327],{},"The API changed and it uses a SQL-like query language. The migration docs mentioned this:",[210,12329,12330],{},[11,12331,12332],{},"The new API is backed by SQL and content queries happens within a specific collection.",[11,12334,12335],{},"Now the above query is written more like this with the new version of Nuxt Content:",[459,12337,12341],{"className":12338,"code":12339,"language":12340,"meta":464,"style":464},"language-js shiki shiki-themes github-light github-dark monokai","const { data: paginatedItems } = await useAsyncData(route.path, () =>\n  queryCollection(\"blog\")\n    .order(\"date\", \"DESC\")\n    .limit(10)\n    .skip(9 * (pageNo - 1))\n    .all()\n)\n","js",[30,12342,12343,12379,12392,12412,12426,12452,12462],{"__ignoreMap":464},[151,12344,12345,12349,12352,12356,12358,12362,12365,12367,12370,12373,12376],{"class":469,"line":470},[151,12346,12348],{"class":12347},"sq6CD","const",[151,12350,12351],{"class":503}," { ",[151,12353,12355],{"class":12354},"sOrwc","data",[151,12357,6208],{"class":503},[151,12359,12361],{"class":12360},"sXSQT","paginatedItems",[151,12363,12364],{"class":503}," } ",[151,12366,1876],{"class":1869},[151,12368,12369],{"class":1869}," await",[151,12371,12372],{"class":473}," useAsyncData",[151,12374,12375],{"class":503},"(route.path, () ",[151,12377,12378],{"class":12347},"=>\n",[151,12380,12381,12384,12387,12390],{"class":469,"line":488},[151,12382,12383],{"class":473},"  queryCollection",[151,12385,12386],{"class":503},"(",[151,12388,12389],{"class":481},"\"blog\"",[151,12391,3640],{"class":503},[151,12393,12394,12397,12400,12402,12405,12407,12410],{"class":469,"line":500},[151,12395,12396],{"class":503},"    .",[151,12398,12399],{"class":473},"order",[151,12401,12386],{"class":503},[151,12403,12404],{"class":481},"\"date\"",[151,12406,106],{"class":503},[151,12408,12409],{"class":481},"\"DESC\"",[151,12411,3640],{"class":503},[151,12413,12414,12416,12419,12421,12424],{"class":469,"line":509},[151,12415,12396],{"class":503},[151,12417,12418],{"class":473},"limit",[151,12420,12386],{"class":503},[151,12422,12423],{"class":477},"10",[151,12425,3640],{"class":503},[151,12427,12428,12430,12433,12435,12437,12440,12443,12446,12449],{"class":469,"line":517},[151,12429,12396],{"class":503},[151,12431,12432],{"class":473},"skip",[151,12434,12386],{"class":503},[151,12436,7918],{"class":477},[151,12438,12439],{"class":1869}," *",[151,12441,12442],{"class":503}," (pageNo ",[151,12444,12445],{"class":1869},"-",[151,12447,12448],{"class":477}," 1",[151,12450,12451],{"class":503},"))\n",[151,12453,12454,12456,12459],{"class":469,"line":534},[151,12455,12396],{"class":503},[151,12457,12458],{"class":473},"all",[151,12460,12461],{"class":503},"()\n",[151,12463,12464],{"class":469,"line":1413},[151,12465,3640],{"class":503},[56,12467,12469],{"id":12468},"yarn-generate",[30,12470,12471],{},"yarn generate",[11,12473,12474,12475,12477,12478,12480,12481,12484,12485,12487],{},"After fixing the Nuxt Content query syntax for Nuxt Content v3, I tried running ",[30,12476,12471],{}," to build my static site. This can often uncover issues that you won't see when just running ",[30,12479,12175],{},". Right away I found an issue with my Pinia Store. It couldn't find the file for my store. The codemod migration tool did not move my ",[30,12482,12483],{},"store"," folder into the ",[30,12486,1050],{}," directory, so I had to do that manually.",[11,12489,12490,12491,12493],{},"OK! After fixing the Pinia Store issues I was able to run ",[30,12492,12471],{}," successfully!",[459,12495,12498],{"className":12496,"code":12497,"language":997},[995],"ℹ Prerendered 785 routes in 93.991 seconds\n✔ Generated public .output/public\n✔ You can preview this build using npx serve .output/public\n✔ You can now deploy .output/public to any static hosting!\n✨  Done in 105.98s.\n",[30,12499,12497],{"__ignoreMap":464},[56,12501,12503],{"id":12502},"cicd","CI/CD",[11,12505,12506,12507,12509,12510,12513,12514,643],{},"To deploy my blog to ",[30,12508,662],{}," I just have to push my working changes to the ",[30,12511,12512],{},"master"," branch of my GitHub repository. GitHub Actions builds the site and pushes the build assets to GitHub Pages for me. To avoid any issues I need to make sure I'm using the correct version of ",[30,12515,12516],{},"node",[459,12518,12521],{"className":12519,"code":12520,"language":997},[995],"- name: Setup Node\n  uses: actions/setup-node@v4\n  with:\n    node-version: \"20\"\n",[30,12522,12520],{"__ignoreMap":464},[11,12524,12525,12526,313,12529],{},"I updated ",[30,12527,12528],{},"\"20\"",[30,12530,12531],{},"\"22.17.1\"",[56,12533,12535],{"id":12534},"datacontentcontentssqlite",[30,12536,12537],{},".data/content/contents.sqlite",[11,12539,12540,12541,313,12544,129,12547,748],{},"Something else I noticed is this new sqlite file. This is just for debugging, so we can add ",[30,12542,12543],{},".data",[30,12545,12546],{},".gitignore",[20,12548,12549],{"href":12549,"rel":12550},"https://content.nuxt.com/docs/advanced/tools#locate-your-sqlite-database",[24],[56,12552,12554,5032],{"id":12553},"yarn-lint-error",[30,12555,12556],{},"yarn lint",[11,12558,12559,12560,12562],{},"I forgot to see if the code would properly lint, so on pushing to the master branch I got a failed deployment pipeline. Running ",[30,12561,12556],{}," from my Mac I get the same error:",[459,12564,12567],{"className":12565,"code":12566,"language":997},[995],"~/git/briancaffey.github.io$ yarn lint\nyarn run v1.22.22\n$ eslint .\n\nOops! Something went wrong! :(\n\nESLint: 9.31.0\n\nTypeError [ERR_INVALID_MODULE_SPECIFIER]: Invalid module \".nuxt/eslint.config.mjs\" is not a valid package name imported from /Users/brian/git/briancaffey.github.io/eslint.config.mjs\n    at parsePackageName (node:internal/modules/package_json_reader:211:11)\n    at Object.getPackageJSONURL (node:internal/modules/package_json_reader:222:53)\n    at packageResolve (node:internal/modules/esm/resolve:768:81)\n    at moduleResolve (node:internal/modules/esm/resolve:854:18)\n    at defaultResolve (node:internal/modules/esm/resolve:984:11)\n    at ModuleLoader.defaultResolve (node:internal/modules/esm/loader:780:12)\n    at #cachedDefaultResolve (node:internal/modules/esm/loader:704:25)\n    at ModuleLoader.resolve (node:internal/modules/esm/loader:687:38)\n    at ModuleLoader.getModuleJobForImport (node:internal/modules/esm/loader:305:38)\n    at ModuleJob._link (node:internal/modules/esm/module_job:175:49)\nerror Command failed with exit code 2.\ninfo Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.\n",[30,12568,12566],{"__ignoreMap":464},[11,12570,12571,12572,12575,12576,12579],{},"The error seems to be related to the file ",[30,12573,12574],{},"eslint.config.mjs",". I'm not familiar with what ",[30,12577,12578],{},".mjs"," files are, so I asked cursor:",[210,12581,12582,12585],{},[11,12583,12584],{},"What are .mjs files?",[11,12586,12587],{},".mjs stands for \"Module JavaScript\" and is a file extension that explicitly tells Node.js to treat the file as an ES module, regardless of your project's configuration.",[11,12589,12590],{},"I was able to fix the error by making a small change to the input line that was throwing the error:",[459,12592,12595],{"className":12593,"code":12594,"language":997},[995],"import withNuxt from '.nuxt/eslint.config.mjs'\n",[30,12596,12594],{"__ignoreMap":464},[11,12598,12599],{},"changed to",[459,12601,12604],{"className":12602,"code":12603,"language":997},[995],"import withNuxt from './.nuxt/eslint.config.mjs'\n",[30,12605,12603],{"__ignoreMap":464},[11,12607,12608,12609,12611,12612,12614],{},"Now running ",[30,12610,12556],{}," gave me small error in one of my components. Fixing that resulted in a successful ",[30,12613,12556],{},"! OK, it should work this time!",[56,12616,12618],{"id":12617},"typescript","TypeScript",[11,12620,12621],{},"I also migrated this site from JavaScript to TypeScript, something I have been meaning to do for a while now. I was able to work down to 0 Problems in the VS Code/Cursor terminal toolbar.",[589,12623,12624],{},"html pre.shiki code .srTi1, html code.shiki .srTi1{--shiki-default:#6F42C1;--shiki-dark:#B392F0;--shiki-sepia:#A6E22E}html pre.shiki code .sstjo, html code.shiki .sstjo{--shiki-default:#032F62;--shiki-dark:#9ECBFF;--shiki-sepia:#E6DB74}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html .sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html.sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html pre.shiki code .s8-w5, html code.shiki .s8-w5{--shiki-default:#6A737D;--shiki-dark:#6A737D;--shiki-sepia:#88846F}html pre.shiki code .sq6CD, html code.shiki .sq6CD{--shiki-default:#D73A49;--shiki-default-font-style:inherit;--shiki-dark:#F97583;--shiki-dark-font-style:inherit;--shiki-sepia:#66D9EF;--shiki-sepia-font-style:italic}html pre.shiki code .sMOD_, html code.shiki .sMOD_{--shiki-default:#24292E;--shiki-dark:#E1E4E8;--shiki-sepia:#F8F8F2}html pre.shiki code .sOrwc, html code.shiki .sOrwc{--shiki-default:#E36209;--shiki-dark:#FFAB70;--shiki-sepia:#F8F8F2}html pre.shiki code .sXSQT, html code.shiki .sXSQT{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#F8F8F2}html pre.shiki code .sC2Qs, html code.shiki .sC2Qs{--shiki-default:#D73A49;--shiki-dark:#F97583;--shiki-sepia:#F92672}html pre.shiki code .s7F3e, html code.shiki .s7F3e{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#AE81FF}",{"title":464,"searchDepth":488,"depth":488,"links":12626},[12627,12628,12629,12630,12631,12632,12633,12634,12635,12637],{"id":11967,"depth":488,"text":11968},{"id":11993,"depth":488,"text":11994},{"id":12016,"depth":488,"text":12017},{"id":12044,"depth":488,"text":12045},{"id":12064,"depth":488,"text":12065},{"id":12468,"depth":488,"text":12471},{"id":12502,"depth":488,"text":12503},{"id":12534,"depth":488,"text":12537},{"id":12553,"depth":488,"text":12636},"yarn lint error",{"id":12617,"depth":488,"text":12618},"2025-07-21","This article details my process of upgrading my static GitHub Pages Blog from Nuxt 3 to Nuxt 4, and an upgrade of the headless CMS that powers my site, Nuxt Content, from v2 to v3","/static/nuxt4/nuxt4.png",{},"/2025/07/21/upgrading-my-static-nuxt-blog-from-nuxt-3-to-nuxt-4",{"title":11809,"description":12639},"2025/07/21/upgrading-my-static-nuxt-blog-from-nuxt-3-to-nuxt-4",[11803,12646,12647,12648,12617],"vue","blogging","cms","0LeY0hDM3l2eWfuelcmC0mQssUSI1zKnSsLRjS1MBYY",{"id":12651,"title":12652,"body":12653,"comments":609,"date":14044,"description":14045,"draft":602,"extension":605,"external":14046,"image":14049,"meta":14050,"navigation":609,"path":14051,"seo":14052,"stem":14053,"tags":14054,"__hash__":14056},"blog/2025/07/17/flux-plugin-for-project-g-assist-hackathon.md","Flux Plug-in for Project G-Assist",{"type":8,"value":12654,"toc":14002},[12655,12658,12661,12664,12667,12670,12674,12678,12704,12708,12734,12738,12770,12774,12800,12804,12821,12825,12828,12845,12852,12856,12860,12863,12866,12870,12877,12919,12923,12939,12943,12957,13063,13067,13076,13108,13112,13127,13131,13134,13284,13294,13298,13301,13304,13307,13310,13313,13317,13324,13330,13333,13339,13342,13348,13351,13355,13366,13370,13529,13540,13544,13550,13553,13560,13564,13567,13569,13577,13581,13584,13590,13593,13597,13600,13620,13626,13629,13635,13652,13656,13662,13668,13671,13686,13689,13693,13707,13711,13722,13726,13740,13744,13747,13762,13766,13769,13916,13920,13927,13941,13945,13971,13975,13982,13986,13993,13996,13999],[11,12656,12657],{},"This article will discuss my submission for the Project G-Assist Hackathon: Flux Plug-in for Project G-Assist. This plugin allows RTX AI PC users to tap into their GPU's image generation capabilities through the Flux.1-dev NVIDIA NIM. G-Assist now generates images from your commands on demand!",[12659,12660],"flux-plugin-tweet",{},[11,12662,12663],{},"The Flux Plugin for G-Assist is a plugin developed for NVIDIA’s Project G-Assist that brings real-time AI image generation directly to the desktop through natural language commands. It allows users to create high-quality images using the Flux family of models from Black Forest Labs, seamlessly integrated with the G-Assist interface. Users can simply type or speak prompts such as “a futuristic robot painter in a neon-lit workshop”, and the plugin handles the entire process—from submitting the prompt to generating and displaying the image—all without leaving the G-Assist chat window.",[11,12665,12666],{},"The plugin supports multiple deployment backends for inference, including Flux NIMs running locally via WSL, on the cloud via build.nvidia.com, or through InvokeAI using the Flux Kontext model for image-to-image and screenshot-based transformations. The plugin includes additional tools for managing the inference service, such as checking status and turning the NIM service on or off, enabling a user-friendly experience for both beginners and power users.",[11,12668,12669],{},"The goal of this plugin is to make generative image workflows faster, more accessible, and more fun—leveraging the strengths of NVIDIA’s hardware, AI models, and desktop ecosystem. By combining the power of FLUX with the voice-enabled G-Assist interface, the plugin turns any PC into a hands-free creative studio.",[56,12671,12673],{"id":12672},"what-can-it-do","What Can It Do?",[736,12675,12677],{"id":12676},"image-generation","Image Generation",[76,12679,12680,12686,12692,12698],{},[79,12681,12682,12685],{},[15,12683,12684],{},"Generate images from text prompts"," using Flux AI model",[79,12687,12688,12691],{},[15,12689,12690],{},"Support for multiple backends",": Local NIM servers, NVIDIA hosted services, or InvokeAI",[79,12693,12694,12697],{},[15,12695,12696],{},"Automatic desktop background setting"," - generated images can be set as your wallpaper",[79,12699,12700,12703],{},[15,12701,12702],{},"High-quality output"," with customizable parameters (resolution, steps, CFG scale)",[736,12705,12707],{"id":12706},"nim-server-management","NIM Server Management",[76,12709,12710,12716,12722,12728],{},[79,12711,12712,12715],{},[15,12713,12714],{},"Start/stop local Flux NIM servers"," using WSL and Podman",[79,12717,12718,12721],{},[15,12719,12720],{},"Check NIM server status"," to see if the service is running",[79,12723,12724,12727],{},[15,12725,12726],{},"Health endpoint testing"," for local servers",[79,12729,12730,12733],{},[15,12731,12732],{},"Automatic configuration"," using NGC API keys and Hugging Face tokens",[736,12735,12737],{"id":12736},"invokeai-integration","InvokeAI Integration",[76,12739,12740,12746,12752,12758,12764],{},[79,12741,12742,12745],{},[15,12743,12744],{},"Upload screenshots to InvokeAI"," for image-to-image workflows",[79,12747,12748,12751],{},[15,12749,12750],{},"Flux Kontext generation"," using uploaded images as reference",[79,12753,12754,12757],{},[15,12755,12756],{},"Processor control"," - pause and resume InvokeAI processing queues",[79,12759,12760,12763],{},[15,12761,12762],{},"VRAM management"," - empty model cache to free up memory",[79,12765,12766,12769],{},[15,12767,12768],{},"Status monitoring"," - check InvokeAI service health and version",[736,12771,12773],{"id":12772},"smart-configuration","Smart Configuration",[76,12775,12776,12782,12788,12794],{},[79,12777,12778,12781],{},[15,12779,12780],{},"Flexible URL configuration"," - works with local servers or NVIDIA hosted endpoints",[79,12783,12784,12787],{},[15,12785,12786],{},"Automatic API key validation"," - ensures proper NVIDIA API key format",[79,12789,12790,12793],{},[15,12791,12792],{},"Configurable output directories"," for generated images",[79,12795,12796,12799],{},[15,12797,12798],{},"Board management"," for InvokeAI gallery organization",[736,12801,12803],{"id":12802},"example-commands","Example Commands",[76,12805,12806,12809,12812,12815,12818],{},[79,12807,12808],{},"\"hey flux, generate an image of a cyberpunk city at night\"",[79,12810,12811],{},"\"hey flux, start the Flux NIM server\"",[79,12813,12814],{},"\"hey flux, use kontext to make it a cartoon style\" (does image-to-image generation using latest screenshot taken with NVIDIA screenshot shortcut)",[79,12816,12817],{},"\"hey flux, empty the InvokeAI model cache to free up VRAM\"",[79,12819,12820],{},"\"hey flux, Check if the NIM server is running\"",[56,12822,12824],{"id":12823},"before-you-start","Before You Start",[11,12826,12827],{},"Make sure you have:",[76,12829,12830,12833,12836,12839,12842],{},[79,12831,12832],{},"Windows PC",[79,12834,12835],{},"Python 3.12 or higher",[79,12837,12838],{},"G-Assist installed on your system",[79,12840,12841],{},"pywin32 >= 223",[79,12843,12844],{},"Basic knowledge of Python",[11,12846,12847,12848,12851],{},"💡 ",[15,12849,12850],{},"Tip",": Use a virtual environment to keep your plugin dependencies isolated from other Python projects!",[56,12853,12855],{"id":12854},"installation-guide","Installation Guide",[736,12857,12859],{"id":12858},"step-1-get-the-files","Step 1: Get the Files",[11,12861,12862],{},"Clone this repository",[11,12864,12865],{},"This downloads the plugin code and all necessary files to your computer.",[736,12867,12869],{"id":12868},"step-2-set-up-python-environment","Step 2: Set Up Python Environment",[11,12871,12872,12873,12876],{},"Run the ",[30,12874,12875],{},"setup.bat"," file. This will take care of creating a Python virtual environment will install project dependencies.",[459,12878,12880],{"className":461,"code":12879,"language":463,"meta":464,"style":464},"python -m venv .venv\n.venv\\Scripts\\activate\npython -m pip install -r requirements.txt\n",[30,12881,12882,12896,12901],{"__ignoreMap":464},[151,12883,12884,12887,12890,12893],{"class":469,"line":470},[151,12885,12886],{"class":473},"python",[151,12888,12889],{"class":477}," -m",[151,12891,12892],{"class":481}," venv",[151,12894,12895],{"class":481}," .venv\n",[151,12897,12898],{"class":469,"line":488},[151,12899,12900],{"class":473},".venv\\Scripts\\activate\n",[151,12902,12903,12905,12907,12910,12913,12916],{"class":469,"line":500},[151,12904,12886],{"class":473},[151,12906,12889],{"class":477},[151,12908,12909],{"class":481}," pip",[151,12911,12912],{"class":481}," install",[151,12914,12915],{"class":477}," -r",[151,12917,12918],{"class":481}," requirements.txt\n",[736,12920,12922],{"id":12921},"step-3-build-the-project","Step 3: Build the project",[11,12924,10635,12925,12928,12929,187,12932,12935,12936,643],{},[30,12926,12927],{},"build.bat"," to build the project. This script will also place the ",[30,12930,12931],{},"g-assist-plugin-flux.exe",[30,12933,12934],{},"manifest.json"," files in ",[30,12937,12938],{},"%PROGRAMDATA%\\NVIDIA Corporation\\nvtopps\\rise\\plugins\\flux",[736,12940,12942],{"id":12941},"step-4-configuration","Step 4: Configuration",[11,12944,12945,12946,12949,12950,12953,12954,12956],{},"Copy the ",[30,12947,12948],{},"config.example.json"," file and rename it as ",[30,12951,12952],{},"config.json"," and place it in ",[30,12955,12938],{},". Customize it with the appropriate configuration values (see below for configuration details). Here is an example configuration:",[459,12958,12960],{"className":6194,"code":12959,"language":6196,"meta":464,"style":464},"{\n    \"GALLERY_DIRECTORY\": \"E:\\\\NVIDIA\",\n    \"FLUX_NIM_URL\": \"http://localhost:8000\",\n    \"NGC_API_KEY\": \"xxxxxxxx\",\n    \"HF_TOKEN\": \"hf_xxxxxxxx\",\n    \"LOCAL_NIM_CACHE\": \"~/.cache/nim\",\n    \"INVOKEAI_URL\": \"http://localhost:9090\",\n    \"OUTPUT_DIRECTORY\": \"E:\\\\Flux\"\n}\n",[30,12961,12962,12967,12985,12997,13009,13021,13033,13045,13059],{"__ignoreMap":464},[151,12963,12964],{"class":469,"line":470},[151,12965,12966],{"class":503},"{\n",[151,12968,12969,12972,12974,12977,12980,12983],{"class":469,"line":488},[151,12970,12971],{"class":6205},"    \"GALLERY_DIRECTORY\"",[151,12973,6208],{"class":503},[151,12975,12976],{"class":6211},"\"E:",[151,12978,12979],{"class":477},"\\\\",[151,12981,12982],{"class":6211},"NVIDIA\"",[151,12984,9417],{"class":503},[151,12986,12987,12990,12992,12995],{"class":469,"line":500},[151,12988,12989],{"class":6205},"    \"FLUX_NIM_URL\"",[151,12991,6208],{"class":503},[151,12993,12994],{"class":6211},"\"http://localhost:8000\"",[151,12996,9417],{"class":503},[151,12998,12999,13002,13004,13007],{"class":469,"line":509},[151,13000,13001],{"class":6205},"    \"NGC_API_KEY\"",[151,13003,6208],{"class":503},[151,13005,13006],{"class":6211},"\"xxxxxxxx\"",[151,13008,9417],{"class":503},[151,13010,13011,13014,13016,13019],{"class":469,"line":517},[151,13012,13013],{"class":6205},"    \"HF_TOKEN\"",[151,13015,6208],{"class":503},[151,13017,13018],{"class":6211},"\"hf_xxxxxxxx\"",[151,13020,9417],{"class":503},[151,13022,13023,13026,13028,13031],{"class":469,"line":534},[151,13024,13025],{"class":6205},"    \"LOCAL_NIM_CACHE\"",[151,13027,6208],{"class":503},[151,13029,13030],{"class":6211},"\"~/.cache/nim\"",[151,13032,9417],{"class":503},[151,13034,13035,13038,13040,13043],{"class":469,"line":1413},[151,13036,13037],{"class":6205},"    \"INVOKEAI_URL\"",[151,13039,6208],{"class":503},[151,13041,13042],{"class":6211},"\"http://localhost:9090\"",[151,13044,9417],{"class":503},[151,13046,13047,13050,13052,13054,13056],{"class":469,"line":1418},[151,13048,13049],{"class":6205},"    \"OUTPUT_DIRECTORY\"",[151,13051,6208],{"class":503},[151,13053,12976],{"class":6211},[151,13055,12979],{"class":477},[151,13057,13058],{"class":6211},"Flux\"\n",[151,13060,13061],{"class":469,"line":2462},[151,13062,6274],{"class":503},[736,13064,13066],{"id":13065},"step-5-set-up-flux","Step 5: Set up Flux",[11,13068,13069,13070,13075],{},"Follow instructions ",[20,13071,13074],{"href":13072,"rel":13073},"https://build.nvidia.com/black-forest-labs/flux_1-dev/deploy?environment=wsl2.md",[24],"here"," for installing the FLUX.1-dev model from Black Forest Labs using NVIDIA NIM on WSL. Be sure to do the following:",[76,13077,13078,13086,13095,13098],{},[79,13079,13080,13081,13085],{},"follow instructions here: ",[20,13082,13083],{"href":13083,"rel":13084},"https://docs.nvidia.com/nim/wsl2/latest/getting-started.html",[24]," for getting started with WSL",[79,13087,13088,13089,13094],{},"use the ",[20,13090,13093],{"href":13091,"rel":13092},"https://docs.nvidia.com/nim/wsl2/latest/getting-started.html#use-the-nvidia-nim-wsl2-installer-recommended",[24],"NVIDIA NIM WSL2 Installer"," for configuring a new WSL environment configured with all of the required NVIDIA dependencies",[79,13096,13097],{},"In your Hugging Face account read and accept FLUX.1-dev, FLUX.1-Canny-dev, FLUX.1-Depth-dev and FLUX.1-dev-onnx License Agreements and Acceptable Use Policy. You must accept the agreements/policies for all of the models even though this plugin does not directly use the Canny or Depth modes.",[79,13099,13100,13101,13104,13105,13107],{},"Make sure to map port ",[30,13102,13103],{},"8000"," in the NIM to port ",[30,13106,13103],{}," on the WSL host as shown in the setup link above.",[736,13109,13111],{"id":13110},"step-6-install-invokeai-optional","Step 6: Install InvokeAI (optional)",[11,13113,13114,13115,13119,13120,13123,13124,643],{},"InvokeAI has a Windows installer that can be found here: ",[20,13116,13117],{"href":13117,"rel":13118},"https://invoke-ai.github.io/InvokeAI/installation/quick_start/#step-2-download",[24],". Download the Flux models including the ",[15,13121,13122],{},"FLUX.1 Kontext dev (Quantized)"," model for using Flux Kontext in the Flux Plug-in for G-Assist. By default InvokeAI runs on ",[30,13125,13126],{},"http://localhost:9090",[736,13128,13130],{"id":13129},"step-7-start-the-nvidia-nim","Step 7: Start the NVIDIA NIM",[11,13132,13133],{},"Ask flux if this NIM is running. If it is not running, ask flux to start the NIM. This will run a command to start the NIM container in WSL using podman:",[459,13135,13138],{"className":13136,"code":13137,"language":12886,"meta":464,"style":464},"language-python shiki shiki-themes github-light github-dark monokai","podman_cmd = [\n    'wsl', '-d', 'NVIDIA-Workbench',\n    'podman', 'run', '-d', '--rm', '--name=nim-server',\n    '--device', 'nvidia.com/gpu=all',\n    '-e', f'NGC_API_KEY={NGC_API_KEY}',\n    '-e', f'HF_TOKEN={HF_TOKEN}',\n    '-p', '8000:8000',\n    '-v', f'{LOCAL_NIM_CACHE}:/opt/nim/.cache/',\n    'nvcr.io/nim/black-forest-labs/flux.1-dev:1.0.0'\n]\n",[30,13139,13140,13150,13167,13193,13205,13226,13244,13256,13275,13280],{"__ignoreMap":464},[151,13141,13142,13145,13147],{"class":469,"line":470},[151,13143,13144],{"class":503},"podman_cmd ",[151,13146,1876],{"class":1869},[151,13148,13149],{"class":503}," [\n",[151,13151,13152,13155,13157,13160,13162,13165],{"class":469,"line":488},[151,13153,13154],{"class":481},"    'wsl'",[151,13156,106],{"class":503},[151,13158,13159],{"class":481},"'-d'",[151,13161,106],{"class":503},[151,13163,13164],{"class":481},"'NVIDIA-Workbench'",[151,13166,9417],{"class":503},[151,13168,13169,13172,13174,13177,13179,13181,13183,13186,13188,13191],{"class":469,"line":500},[151,13170,13171],{"class":481},"    'podman'",[151,13173,106],{"class":503},[151,13175,13176],{"class":481},"'run'",[151,13178,106],{"class":503},[151,13180,13159],{"class":481},[151,13182,106],{"class":503},[151,13184,13185],{"class":481},"'--rm'",[151,13187,106],{"class":503},[151,13189,13190],{"class":481},"'--name=nim-server'",[151,13192,9417],{"class":503},[151,13194,13195,13198,13200,13203],{"class":469,"line":509},[151,13196,13197],{"class":481},"    '--device'",[151,13199,106],{"class":503},[151,13201,13202],{"class":481},"'nvidia.com/gpu=all'",[151,13204,9417],{"class":503},[151,13206,13207,13210,13212,13215,13218,13221,13224],{"class":469,"line":517},[151,13208,13209],{"class":481},"    '-e'",[151,13211,106],{"class":503},[151,13213,13214],{"class":12347},"f",[151,13216,13217],{"class":481},"'NGC_API_KEY=",[151,13219,13220],{"class":477},"{NGC_API_KEY}",[151,13222,13223],{"class":481},"'",[151,13225,9417],{"class":503},[151,13227,13228,13230,13232,13234,13237,13240,13242],{"class":469,"line":534},[151,13229,13209],{"class":481},[151,13231,106],{"class":503},[151,13233,13214],{"class":12347},[151,13235,13236],{"class":481},"'HF_TOKEN=",[151,13238,13239],{"class":477},"{HF_TOKEN}",[151,13241,13223],{"class":481},[151,13243,9417],{"class":503},[151,13245,13246,13249,13251,13254],{"class":469,"line":1413},[151,13247,13248],{"class":481},"    '-p'",[151,13250,106],{"class":503},[151,13252,13253],{"class":481},"'8000:8000'",[151,13255,9417],{"class":503},[151,13257,13258,13261,13263,13265,13267,13270,13273],{"class":469,"line":1418},[151,13259,13260],{"class":481},"    '-v'",[151,13262,106],{"class":503},[151,13264,13214],{"class":12347},[151,13266,13223],{"class":481},[151,13268,13269],{"class":477},"{LOCAL_NIM_CACHE}",[151,13271,13272],{"class":481},":/opt/nim/.cache/'",[151,13274,9417],{"class":503},[151,13276,13277],{"class":469,"line":2462},[151,13278,13279],{"class":481},"    'nvcr.io/nim/black-forest-labs/flux.1-dev:1.0.0'\n",[151,13281,13282],{"class":469,"line":2471},[151,13283,3691],{"class":503},[11,13285,13286,13287,187,13290,13293],{},"Then ask if the NIM is ready. This will check the ",[30,13288,13289],{},"/v1/health/live",[30,13291,13292],{},"/v1/health/ready"," endpoints of the Flux NIM.",[736,13295,13297],{"id":13296},"step-8-generate-ai-images-using-the-flux-plug-in-in-the-g-assist-chat-window","Step 8: Generate AI images using the Flux Plug-in in the G-Assist chat window",[11,13299,13300],{},"Send a message to G-Assist:",[11,13302,13303],{},"\"hey flux, generate a cat piloting a spaceship\"",[11,13305,13306],{},"The Flux Plug-in will respond with:",[11,13308,13309],{},"flux> Your image generation request is in progress! Prompt: \"a cat piloting a spaceship\"",[11,13311,13312],{},"When the image generation is complete you will find the image on your Desktop background, and the image will be saved to the output directory specified in your configuration file.",[736,13314,13316],{"id":13315},"step-9-transform-a-screenshot-with-flux-kontext","Step 9: Transform a screenshot with Flux Kontext",[11,13318,13319,13320,13323],{},"Take a screenshot using the NVIDIA Screenshot hotkey (usually ",[30,13321,13322],{},"Alt + F1","), and then ask the Flux Plug-in to transform it to any style using Kontext.",[11,13325,13326],{},[2718,13327],{"alt":13328,"src":13329},"AI-generated image of a cat piloting a spaceship","/static/flux/cat_spaceship.png",[11,13331,13332],{},"hey flux, use kontext with the prompt: cartoon style",[11,13334,13335],{},[2718,13336],{"alt":13337,"src":13338},"AI-generated cartoon-style image of a cat piloting a spaceship using Flux Kontext","/static/flux/cat_spaceship_cartoon.png",[11,13340,13341],{},"Flux does this by triggering an InvokeAI graph workflow. The generated image and the workflow can both be viewed in the InvokeAI UI:",[11,13343,13344],{},[2718,13345],{"alt":13346,"src":13347},"InvokeAI Flux Kontext workflow visualization showing image editing pipeline","/static/flux/invokeai_workflow.png",[11,13349,13350],{},"You can ask the Flux Plug-in to pause/resume InvokeAI processing to avoid running Flux Kontext image generation while your GPU is busy with other tasks. Also you can ask flux to empty the model cache in order to free up VRAM on your GPU.",[56,13352,13354],{"id":13353},"configuration","Configuration",[11,13356,13357,13358,13360,13361,313,13363,13365],{},"The Flux plugin uses a ",[30,13359,12952],{}," file to manage all settings. Copy ",[30,13362,12948],{},[30,13364,12952],{}," and customize the values for your setup.",[736,13367,13369],{"id":13368},"configuration-options","Configuration Options",[1131,13371,13372,13385],{},[1134,13373,13374],{},[1137,13375,13376,13379,13382],{},[1140,13377,13378],{},"Option",[1140,13380,13381],{},"Example Values",[1140,13383,13384],{},"Required",[1153,13386,13387,13405,13420,13437,13451,13465,13479,13495,13512],{},[1137,13388,13389,13394,13402],{},[1158,13390,13391],{},[30,13392,13393],{},"GALLERY_DIRECTORY",[1158,13395,13396,106,13399],{},[30,13397,13398],{},"\"D:\\\\Screenshots\"",[30,13400,13401],{},"\"C:\\\\NVIDIA\"",[1158,13403,13404],{},"No",[1137,13406,13407,13412,13417],{},[1158,13408,13409],{},[30,13410,13411],{},"NVIDIA_API_KEY",[1158,13413,13414],{},[30,13415,13416],{},"\"nvapi-your-key-here\"",[1158,13418,13419],{},"No*",[1137,13421,13422,13427,13434],{},[1158,13423,13424],{},[30,13425,13426],{},"FLUX_NIM_URL",[1158,13428,13429,106,13431],{},[30,13430,12994],{},[30,13432,13433],{},"\"http://192.168.1.100:8000\"",[1158,13435,13436],{},"Yes",[1137,13438,13439,13443,13448],{},[1158,13440,13441],{},[30,13442,4786],{},[1158,13444,13445],{},[30,13446,13447],{},"\"your-ngc-key\"",[1158,13449,13450],{},"Yes**",[1137,13452,13453,13458,13463],{},[1158,13454,13455],{},[30,13456,13457],{},"HF_TOKEN",[1158,13459,13460],{},[30,13461,13462],{},"\"hf_your-token\"",[1158,13464,13450],{},[1137,13466,13467,13472,13477],{},[1158,13468,13469],{},[30,13470,13471],{},"LOCAL_NIM_CACHE",[1158,13473,13474],{},[30,13475,13476],{},"~/.cache/nim",[1158,13478,13450],{},[1137,13480,13481,13486,13493],{},[1158,13482,13483],{},[30,13484,13485],{},"INVOKEAI_URL",[1158,13487,13488,106,13490],{},[30,13489,13042],{},[30,13491,13492],{},"\"http://192.168.1.100:9090\"",[1158,13494,13404],{},[1137,13496,13497,13502,13510],{},[1158,13498,13499],{},[30,13500,13501],{},"BOARD_ID",[1158,13503,13504,106,13507],{},[30,13505,13506],{},"\"my-gallery-board\"",[30,13508,13509],{},"\"flux-gallery\"",[1158,13511,13404],{},[1137,13513,13514,13519,13527],{},[1158,13515,13516],{},[30,13517,13518],{},"OUTPUT_DIRECTORY",[1158,13520,13521,106,13524],{},[30,13522,13523],{},"\"C:\\\\GeneratedImages\"",[30,13525,13526],{},"\"D:\\\\flux-output\"",[1158,13528,13404],{},[11,13530,13531,13532,13534,13535,13539],{},"*Required only when using NVIDIA hosted Flux service (",[30,13533,13426],{}," starts with \"",[20,13536,13537],{"href":13537,"rel":13538},"https://ai.api.nvidia.com",[24],"\")\n**Required only when using local NIM server",[56,13541,13543],{"id":13542},"using-the-flux1-dev-nvidia-nim-for-text-to-image-generation","Using the Flux.1-dev NVIDIA NIM for text-to-image generation",[11,13545,13546],{},[2718,13547],{"alt":13548,"src":13549},"AI-generated image of a desert nomad in a sandy landscape","/static/flux/desert_nomad.png",[11,13551,13552],{},"On NVIDIA GeForce RTX AI PCs, the best way to do AI image inference is by using NVIDIA NIMs. Windows currently has beta support for running NVIDIA NIMs in WSL with Podman (a program for running containers, similar to Docker).",[11,13554,13555,13556,13559],{},"NVIDIA provides an installer that installs a WSL distribution with all dependencies installed. You can find those resources here: ",[20,13557,13083],{"href":13083,"rel":13558},[24]," (WSL2 is required for hosting any NIM. Refer to the official NVIDIA NIM on WSL2 documentation for setup instructions.)",[736,13561,13563],{"id":13562},"how-it-works","How It Works",[11,13565,13566],{},"You can request an image to be generated by simply saying something like:",[11,13568,13303],{},[11,13570,13571,13572,13576],{},"The Flux Plugin will make an API request to the Flux NIM URL (configured in your config.json, defaults to ",[20,13573,13574],{"href":13574,"rel":13575},"http://localhost:8000",[24],").",[736,13578,13580],{"id":13579},"asynchronous-processing","Asynchronous Processing",[11,13582,13583],{},"The G-Assist chat assistant has a timeout of 10 seconds, so the chat assistant returns immediately with:",[459,13585,13588],{"className":13586,"code":13587,"language":997},[995],"flux> Your image generation request is in progress! Prompt: \"a cat piloting a spaceship\"\n",[30,13589,13587],{"__ignoreMap":464},[11,13591,13592],{},"The image generation request runs on a separate thread, since the image generation process with the NVIDIA NIM can take up to 30 seconds (depending on the number of steps, 50 steps is used in the plugin for best results). The plugin then creates an image file from the base64 encoded image in the response from the Flux NIM server, and it sets this image as your desktop background image.",[736,13594,13596],{"id":13595},"nim-management","NIM Management",[11,13598,13599],{},"There are also commands for starting and stopping the Flux NIM, which runs Podman commands inside of the NVIDIA-Workbench WSL distribution:",[76,13601,13602,13608,13614],{},[79,13603,13604,13607],{},[15,13605,13606],{},"Start NIM",": \"hey flux, start the Flux NIM server\"",[79,13609,13610,13613],{},[15,13611,13612],{},"Stop NIM",": \"hey flux, stop the Flux NIM server\"",[79,13615,13616,13619],{},[15,13617,13618],{},"Check Status",": \"hey flux, check if the NIM server is running\"",[11,13621,13622],{},[2718,13623],{"alt":13624,"src":13625},"Flux Plug-in controls showing start, stop, and status buttons for NIM server management","/static/flux/controls.png",[736,13627,13354],{"id":13628},"configuration-1",[11,13630,13631,13632,13634],{},"Make sure your ",[30,13633,12952],{}," includes the necessary credentials:",[76,13636,13637,13642,13647],{},[79,13638,13639,13641],{},[30,13640,4786],{},": Your NVIDIA NGC API key for downloading models",[79,13643,13644,13646],{},[30,13645,13457],{},": Your Hugging Face token for model access",[79,13648,13649,13651],{},[30,13650,13471],{},": Path to your local NIM cache directory",[56,13653,13655],{"id":13654},"image-to-image-generation-with-flux-kontext-and-invokeai","Image-to-image generation with Flux Kontext and InvokeAI",[11,13657,13658],{},[2718,13659],{"alt":13660,"src":13661},"AI-generated image of a helicopter flying over New York City skyline","/static/flux/nyc_heli.png",[11,13663,13664],{},[2718,13665],{"alt":13666,"src":13667},"AI-generated watercolor-style transformation of the same NYC helicopter image using Flux Kontext","/static/flux/nyc_heli_watercolor.png",[11,13669,13670],{},"The Flux Plug-in supports image-to-image generation using an open source image generation tool called InvokeAI. This tool is similar to ComfyUI and it has solid API support.",[11,13672,13673,13676,13677,13679,13680,13685],{},[15,13674,13675],{},"Update:"," NVIDIA released the FLUX.1 Kontext ",[151,13678,10715],{}," NIM microservice in August 2025, which can be used instead of InvokeAI for image-to-image workflows! The NIM version offers optimized performance with quantization reducing VRAM requirements from 24GB to 12GB (FP8 on Ada Generation GPUs) and 7GB (FP4). Check the ",[20,13681,13684],{"href":13682,"rel":13683},"https://blogs.nvidia.com/blog/rtx-ai-garage-flux-kontext-nim-microservice-siggraph/",[24],"NVIDIA blog announcement"," for details.",[11,13687,13688],{},"You can interact with the InvokeAI program in a few different ways:",[736,13690,13692],{"id":13691},"screenshot-based-image-generation","Screenshot-based Image Generation",[76,13694,13695,13701,13704],{},[79,13696,13697,13700],{},[15,13698,13699],{},"Upload your latest screenshot"," and perform image-to-image generation using Flux Kontext. This allows you to apply any type of manipulation to your screenshot (for example, you can say \"hey flux, use kontext to make it in the style of a cartoon\")",[79,13702,13703],{},"The plugin automatically finds your most recent screenshot and uploads it to InvokeAI",[79,13705,13706],{},"You can provide custom prompts to guide the transformation process",[736,13708,13710],{"id":13709},"processing-control","Processing Control",[76,13712,13713,13719],{},[79,13714,13715,13718],{},[15,13716,13717],{},"Pause or resume processing",": This is useful if you are playing a GPU intensive game. You can pause processing, but still submit image-to-image generation tasks using Flux Kontext. The tasks will be queued and they can be resumed later when your GPU is not busy with other tasks.",[79,13720,13721],{},"Monitor the processing queue status and control when generation happens",[736,13723,13725],{"id":13724},"memory-management","Memory Management",[76,13727,13728,13734,13737],{},[79,13729,13730,13733],{},[15,13731,13732],{},"Empty the model cache",": InvokeAI keeps models cached between generation, but you can empty the model cache by simply telling it to do so",[79,13735,13736],{},"This helps free up VRAM when you're not actively using InvokeAI",[79,13738,13739],{},"Useful for switching between different AI workloads or when playing games",[736,13741,13743],{"id":13742},"setup-requirements","Setup Requirements",[11,13745,13746],{},"To use the image-to-image features, you'll need:",[76,13748,13749,13754,13757],{},[79,13750,13751,13752,748],{},"InvokeAI installed and running locally (typically on ",[30,13753,13126],{},[79,13755,13756],{},"Flux Kontext model loaded in InvokeAI",[79,13758,13759,13760,4231],{},"Proper configuration in your ",[30,13761,12952],{},[56,13763,13765],{"id":13764},"supported-commands","Supported Commands",[11,13767,13768],{},"The Flux Plug-in for G-Assist supports the following commands:",[1131,13770,13771,13784],{},[1134,13772,13773],{},[1137,13774,13775,13778,13781],{},[1140,13776,13777],{},"Function",[1140,13779,13780],{},"Description",[1140,13782,13783],{},"Example",[1153,13785,13786,13799,13812,13825,13838,13851,13864,13877,13890,13903],{},[1137,13787,13788,13793,13796],{},[1158,13789,13790],{},[30,13791,13792],{},"flux_nim_ready_check",[1158,13794,13795],{},"Tests health endpoints of the Flux NIM server",[1158,13797,13798],{},"\"hey flux, check if the flux nim server is ready\"",[1137,13800,13801,13806,13809],{},[1158,13802,13803],{},[30,13804,13805],{},"check_nim_status",[1158,13807,13808],{},"Checks if the Flux NIM server is running",[1158,13810,13811],{},"\"hey flux, check if the nim server is running\"",[1137,13813,13814,13819,13822],{},[1158,13815,13816],{},[30,13817,13818],{},"stop_nim",[1158,13820,13821],{},"Stops the Flux NIM server",[1158,13823,13824],{},"\"hey flux, stop the flux nim server\"",[1137,13826,13827,13832,13835],{},[1158,13828,13829],{},[30,13830,13831],{},"start_nim",[1158,13833,13834],{},"Starts the Flux NIM server",[1158,13836,13837],{},"\"hey flux, start the flux nim server\"",[1137,13839,13840,13845,13848],{},[1158,13841,13842],{},[30,13843,13844],{},"generate_image",[1158,13846,13847],{},"Generates an image from text prompt using Flux",[1158,13849,13850],{},"\"hey flux, generate an image of a cyberpunk city\"",[1137,13852,13853,13858,13861],{},[1158,13854,13855],{},[30,13856,13857],{},"generate_image_using_kontext",[1158,13859,13860],{},"Performs image-to-image generation using Flux Kontext",[1158,13862,13863],{},"\"hey flux, use kontext to make it a cartoon style\"",[1137,13865,13866,13871,13874],{},[1158,13867,13868],{},[30,13869,13870],{},"invokeai_status",[1158,13872,13873],{},"Checks the status of the InvokeAI service",[1158,13875,13876],{},"\"hey flux, check invokeai status\"",[1137,13878,13879,13884,13887],{},[1158,13880,13881],{},[30,13882,13883],{},"pause_invokeai_processor",[1158,13885,13886],{},"Pauses the InvokeAI processing queue",[1158,13888,13889],{},"\"hey flux, pause the invokeai processor\"",[1137,13891,13892,13897,13900],{},[1158,13893,13894],{},[30,13895,13896],{},"resume_invokeai_processor",[1158,13898,13899],{},"Resumes the InvokeAI processing queue",[1158,13901,13902],{},"\"hey flux, resume the invokeai processor\"",[1137,13904,13905,13910,13913],{},[1158,13906,13907],{},[30,13908,13909],{},"invokeai_empty_model_cache",[1158,13911,13912],{},"Empties the InvokeAI model cache to free VRAM",[1158,13914,13915],{},"\"hey flux, empty the invokeai model cache\"",[56,13917,13919],{"id":13918},"logging","Logging",[11,13921,13922,13923,13926],{},"Your plugin automatically logs to ",[30,13924,13925],{},"flux_plugin.log"," in your user's profile directory. It tracks:",[76,13928,13929,13932,13935,13938],{},[79,13930,13931],{},"Plugin startup and shutdown",[79,13933,13934],{},"Command reception and processing",[79,13936,13937],{},"Error conditions",[79,13939,13940],{},"Function execution details",[56,13942,13944],{"id":13943},"troubleshooting-tips","Troubleshooting Tips",[76,13946,13947,13953,13959,13965],{},[79,13948,13949,13952],{},[15,13950,13951],{},"Plugin not starting?"," Check if Python 3.12+ is installed and in PATH",[79,13954,13955,13958],{},[15,13956,13957],{},"Communication errors?"," Verify pywin32 is installed correctly",[79,13960,13961,13964],{},[15,13962,13963],{},"Commands not working?"," Double-check your command registration",[79,13966,13967,13970],{},[15,13968,13969],{},"Missing logs?"," Ensure write permissions in user profile directory",[56,13972,13974],{"id":13973},"want-to-contribute","Want to Contribute?",[11,13976,13977,13978,13981],{},"We'd love your help making this template even better! Check out ",[30,13979,13980],{},"CONTRIBUTING.md"," for guidelines on how to contribute.",[56,13983,13985],{"id":13984},"license","License",[11,13987,13988,13989,13992],{},"This project is licensed under the Apache License 2.0 - see the ",[30,13990,13991],{},"LICENSE"," file for details.",[56,13994,1638],{"id":13995},"hardware",[11,13997,13998],{},"The Flux Plug-in for G-Assist was developed and tested on a PC with a GeForce RTX 4090 GPU.",[589,14000,14001],{},"html pre.shiki code .srTi1, html code.shiki .srTi1{--shiki-default:#6F42C1;--shiki-dark:#B392F0;--shiki-sepia:#A6E22E}html pre.shiki code .s7F3e, html code.shiki .s7F3e{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#AE81FF}html pre.shiki code .sstjo, html code.shiki .sstjo{--shiki-default:#032F62;--shiki-dark:#9ECBFF;--shiki-sepia:#E6DB74}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html .sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html.sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html pre.shiki code .sMOD_, html code.shiki .sMOD_{--shiki-default:#24292E;--shiki-dark:#E1E4E8;--shiki-sepia:#F8F8F2}html pre.shiki code .s-m8C, html code.shiki .s-m8C{--shiki-default:#005CC5;--shiki-default-font-style:inherit;--shiki-dark:#79B8FF;--shiki-dark-font-style:inherit;--shiki-sepia:#66D9EF;--shiki-sepia-font-style:italic}html pre.shiki code .sCZoN, html code.shiki .sCZoN{--shiki-default:#032F62;--shiki-dark:#9ECBFF;--shiki-sepia:#CFCFC2}html pre.shiki code .sC2Qs, html code.shiki .sC2Qs{--shiki-default:#D73A49;--shiki-dark:#F97583;--shiki-sepia:#F92672}html pre.shiki code .sq6CD, html code.shiki .sq6CD{--shiki-default:#D73A49;--shiki-default-font-style:inherit;--shiki-dark:#F97583;--shiki-dark-font-style:inherit;--shiki-sepia:#66D9EF;--shiki-sepia-font-style:italic}",{"title":464,"searchDepth":488,"depth":488,"links":14003},[14004,14011,14012,14023,14026,14032,14038,14039,14040,14041,14042,14043],{"id":12672,"depth":488,"text":12673,"children":14005},[14006,14007,14008,14009,14010],{"id":12676,"depth":500,"text":12677},{"id":12706,"depth":500,"text":12707},{"id":12736,"depth":500,"text":12737},{"id":12772,"depth":500,"text":12773},{"id":12802,"depth":500,"text":12803},{"id":12823,"depth":488,"text":12824},{"id":12854,"depth":488,"text":12855,"children":14013},[14014,14015,14016,14017,14018,14019,14020,14021,14022],{"id":12858,"depth":500,"text":12859},{"id":12868,"depth":500,"text":12869},{"id":12921,"depth":500,"text":12922},{"id":12941,"depth":500,"text":12942},{"id":13065,"depth":500,"text":13066},{"id":13110,"depth":500,"text":13111},{"id":13129,"depth":500,"text":13130},{"id":13296,"depth":500,"text":13297},{"id":13315,"depth":500,"text":13316},{"id":13353,"depth":488,"text":13354,"children":14024},[14025],{"id":13368,"depth":500,"text":13369},{"id":13542,"depth":488,"text":13543,"children":14027},[14028,14029,14030,14031],{"id":13562,"depth":500,"text":13563},{"id":13579,"depth":500,"text":13580},{"id":13595,"depth":500,"text":13596},{"id":13628,"depth":500,"text":13354},{"id":13654,"depth":488,"text":13655,"children":14033},[14034,14035,14036,14037],{"id":13691,"depth":500,"text":13692},{"id":13709,"depth":500,"text":13710},{"id":13724,"depth":500,"text":13725},{"id":13742,"depth":500,"text":13743},{"id":13764,"depth":488,"text":13765},{"id":13918,"depth":488,"text":13919},{"id":13943,"depth":488,"text":13944},{"id":13973,"depth":488,"text":13974},{"id":13984,"depth":488,"text":13985},{"id":13995,"depth":488,"text":1638},"2025-07-20","A Plug-in for Project G-Assist that puts the power of AI image generation right at your fingertips",[14047],{"link":14048,"site":11126},"https://x.com/briancaffey/status/1946684573095497992","/static/flux/flux_plugin_for_project_g_assist.png",{},"/2025/07/17/flux-plugin-for-project-g-assist-hackathon",{"title":12652,"description":14045},"2025/07/17/flux-plugin-for-project-g-assist-hackathon",[11133,614,14055,5404,11802],"rtx","3WPXvSZyqmc7YwoTW_DLLpdz2VYBdrnSkuWmKTQLkPM",{"id":14058,"title":14059,"body":14060,"comments":609,"date":16096,"description":16097,"draft":602,"extension":605,"external":16098,"image":16100,"meta":16101,"navigation":609,"path":16102,"seo":16103,"stem":16104,"tags":16105,"__hash__":16107},"blog/2025/05/27/mediation-simulator-project-for-nvidia-agent-intelligence-toolkit.md","Mediation Simulator: My submission for the NVIDIA Agent Intelligence Toolkit Hackathon",{"type":8,"value":14061,"toc":16077},[14062,14067,14072,14079,14088,14091,14095,14098,14101,14121,14125,14128,14148,14151,14160,14166,14175,14181,14185,14188,14234,14238,14241,14309,14316,14319,14325,14329,14332,14335,14340,14347,14351,14354,14357,15148,15153,15157,15160,15166,15169,15175,15179,15182,15186,15189,15193,15196,15618,15625,15631,15634,15729,15733,15736,15740,15743,15784,15787,15791,15794,15797,15800,16019,16022,16028,16031,16034,16037,16057,16060,16064,16067,16071,16074],[14063,14064,14066],"h1",{"id":14065},"building-mediation-simulator-for-the-nvidia-agent-intelligence-toolkit-hackathon","Building Mediation Simulator for the NVIDIA Agent Intelligence Toolkit Hackathon!",[11,14068,14069,14071],{},[15,14070,11825],{}," This article was published in May 2025. As of March 2026, Qwen3.5 is available (the article references Qwen3-8b). The NVIDIA Agent Intelligence Toolkit and associated NIMs may have updated versions since publication. Check official documentation for the latest releases.",[11,14073,14074,14075,14078],{},"I'm excited to share a project I've been building for the NVIDIA Agent Intelligence Toolkit Hackathon: ",[15,14076,14077],{},"Mediation Simulator","! The NVIDIA Agent Intelligence Toolkit (or AgentIQ Toolkit, as it's often called) is a powerful open-source library designed for connecting, evaluating, and accelerating teams of AI agents. My goal? To see if I could leverage this toolkit to build AI agent teams capable of simulating the entire, complex process of a law school mediation competition.",[11,14080,14081,14082,14087],{},"Check out my ",[20,14083,14086],{"href":14084,"rel":14085},"https://x.com/briancaffey/status/1926036369597510117",[24],"𝕏 Post"," that introduces the project:",[14089,14090],"mediation-simulator-tweet",{},[56,14092,14094],{"id":14093},"what-is-mediation-simulator-and-why-mediation","What is Mediation Simulator? And Why Mediation?",[11,14096,14097],{},"At its core, Mediation Simulator is my attempt to model the nuanced, semi-structured, three-way conversations that happen in mediation. Law school mediation tournaments are events where students practice negotiation and dispute resolution skills. This seemed like a really interesting challenge! How do you get language models to effectively navigate such a dynamic environment?",[11,14099,14100],{},"Mediation competitions also present some unique data challenges that are perfect for AI:",[76,14102,14103,14109,14115],{},[79,14104,14105,14108],{},[15,14106,14107],{},"Layered Information:"," There are \"common facts\" known to everyone, and \"confidential facts\" privy only to one party (and sometimes shared strategically with the mediator or the other side).",[79,14110,14111,14114],{},[15,14112,14113],{},"Dynamic Conversations:"," The main discussion involves all three parties (mediator and two disputants), but it also features \"caucuses\"—private, two-way conversations between the mediator and one party.",[79,14116,14117,14120],{},[15,14118,14119],{},"Creative Scenarios:"," The cases are entirely fictional, often involving made-up companies, fictional countries, and even fictional currencies! This is a deliberate choice to help students avoid real-world biases. And guess what? AI is really good at generating fake data!",[736,14122,14124],{"id":14123},"what-i-built","What I built",[11,14126,14127],{},"Mediation Simulator consists of three main components:",[700,14129,14130,14136,14142],{},[79,14131,14132,14135],{},[15,14133,14134],{},"Case Generation Workflow",": A CLI tool that uses LLMs to generate realistic mediation case scenarios, complete with common facts, confidential information for each party, and supporting documents. This workflow creates the foundation for all mediation simulations. The data for the case scenarios is saved in both YAML files and in my Redis database. I'll talk about why I did chose to store data in local files and on Redis later in this article.",[79,14137,14138,14141],{},[15,14139,14140],{},"Automated Mediation Workflow",": Another CLI tool that orchestrates a full mediation session between three AI agents (mediator, requesting party, and responding party). This workflow simulates the entire mediation process, from opening statements through negotiation to conclusion, with the AI agents engaging in realistic dialogue based on their roles and the case information. A clerk agent helps to guide the conversation, controlling who the next speaker should be based on a summary of what has already been said by different parties.",[79,14143,14144,14147],{},[15,14145,14146],{},"Interactive Mediation API",": A REST API that allows a human user to participate in a mediation session by taking on the role of either the requesting or responding party. The API manages the session state and coordinates the interaction between the human participant and the AI mediator and opposing party. The conversation history for the mediation session is stored in Redis using a memory backend that I implemented with NVIDIA Agent Intelligence Toolkit.",[11,14149,14150],{},"To make the results of these workflows easily viewable, I also built two web interfaces:",[76,14152,14153],{},[79,14154,14155,14156,14159],{},"A ",[15,14157,14158],{},"Viewer Interface"," that displays the full three-party dialogue from automated mediation sessions, making it easy to review and analyze the AI agents' interactions.",[11,14161,14162],{},[2718,14163],{"alt":14164,"src":14165},"Mediation Simulator Viewer","/static/mediation-simulator/mediation_simulator_viewer.png",[76,14167,14168],{},[79,14169,14170,14171,14174],{},"An ",[15,14172,14173],{},"Interactive Interface"," that provides a chat-like experience for human participants in the interactive mediation mode, with real-time updates and a clean, intuitive design.",[11,14176,14177],{},[2718,14178],{"alt":14179,"src":14180},"Mediation Simulator Interactive","/static/mediation-simulator/interactive_mediation_screenshot.png",[736,14182,14184],{"id":14183},"the-genesis-from-idea-to-data","The Genesis: From Idea to Data",[11,14186,14187],{},"The first major hurdle was generating the foundational case data. Here's how that unfolded:",[700,14189,14190,14196,14202,14208,14228],{},[79,14191,14192,14195],{},[15,14193,14194],{},"Deep Dive Research:"," I started by brainstorming with an LLM (OpenAI's o3, in this case) to get a comprehensive understanding of law school mediation competitions. I wanted to know everything: the rules, structure, participants, judging criteria and different types of cases.",[79,14197,14198,14201],{},[15,14199,14200],{},"Prompt Engineering for Cases:"," With that knowledge, I tasked another LLM (GPT-4o) with generating prompts to be used for creating diverse and realistic (albeit fictional) mediation case scenarios.",[79,14203,14204,14207],{},[15,14205,14206],{},"Case Generation:"," Using these prompts, I generated sets of distinct cases facts and related documents.",[79,14209,14210,14213,14214],{},[15,14211,14212],{},"Structuring with LangGraph:"," To manage the data for each case, I used LangGraph. I designed a state object to encapsulate all crucial elements:\n",[76,14215,14216,14219,14222],{},[79,14217,14218],{},"Common facts.",[79,14220,14221],{},"Confidential facts for both the requesting and responding parties.",[79,14223,14224,14227],{},[15,14225,14226],{},"Related Documents!"," This was my own little twist. I wanted to test RAG (Retrieval Augmented Generation) integration within the agentic workflow. Could parties use tools to search for information in these documents to bolster their arguments during mediation? Ultimately I couldn't really get this to work. I'll share more on why later in this article.",[79,14229,14230,14233],{},[15,14231,14232],{},"Data Persistence:"," With the data structured, I saved it in accessible formats: the LangGraph state was serialized to YAML, and the case details (like facts and documents) were stored in Markdown files. I also stored same LangGraph state to Redis using a simple JSON string.",[736,14235,14237],{"id":14236},"orchestrating-the-simulation","Orchestrating the Simulation",[11,14239,14240],{},"With the case data ready, the next step was to build out the mediation simulation itself:",[700,14242,14243,14266,14293,14299],{},[79,14244,14245,14248,14249],{},[15,14246,14247],{},"Defining the Flow:"," I broke down the mediation process into its typical phases:\n",[76,14250,14251,14254,14257,14260,14263],{},[79,14252,14253],{},"Opening Statements",[79,14255,14256],{},"Information Gathering",[79,14258,14259],{},"Caucuses (for each party)",[79,14261,14262],{},"Negotiation",[79,14264,14265],{},"Conclusion",[79,14267,14268,14271,14272],{},[15,14269,14270],{},"Assembling the Agent Team:"," I set up a LangGraph graph consisting of:\n",[76,14273,14274,14280,14287],{},[79,14275,14155,14276,14279],{},[15,14277,14278],{},"Mediator"," agent",[79,14281,14282,14283,14286],{},"Two ",[15,14284,14285],{},"Party"," agents (Requesting and Responding parties)",[79,14288,14155,14289,14292],{},[15,14290,14291],{},"Clerk"," agent, whose job is to help manage the conversation flow and transition the simulation between the different phases.",[79,14294,14295,14298],{},[15,14296,14297],{},"Dynamic Prompting:"," The prompts for the mediator and the parties (both when initiating a statement and when responding) change significantly based on the current phase of the mediation. Critically, these prompts also dynamically include a summary log of what has already been said, providing context.",[79,14300,14301,14304,14305,14308],{},[15,14302,14303],{},"Message Logging:"," Each time a party speaks, I store the message using my Redis backend and also store additional metadata in the ",[30,14306,14307],{},"additional_kwargs"," section of each message, such as the speaker, the current phase of mediation and a summary of the message (the summary is generated by another LLM call that just summarizes the response.)",[11,14310,14311,14312,14315],{},"Getting to a functional mediation workflow was crucial! It allowed me to see the actual dialogue unfold and immediately highlighted areas for improvement. For instance, I realized that prompts needed to guide parties to ask one clear question at a time, directed at a specific participant, rather than posing multiple questions to several people at once. This really helps keep the simulated conversation straightforward and more realistic. I also had to instruct the LLM to use the names of the different parties, and I had to provide the names of the parties to the prompt. Without this instruction the LLM would give responses like this: \"Hello ",[151,14313,14314],{},"Requesting Party Name",", thank you for sharing your opinion.\"",[11,14317,14318],{},"Here's a look at the main workflow generated from the LangGraph code:",[11,14320,14321],{},[2718,14322],{"alt":14323,"src":14324},"Mediation Simulator Workflow with LangGraph","/static/mediation-simulator/mediation_workflow.png",[736,14326,14328],{"id":14327},"bringing-it-to-life-the-vibe-coding-web-viewer","Bringing it to Life: The \"Vibe Coding\" Web Viewer!",[11,14330,14331],{},"Reading through raw Markdown files or terminal output to follow a complex, multi-turn mediation isn't ideal. I needed a better way to visualize the results! And this is where a bit of \"vibe coding\" came in incredibly handy.",[11,14333,14334],{},"I prompted an LLM to generate a single HTML page using Vue.js and Tailwind CSS. My requirements were simple: list all generated mediation cases, and when a case is selected, display the full dialogue. The amazing part? I never actually looked at the generated code in detail! I was able to make incremental improvements by simply describing changes or new features I wanted, and the LLM iterated until it was almost exactly what I envisioned. This was super easy and fast! I also made a page that lists all of the different mediation sessions with cover images that I generated with the NVIDIA Flux.1 Dev NIM:",[11,14336,14337],{},[2718,14338],{"alt":14164,"src":14339},"/static/mediation-simulator/viewer.png",[11,14341,14342,14343,14346],{},"Having this simple web interface has been a game-changer for reviewing simulations. Plus, keeping it in my project repository means I can easily host it on GitHub Pages at ",[30,14344,14345],{},"briancaffey.github.io/mediation-simulator"," to share the results of my project. It's just so much better than staring at text files, and took just a few minutes to put together.",[56,14348,14350],{"id":14349},"a-deeper-look-into-the-nvidia-agent-intelligence-toolkit","A deeper look into the NVIDIA Agent Intelligence Toolkit",[11,14352,14353],{},"Now, I know what some developers think about LLM frameworks like LangChain/LangGraph, LlamaIndex, and CrewAI—there's often a bit of \"framework fatigue.\" But hear me out! The NVIDIA Agent Intelligence Toolkit brings all of these Frameworks together in a manageable way and it makes it really easy to not only write an agentic program, but it also makes it really easy to read other programs written with the framework.",[11,14355,14356],{},"The key to understanding the AIQ Toolkit is the config files. These are YAML files that neatly list out all of the dependencies of your agentic application. Let's take a look at the config file I made for mediation simulator. It's a long file, so I'll share it and then break down the important sections:",[459,14358,14362],{"className":14359,"code":14360,"language":14361,"meta":464,"style":464},"language-yaml shiki shiki-themes github-light github-dark monokai","general:\n  use_uvloop: true\n  front_end:\n    _type: fastapi\n    cors:\n      allow_origins: ['*']\n      allow_methods:\n        - GET\n        - POST\n        - OPTIONS\n    endpoints:\n      - path: /case/{case_id}\n        method: GET\n        description: Gets the mediation case for the given case ID.\n        function_name: get_mediation_case\n      - path: /case/{case_id}/session/{session_id}\n        method: GET\n        description: Gets the mediation session data for the given case ID and session ID.\n        function_name: get_mediation_session\n      - path: /case/{case_id}/session/{session_id}/send\n        method: POST\n        description: Sends a message to the mediation session for the given case ID and session ID.\n        function_name: send_message_to_mediation_session\n  telemetry:\n    enabled: false\n    tracing:\n      phoenix:\n        _type: phoenix\n        endpoint: http://localhost:6006/v1/traces\n        project: default\n\nretrievers:\n  milvus_retriever:\n    _type: milvus_retriever\n    uri: \"http://localhost:19530\"\n    embedding_model: \"nv-embedqa-e5-v5\"\n    collection_name: \"aiq_case_documents\"\n    vector_field: \"embedding\"\n    search_params:\n      metric_type: \"IP\" # works best with nv-embedqa-e5-v5\n\nllms:\n  nim_llm:\n    _type: nim\n    base_url: http://192.168.5.96:1234/v1\n    model_name: qwen3-8b\n    max_tokens: 10000\n    temperature: 0.7\n  mediation_llm:\n    _type: nim\n    base_url: http://192.168.5.96:1234/v1\n    model_name: qwen3-8b\n    max_tokens: 10000\n    temperature: 0.7\n\nmemory:\n  redis_memory:\n    _type: redis_memory\n    connection_url: redis://localhost:6379/0\n\nfunctions:\n  case_document_rag:\n    _type: case_document_rag\n    retriever: milvus_retriever\n    llm_name: nim_llm\n    collection_name: \"mediation_simulator_case_documents\"\n    top_k: 5\n  case_query_agent:\n    _type: case_query_agent\n    llm_name: nim_llm\n    tool_names:\n      - case_document_rag\n    verbose: true\n    max_iterations: 5\n\n  # server route functions\n  get_mediation_case:\n    _type: server/get_mediation_case\n  get_mediation_session:\n    _type: server/get_mediation_session\n  send_message_to_mediation_session:\n    _type: mediation\n\nembedders:\n  nv-embedqa-e5-v5:\n    _type: nim\n    base_url: http://192.168.5.96:8000/v1\n    model_name: nvidia/nv-embedqa-e5-v5\n\nworkflow:\n  _type: mediation\n  llm: mediation_llm\n  data_dir: ./data\n","yaml",[30,14363,14364,14373,14383,14390,14400,14407,14419,14426,14434,14441,14448,14455,14468,14477,14487,14497,14508,14516,14525,14534,14545,14553,14562,14571,14578,14588,14595,14602,14612,14622,14632,14636,14643,14650,14659,14669,14679,14689,14699,14706,14719,14723,14730,14737,14746,14756,14766,14776,14786,14793,14801,14809,14817,14825,14833,14837,14844,14851,14860,14870,14874,14881,14888,14897,14906,14916,14925,14935,14942,14951,14959,14966,14972,14981,14990,14994,14999,15007,15017,15025,15035,15043,15053,15058,15066,15074,15083,15093,15103,15108,15116,15126,15137],{"__ignoreMap":464},[151,14365,14366,14370],{"class":469,"line":470},[151,14367,14369],{"class":14368},"s5clZ","general",[151,14371,14372],{"class":503},":\n",[151,14374,14375,14378,14380],{"class":469,"line":488},[151,14376,14377],{"class":14368},"  use_uvloop",[151,14379,6208],{"class":503},[151,14381,14382],{"class":477},"true\n",[151,14384,14385,14388],{"class":469,"line":500},[151,14386,14387],{"class":14368},"  front_end",[151,14389,14372],{"class":503},[151,14391,14392,14395,14397],{"class":469,"line":509},[151,14393,14394],{"class":14368},"    _type",[151,14396,6208],{"class":503},[151,14398,14399],{"class":481},"fastapi\n",[151,14401,14402,14405],{"class":469,"line":517},[151,14403,14404],{"class":14368},"    cors",[151,14406,14372],{"class":503},[151,14408,14409,14412,14414,14417],{"class":469,"line":534},[151,14410,14411],{"class":14368},"      allow_origins",[151,14413,8365],{"class":503},[151,14415,14416],{"class":481},"'*'",[151,14418,3691],{"class":503},[151,14420,14421,14424],{"class":469,"line":1413},[151,14422,14423],{"class":14368},"      allow_methods",[151,14425,14372],{"class":503},[151,14427,14428,14431],{"class":469,"line":1418},[151,14429,14430],{"class":503},"        - ",[151,14432,14433],{"class":481},"GET\n",[151,14435,14436,14438],{"class":469,"line":2462},[151,14437,14430],{"class":503},[151,14439,14440],{"class":481},"POST\n",[151,14442,14443,14445],{"class":469,"line":2471},[151,14444,14430],{"class":503},[151,14446,14447],{"class":481},"OPTIONS\n",[151,14449,14450,14453],{"class":469,"line":2480},[151,14451,14452],{"class":14368},"    endpoints",[151,14454,14372],{"class":503},[151,14456,14457,14460,14463,14465],{"class":469,"line":2489},[151,14458,14459],{"class":503},"      - ",[151,14461,14462],{"class":14368},"path",[151,14464,6208],{"class":503},[151,14466,14467],{"class":481},"/case/{case_id}\n",[151,14469,14470,14473,14475],{"class":469,"line":2497},[151,14471,14472],{"class":14368},"        method",[151,14474,6208],{"class":503},[151,14476,14433],{"class":481},[151,14478,14479,14482,14484],{"class":469,"line":3140},[151,14480,14481],{"class":14368},"        description",[151,14483,6208],{"class":503},[151,14485,14486],{"class":481},"Gets the mediation case for the given case ID.\n",[151,14488,14489,14492,14494],{"class":469,"line":3149},[151,14490,14491],{"class":14368},"        function_name",[151,14493,6208],{"class":503},[151,14495,14496],{"class":481},"get_mediation_case\n",[151,14498,14499,14501,14503,14505],{"class":469,"line":3158},[151,14500,14459],{"class":503},[151,14502,14462],{"class":14368},[151,14504,6208],{"class":503},[151,14506,14507],{"class":481},"/case/{case_id}/session/{session_id}\n",[151,14509,14510,14512,14514],{"class":469,"line":3167},[151,14511,14472],{"class":14368},[151,14513,6208],{"class":503},[151,14515,14433],{"class":481},[151,14517,14518,14520,14522],{"class":469,"line":3175},[151,14519,14481],{"class":14368},[151,14521,6208],{"class":503},[151,14523,14524],{"class":481},"Gets the mediation session data for the given case ID and session ID.\n",[151,14526,14527,14529,14531],{"class":469,"line":3184},[151,14528,14491],{"class":14368},[151,14530,6208],{"class":503},[151,14532,14533],{"class":481},"get_mediation_session\n",[151,14535,14536,14538,14540,14542],{"class":469,"line":3193},[151,14537,14459],{"class":503},[151,14539,14462],{"class":14368},[151,14541,6208],{"class":503},[151,14543,14544],{"class":481},"/case/{case_id}/session/{session_id}/send\n",[151,14546,14547,14549,14551],{"class":469,"line":3720},[151,14548,14472],{"class":14368},[151,14550,6208],{"class":503},[151,14552,14440],{"class":481},[151,14554,14555,14557,14559],{"class":469,"line":3729},[151,14556,14481],{"class":14368},[151,14558,6208],{"class":503},[151,14560,14561],{"class":481},"Sends a message to the mediation session for the given case ID and session ID.\n",[151,14563,14564,14566,14568],{"class":469,"line":3735},[151,14565,14491],{"class":14368},[151,14567,6208],{"class":503},[151,14569,14570],{"class":481},"send_message_to_mediation_session\n",[151,14572,14573,14576],{"class":469,"line":3745},[151,14574,14575],{"class":14368},"  telemetry",[151,14577,14372],{"class":503},[151,14579,14580,14583,14585],{"class":469,"line":3754},[151,14581,14582],{"class":14368},"    enabled",[151,14584,6208],{"class":503},[151,14586,14587],{"class":477},"false\n",[151,14589,14590,14593],{"class":469,"line":3760},[151,14591,14592],{"class":14368},"    tracing",[151,14594,14372],{"class":503},[151,14596,14597,14600],{"class":469,"line":3773},[151,14598,14599],{"class":14368},"      phoenix",[151,14601,14372],{"class":503},[151,14603,14604,14607,14609],{"class":469,"line":3782},[151,14605,14606],{"class":14368},"        _type",[151,14608,6208],{"class":503},[151,14610,14611],{"class":481},"phoenix\n",[151,14613,14614,14617,14619],{"class":469,"line":3791},[151,14615,14616],{"class":14368},"        endpoint",[151,14618,6208],{"class":503},[151,14620,14621],{"class":481},"http://localhost:6006/v1/traces\n",[151,14623,14624,14627,14629],{"class":469,"line":3803},[151,14625,14626],{"class":14368},"        project",[151,14628,6208],{"class":503},[151,14630,14631],{"class":481},"default\n",[151,14633,14634],{"class":469,"line":3811},[151,14635,1090],{"emptyLinePlaceholder":609},[151,14637,14638,14641],{"class":469,"line":3820},[151,14639,14640],{"class":14368},"retrievers",[151,14642,14372],{"class":503},[151,14644,14645,14648],{"class":469,"line":7084},[151,14646,14647],{"class":14368},"  milvus_retriever",[151,14649,14372],{"class":503},[151,14651,14652,14654,14656],{"class":469,"line":7148},[151,14653,14394],{"class":14368},[151,14655,6208],{"class":503},[151,14657,14658],{"class":481},"milvus_retriever\n",[151,14660,14661,14664,14666],{"class":469,"line":7211},[151,14662,14663],{"class":14368},"    uri",[151,14665,6208],{"class":503},[151,14667,14668],{"class":481},"\"http://localhost:19530\"\n",[151,14670,14671,14674,14676],{"class":469,"line":7273},[151,14672,14673],{"class":14368},"    embedding_model",[151,14675,6208],{"class":503},[151,14677,14678],{"class":481},"\"nv-embedqa-e5-v5\"\n",[151,14680,14681,14684,14686],{"class":469,"line":7335},[151,14682,14683],{"class":14368},"    collection_name",[151,14685,6208],{"class":503},[151,14687,14688],{"class":481},"\"aiq_case_documents\"\n",[151,14690,14691,14694,14696],{"class":469,"line":7398},[151,14692,14693],{"class":14368},"    vector_field",[151,14695,6208],{"class":503},[151,14697,14698],{"class":481},"\"embedding\"\n",[151,14700,14701,14704],{"class":469,"line":7462},[151,14702,14703],{"class":14368},"    search_params",[151,14705,14372],{"class":503},[151,14707,14708,14711,14713,14716],{"class":469,"line":7467},[151,14709,14710],{"class":14368},"      metric_type",[151,14712,6208],{"class":503},[151,14714,14715],{"class":481},"\"IP\"",[151,14717,14718],{"class":1527}," # works best with nv-embedqa-e5-v5\n",[151,14720,14721],{"class":469,"line":7532},[151,14722,1090],{"emptyLinePlaceholder":609},[151,14724,14725,14728],{"class":469,"line":7537},[151,14726,14727],{"class":14368},"llms",[151,14729,14372],{"class":503},[151,14731,14732,14735],{"class":469,"line":7603},[151,14733,14734],{"class":14368},"  nim_llm",[151,14736,14372],{"class":503},[151,14738,14739,14741,14743],{"class":469,"line":7608},[151,14740,14394],{"class":14368},[151,14742,6208],{"class":503},[151,14744,14745],{"class":481},"nim\n",[151,14747,14748,14751,14753],{"class":469,"line":7673},[151,14749,14750],{"class":14368},"    base_url",[151,14752,6208],{"class":503},[151,14754,14755],{"class":481},"http://192.168.5.96:1234/v1\n",[151,14757,14758,14761,14763],{"class":469,"line":7678},[151,14759,14760],{"class":14368},"    model_name",[151,14762,6208],{"class":503},[151,14764,14765],{"class":481},"qwen3-8b\n",[151,14767,14768,14771,14773],{"class":469,"line":7708},[151,14769,14770],{"class":14368},"    max_tokens",[151,14772,6208],{"class":503},[151,14774,14775],{"class":477},"10000\n",[151,14777,14778,14781,14783],{"class":469,"line":7713},[151,14779,14780],{"class":14368},"    temperature",[151,14782,6208],{"class":503},[151,14784,14785],{"class":477},"0.7\n",[151,14787,14788,14791],{"class":469,"line":7746},[151,14789,14790],{"class":14368},"  mediation_llm",[151,14792,14372],{"class":503},[151,14794,14795,14797,14799],{"class":469,"line":7751},[151,14796,14394],{"class":14368},[151,14798,6208],{"class":503},[151,14800,14745],{"class":481},[151,14802,14803,14805,14807],{"class":469,"line":7816},[151,14804,14750],{"class":14368},[151,14806,6208],{"class":503},[151,14808,14755],{"class":481},[151,14810,14811,14813,14815],{"class":469,"line":7821},[151,14812,14760],{"class":14368},[151,14814,6208],{"class":503},[151,14816,14765],{"class":481},[151,14818,14819,14821,14823],{"class":469,"line":7847},[151,14820,14770],{"class":14368},[151,14822,6208],{"class":503},[151,14824,14775],{"class":477},[151,14826,14827,14829,14831],{"class":469,"line":7852},[151,14828,14780],{"class":14368},[151,14830,6208],{"class":503},[151,14832,14785],{"class":477},[151,14834,14835],{"class":469,"line":7887},[151,14836,1090],{"emptyLinePlaceholder":609},[151,14838,14839,14842],{"class":469,"line":7892},[151,14840,14841],{"class":14368},"memory",[151,14843,14372],{"class":503},[151,14845,14846,14849],{"class":469,"line":7924},[151,14847,14848],{"class":14368},"  redis_memory",[151,14850,14372],{"class":503},[151,14852,14853,14855,14857],{"class":469,"line":7929},[151,14854,14394],{"class":14368},[151,14856,6208],{"class":503},[151,14858,14859],{"class":481},"redis_memory\n",[151,14861,14862,14865,14867],{"class":469,"line":7991},[151,14863,14864],{"class":14368},"    connection_url",[151,14866,6208],{"class":503},[151,14868,14869],{"class":481},"redis://localhost:6379/0\n",[151,14871,14872],{"class":469,"line":7996},[151,14873,1090],{"emptyLinePlaceholder":609},[151,14875,14876,14879],{"class":469,"line":8078},[151,14877,14878],{"class":14368},"functions",[151,14880,14372],{"class":503},[151,14882,14883,14886],{"class":469,"line":8140},[151,14884,14885],{"class":14368},"  case_document_rag",[151,14887,14372],{"class":503},[151,14889,14890,14892,14894],{"class":469,"line":8145},[151,14891,14394],{"class":14368},[151,14893,6208],{"class":503},[151,14895,14896],{"class":481},"case_document_rag\n",[151,14898,14899,14902,14904],{"class":469,"line":8259},[151,14900,14901],{"class":14368},"    retriever",[151,14903,6208],{"class":503},[151,14905,14658],{"class":481},[151,14907,14908,14911,14913],{"class":469,"line":8264},[151,14909,14910],{"class":14368},"    llm_name",[151,14912,6208],{"class":503},[151,14914,14915],{"class":481},"nim_llm\n",[151,14917,14918,14920,14922],{"class":469,"line":8613},[151,14919,14683],{"class":14368},[151,14921,6208],{"class":503},[151,14923,14924],{"class":481},"\"mediation_simulator_case_documents\"\n",[151,14926,14927,14930,14932],{"class":469,"line":8678},[151,14928,14929],{"class":14368},"    top_k",[151,14931,6208],{"class":503},[151,14933,14934],{"class":477},"5\n",[151,14936,14937,14940],{"class":469,"line":8742},[151,14938,14939],{"class":14368},"  case_query_agent",[151,14941,14372],{"class":503},[151,14943,14944,14946,14948],{"class":469,"line":8806},[151,14945,14394],{"class":14368},[151,14947,6208],{"class":503},[151,14949,14950],{"class":481},"case_query_agent\n",[151,14952,14953,14955,14957],{"class":469,"line":8870},[151,14954,14910],{"class":14368},[151,14956,6208],{"class":503},[151,14958,14915],{"class":481},[151,14960,14961,14964],{"class":469,"line":8875},[151,14962,14963],{"class":14368},"    tool_names",[151,14965,14372],{"class":503},[151,14967,14968,14970],{"class":469,"line":8881},[151,14969,14459],{"class":503},[151,14971,14896],{"class":481},[151,14973,14974,14977,14979],{"class":469,"line":8886},[151,14975,14976],{"class":14368},"    verbose",[151,14978,6208],{"class":503},[151,14980,14382],{"class":477},[151,14982,14983,14986,14988],{"class":469,"line":8892},[151,14984,14985],{"class":14368},"    max_iterations",[151,14987,6208],{"class":503},[151,14989,14934],{"class":477},[151,14991,14992],{"class":469,"line":8963},[151,14993,1090],{"emptyLinePlaceholder":609},[151,14995,14996],{"class":469,"line":8969},[151,14997,14998],{"class":1527},"  # server route functions\n",[151,15000,15002,15005],{"class":469,"line":15001},77,[151,15003,15004],{"class":14368},"  get_mediation_case",[151,15006,14372],{"class":503},[151,15008,15010,15012,15014],{"class":469,"line":15009},78,[151,15011,14394],{"class":14368},[151,15013,6208],{"class":503},[151,15015,15016],{"class":481},"server/get_mediation_case\n",[151,15018,15020,15023],{"class":469,"line":15019},79,[151,15021,15022],{"class":14368},"  get_mediation_session",[151,15024,14372],{"class":503},[151,15026,15028,15030,15032],{"class":469,"line":15027},80,[151,15029,14394],{"class":14368},[151,15031,6208],{"class":503},[151,15033,15034],{"class":481},"server/get_mediation_session\n",[151,15036,15038,15041],{"class":469,"line":15037},81,[151,15039,15040],{"class":14368},"  send_message_to_mediation_session",[151,15042,14372],{"class":503},[151,15044,15046,15048,15050],{"class":469,"line":15045},82,[151,15047,14394],{"class":14368},[151,15049,6208],{"class":503},[151,15051,15052],{"class":481},"mediation\n",[151,15054,15056],{"class":469,"line":15055},83,[151,15057,1090],{"emptyLinePlaceholder":609},[151,15059,15061,15064],{"class":469,"line":15060},84,[151,15062,15063],{"class":14368},"embedders",[151,15065,14372],{"class":503},[151,15067,15069,15072],{"class":469,"line":15068},85,[151,15070,15071],{"class":14368},"  nv-embedqa-e5-v5",[151,15073,14372],{"class":503},[151,15075,15077,15079,15081],{"class":469,"line":15076},86,[151,15078,14394],{"class":14368},[151,15080,6208],{"class":503},[151,15082,14745],{"class":481},[151,15084,15086,15088,15090],{"class":469,"line":15085},87,[151,15087,14750],{"class":14368},[151,15089,6208],{"class":503},[151,15091,15092],{"class":481},"http://192.168.5.96:8000/v1\n",[151,15094,15096,15098,15100],{"class":469,"line":15095},88,[151,15097,14760],{"class":14368},[151,15099,6208],{"class":503},[151,15101,15102],{"class":481},"nvidia/nv-embedqa-e5-v5\n",[151,15104,15106],{"class":469,"line":15105},89,[151,15107,1090],{"emptyLinePlaceholder":609},[151,15109,15111,15114],{"class":469,"line":15110},90,[151,15112,15113],{"class":14368},"workflow",[151,15115,14372],{"class":503},[151,15117,15119,15122,15124],{"class":469,"line":15118},91,[151,15120,15121],{"class":14368},"  _type",[151,15123,6208],{"class":503},[151,15125,15052],{"class":481},[151,15127,15129,15132,15134],{"class":469,"line":15128},92,[151,15130,15131],{"class":14368},"  llm",[151,15133,6208],{"class":503},[151,15135,15136],{"class":481},"mediation_llm\n",[151,15138,15140,15143,15145],{"class":469,"line":15139},93,[151,15141,15142],{"class":14368},"  data_dir",[151,15144,6208],{"class":503},[151,15146,15147],{"class":481},"./data\n",[11,15149,15150,15151,2176],{},"Let's start at the top with ",[30,15152,14369],{},[736,15154,15155],{"id":14369},[30,15156,14369],{},[11,15158,15159],{},"This section mainly defines the API routes for the FastAPI integration and the telemetry options I set up to view all of my programs traces. When building applications with LLMs, instrumenting for observability is key! Agent Intelligence Toolkit makes it really easy to hook up not just one observability tool, but really any number of observability tools! It all works through asynchronous calls, so it doesn't slow down the application.",[11,15161,15162],{},[2718,15163],{"alt":15164,"src":15165},"LLM observability","/static/mediation-simulator/Phoenix.png",[11,15167,15168],{},"Defining the API routes was pretty straightforward. You define an AIQ Toolkit function that handles the route's behavior. These routes are also automatically added to the API's documentation page using OpenAPI/Swagger:",[11,15170,15171],{},[2718,15172],{"alt":15173,"src":15174},"API documentation","/static/mediation-simulator/fastapi.png",[736,15176,15177],{"id":14640},[30,15178,14640],{},[11,15180,15181],{},"This section allows you to define different vector storage databases that your application uses. I used Milvus to store embeddings of case documents. Ultimately I wasn't able to incorporate these embeddings into my application.",[736,15183,15184],{"id":14727},[30,15185,14727],{},[11,15187,15188],{},"The LLMs section allows you to plug in to any LLM. I used a combination of LM Studio and NVIDIA NIMs to test my application. You can pretty much use any LLM that provides an OpenAI API interface. Local models have come a long way! New models like Qwen3 and Llama 3.1 have massive context windows (130k tokens!) which is a total game changer. These models are also getting a lot smarter. I was really impressed with how well these models followed my prompts. Using local models is nice because you will not be rate limited. I processed about 2 million prompt tokens and generated about 1.5 million completion tokens during the development of Mediation Simulator. As amazing as these new frontier models are now, I'm still bullish on the capabilities of (small) large language models that can run on consumer hardware like NVIDIA RTX GPUs.",[736,15190,15191],{"id":14841},[30,15192,14841],{},[11,15194,15195],{},"Figure out how memory works in the AIQ toolkit was a big \"ah-ha!\" moment for me. It allows for persisting chat messages between generations, and also storing arbitrary data that you can use in your workflows. I decided to use Redis (Redis Stack) to build a memory backend. Here's a quick look at what that code looks like:",[459,15197,15199],{"className":13136,"code":15198,"language":12886,"meta":464,"style":464},"@register_memory(config_type=RedisMemoryConfig)\nasync def redis_memory(config: RedisMemoryConfig, builder: Builder):\n\n    class RedisMemoryEditor(MemoryEditor):\n        def __init__(self, config: RedisMemoryConfig):\n            self._conn_url = config.connection_url\n            self.redis = Redis.from_url(self._conn_url)\n\n        async def get_client(self, session_id: str) -> RedisChatMessageHistory:\n            conn = RedisChatMessageHistory(\n                session_id=session_id, redis_url=self._conn_url\n            )\n            return conn\n\n        # mediation session state management\n        async def add_messages(\n            self, items: Sequence[BaseMessage], session_id: str\n        ) -> None:\n            client = await self.get_client(session_id)\n            await client.aadd_messages(items)\n\n        async def get_messages(self, session_id: str) -> Sequence[BaseMessage]:\n            client = await self.get_client(session_id)\n            messages = await client.aget_messages()\n            return messages\n\n        # case generation state management\n        async def save_case_description(\n            self, case_description: str, case_id: str\n        ) -> None:\n            \"\"\"\n            sets the case description using the \u003Ccase_id>_case_description as the redis key\n            \"\"\"\n            self.redis.set(f\"{case_id}_case_description\", case_description)\n\n        ...\n",[30,15200,15201,15217,15243,15247,15265,15285,15299,15316,15320,15347,15357,15377,15382,15390,15394,15399,15411,15430,15440,15455,15463,15467,15491,15503,15515,15522,15526,15531,15542,15564,15572,15577,15582,15586,15609,15613],{"__ignoreMap":464},[151,15202,15203,15206,15208,15212,15214],{"class":469,"line":470},[151,15204,15205],{"class":473},"@register_memory",[151,15207,12386],{"class":503},[151,15209,15211],{"class":15210},"sTHNf","config_type",[151,15213,1876],{"class":1869},[151,15215,15216],{"class":503},"RedisMemoryConfig)\n",[151,15218,15219,15222,15225,15228,15230,15234,15237,15240],{"class":469,"line":488},[151,15220,15221],{"class":12347},"async",[151,15223,15224],{"class":12347}," def",[151,15226,15227],{"class":473}," redis_memory",[151,15229,12386],{"class":503},[151,15231,15233],{"class":15232},"so59x","config",[151,15235,15236],{"class":503},": RedisMemoryConfig, ",[151,15238,15239],{"class":15232},"builder",[151,15241,15242],{"class":503},": Builder):\n",[151,15244,15245],{"class":469,"line":500},[151,15246,1090],{"emptyLinePlaceholder":609},[151,15248,15249,15252,15256,15258,15262],{"class":469,"line":509},[151,15250,15251],{"class":12347},"    class",[151,15253,15255],{"class":15254},"sz2Vg"," RedisMemoryEditor",[151,15257,12386],{"class":503},[151,15259,15261],{"class":15260},"s30JN","MemoryEditor",[151,15263,15264],{"class":503},"):\n",[151,15266,15267,15270,15273,15275,15278,15280,15282],{"class":469,"line":517},[151,15268,15269],{"class":12347},"        def",[151,15271,15272],{"class":2226}," __init__",[151,15274,12386],{"class":503},[151,15276,15277],{"class":15232},"self",[151,15279,106],{"class":503},[151,15281,15233],{"class":15232},[151,15283,15284],{"class":503},": RedisMemoryConfig):\n",[151,15286,15287,15291,15294,15296],{"class":469,"line":534},[151,15288,15290],{"class":15289},"sP7S_","            self",[151,15292,15293],{"class":503},"._conn_url ",[151,15295,1876],{"class":1869},[151,15297,15298],{"class":503}," config.connection_url\n",[151,15300,15301,15303,15306,15308,15311,15313],{"class":469,"line":1413},[151,15302,15290],{"class":15289},[151,15304,15305],{"class":503},".redis ",[151,15307,1876],{"class":1869},[151,15309,15310],{"class":503}," Redis.from_url(",[151,15312,15277],{"class":15289},[151,15314,15315],{"class":503},"._conn_url)\n",[151,15317,15318],{"class":469,"line":1418},[151,15319,1090],{"emptyLinePlaceholder":609},[151,15321,15322,15325,15327,15330,15332,15334,15336,15339,15341,15344],{"class":469,"line":2462},[151,15323,15324],{"class":12347},"        async",[151,15326,15224],{"class":12347},[151,15328,15329],{"class":473}," get_client",[151,15331,12386],{"class":503},[151,15333,15277],{"class":15232},[151,15335,106],{"class":503},[151,15337,15338],{"class":15232},"session_id",[151,15340,6208],{"class":503},[151,15342,15343],{"class":6205},"str",[151,15345,15346],{"class":503},") -> RedisChatMessageHistory:\n",[151,15348,15349,15352,15354],{"class":469,"line":2471},[151,15350,15351],{"class":503},"            conn ",[151,15353,1876],{"class":1869},[151,15355,15356],{"class":503}," RedisChatMessageHistory(\n",[151,15358,15359,15362,15364,15367,15370,15372,15374],{"class":469,"line":2480},[151,15360,15361],{"class":15210},"                session_id",[151,15363,1876],{"class":1869},[151,15365,15366],{"class":503},"session_id, ",[151,15368,15369],{"class":15210},"redis_url",[151,15371,1876],{"class":1869},[151,15373,15277],{"class":15289},[151,15375,15376],{"class":503},"._conn_url\n",[151,15378,15379],{"class":469,"line":2489},[151,15380,15381],{"class":503},"            )\n",[151,15383,15384,15387],{"class":469,"line":2497},[151,15385,15386],{"class":1869},"            return",[151,15388,15389],{"class":503}," conn\n",[151,15391,15392],{"class":469,"line":3140},[151,15393,1090],{"emptyLinePlaceholder":609},[151,15395,15396],{"class":469,"line":3149},[151,15397,15398],{"class":1527},"        # mediation session state management\n",[151,15400,15401,15403,15405,15408],{"class":469,"line":3158},[151,15402,15324],{"class":12347},[151,15404,15224],{"class":12347},[151,15406,15407],{"class":473}," add_messages",[151,15409,15410],{"class":503},"(\n",[151,15412,15413,15415,15417,15420,15423,15425,15427],{"class":469,"line":3167},[151,15414,15290],{"class":15232},[151,15416,106],{"class":503},[151,15418,15419],{"class":15232},"items",[151,15421,15422],{"class":503},": Sequence[BaseMessage], ",[151,15424,15338],{"class":15232},[151,15426,6208],{"class":503},[151,15428,15429],{"class":6205},"str\n",[151,15431,15432,15435,15438],{"class":469,"line":3175},[151,15433,15434],{"class":503},"        ) -> ",[151,15436,15437],{"class":477},"None",[151,15439,14372],{"class":503},[151,15441,15442,15445,15447,15449,15452],{"class":469,"line":3184},[151,15443,15444],{"class":503},"            client ",[151,15446,1876],{"class":1869},[151,15448,12369],{"class":1869},[151,15450,15451],{"class":15289}," self",[151,15453,15454],{"class":503},".get_client(session_id)\n",[151,15456,15457,15460],{"class":469,"line":3193},[151,15458,15459],{"class":1869},"            await",[151,15461,15462],{"class":503}," client.aadd_messages(items)\n",[151,15464,15465],{"class":469,"line":3720},[151,15466,1090],{"emptyLinePlaceholder":609},[151,15468,15469,15471,15473,15476,15478,15480,15482,15484,15486,15488],{"class":469,"line":3729},[151,15470,15324],{"class":12347},[151,15472,15224],{"class":12347},[151,15474,15475],{"class":473}," get_messages",[151,15477,12386],{"class":503},[151,15479,15277],{"class":15232},[151,15481,106],{"class":503},[151,15483,15338],{"class":15232},[151,15485,6208],{"class":503},[151,15487,15343],{"class":6205},[151,15489,15490],{"class":503},") -> Sequence[BaseMessage]:\n",[151,15492,15493,15495,15497,15499,15501],{"class":469,"line":3735},[151,15494,15444],{"class":503},[151,15496,1876],{"class":1869},[151,15498,12369],{"class":1869},[151,15500,15451],{"class":15289},[151,15502,15454],{"class":503},[151,15504,15505,15508,15510,15512],{"class":469,"line":3745},[151,15506,15507],{"class":503},"            messages ",[151,15509,1876],{"class":1869},[151,15511,12369],{"class":1869},[151,15513,15514],{"class":503}," client.aget_messages()\n",[151,15516,15517,15519],{"class":469,"line":3754},[151,15518,15386],{"class":1869},[151,15520,15521],{"class":503}," messages\n",[151,15523,15524],{"class":469,"line":3760},[151,15525,1090],{"emptyLinePlaceholder":609},[151,15527,15528],{"class":469,"line":3773},[151,15529,15530],{"class":1527},"        # case generation state management\n",[151,15532,15533,15535,15537,15540],{"class":469,"line":3782},[151,15534,15324],{"class":12347},[151,15536,15224],{"class":12347},[151,15538,15539],{"class":473}," save_case_description",[151,15541,15410],{"class":503},[151,15543,15544,15546,15548,15551,15553,15555,15557,15560,15562],{"class":469,"line":3791},[151,15545,15290],{"class":15232},[151,15547,106],{"class":503},[151,15549,15550],{"class":15232},"case_description",[151,15552,6208],{"class":503},[151,15554,15343],{"class":6205},[151,15556,106],{"class":503},[151,15558,15559],{"class":15232},"case_id",[151,15561,6208],{"class":503},[151,15563,15429],{"class":6205},[151,15565,15566,15568,15570],{"class":469,"line":3803},[151,15567,15434],{"class":503},[151,15569,15437],{"class":477},[151,15571,14372],{"class":503},[151,15573,15574],{"class":469,"line":3811},[151,15575,15576],{"class":481},"            \"\"\"\n",[151,15578,15579],{"class":469,"line":3820},[151,15580,15581],{"class":481},"            sets the case description using the \u003Ccase_id>_case_description as the redis key\n",[151,15583,15584],{"class":469,"line":7084},[151,15585,15576],{"class":481},[151,15587,15588,15590,15593,15595,15597,15599,15601,15603,15606],{"class":469,"line":7148},[151,15589,15290],{"class":15289},[151,15591,15592],{"class":503},".redis.set(",[151,15594,13214],{"class":12347},[151,15596,8592],{"class":481},[151,15598,5729],{"class":477},[151,15600,15559],{"class":503},[151,15602,2001],{"class":477},[151,15604,15605],{"class":481},"_case_description\"",[151,15607,15608],{"class":503},", case_description)\n",[151,15610,15611],{"class":469,"line":7211},[151,15612,1090],{"emptyLinePlaceholder":609},[151,15614,15615],{"class":469,"line":7273},[151,15616,15617],{"class":477},"        ...\n",[11,15619,15620,15621,15624],{},"I found that LangChain has a ",[30,15622,15623],{},"RedisChatMessageHistory"," class that made putting this backend together almost trivial. Redis Stack also ships with a web viewer which really came in handy for debugging my memory backend:",[11,15626,15627],{},[2718,15628],{"alt":15629,"src":15630},"Redis Memory Backend","/static/mediation-simulator/redis.png",[11,15632,15633],{},"For storing other types of data, I was able to implement my own methods and store things like case data or other metadata for a mediation simulator session for things like current_speaker, number of session, current session, etc. I love Redis! The setup is also really easy, I just added a docker compose file:",[459,15635,15637],{"className":14359,"code":15636,"language":14361,"meta":464,"style":464},"services:\n  redis:\n    image: redis/redis-stack:latest\n    volumes:\n      - redis-data:/data\n    container_name: redis\n    ports:\n      - 6379:6379\n      - 8001:8001  # RedisInsight port\n\nvolumes:\n  redis-data:\n",[30,15638,15639,15646,15653,15663,15670,15677,15687,15694,15701,15711,15715,15722],{"__ignoreMap":464},[151,15640,15641,15644],{"class":469,"line":470},[151,15642,15643],{"class":14368},"services",[151,15645,14372],{"class":503},[151,15647,15648,15651],{"class":469,"line":488},[151,15649,15650],{"class":14368},"  redis",[151,15652,14372],{"class":503},[151,15654,15655,15658,15660],{"class":469,"line":500},[151,15656,15657],{"class":14368},"    image",[151,15659,6208],{"class":503},[151,15661,15662],{"class":481},"redis/redis-stack:latest\n",[151,15664,15665,15668],{"class":469,"line":509},[151,15666,15667],{"class":14368},"    volumes",[151,15669,14372],{"class":503},[151,15671,15672,15674],{"class":469,"line":517},[151,15673,14459],{"class":503},[151,15675,15676],{"class":481},"redis-data:/data\n",[151,15678,15679,15682,15684],{"class":469,"line":534},[151,15680,15681],{"class":14368},"    container_name",[151,15683,6208],{"class":503},[151,15685,15686],{"class":481},"redis\n",[151,15688,15689,15692],{"class":469,"line":1413},[151,15690,15691],{"class":14368},"    ports",[151,15693,14372],{"class":503},[151,15695,15696,15698],{"class":469,"line":1418},[151,15697,14459],{"class":503},[151,15699,15700],{"class":481},"6379:6379\n",[151,15702,15703,15705,15708],{"class":469,"line":2462},[151,15704,14459],{"class":503},[151,15706,15707],{"class":481},"8001:8001",[151,15709,15710],{"class":1527},"  # RedisInsight port\n",[151,15712,15713],{"class":469,"line":2471},[151,15714,1090],{"emptyLinePlaceholder":609},[151,15716,15717,15720],{"class":469,"line":2480},[151,15718,15719],{"class":14368},"volumes",[151,15721,14372],{"class":503},[151,15723,15724,15727],{"class":469,"line":2489},[151,15725,15726],{"class":14368},"  redis-data",[151,15728,14372],{"class":503},[736,15730,15731],{"id":14878},[30,15732,14878],{},[11,15734,15735],{},"Functions are the building blocks of the AIQ Toolkit. You need to register the functions in your config file, then you can use them for different things, like the function that handles an API route, or the function that handles an agentic workflow. I defined some functions for RAG to allow my agents to look up case data, but I wasn't able to fully implement this in my main mediation simulator workflow. But the setup was easy!",[736,15737,15738],{"id":15063},[30,15739,15063],{},[11,15741,15742],{},"Embedders is a section that allows you to define embedding models that you would use together with RAG (for converting text to a vector embedding). Since I needed to make a lot of embeddings for all of the documents I generated for case facts, I used a locally hosted NVIDIA NIM:",[459,15744,15746],{"className":14359,"code":15745,"language":14361,"meta":464,"style":464},"embedders:\n  nv-embedqa-e5-v5:\n    _type: nim\n    base_url: http://192.168.5.96:8000/v1\n    model_name: nvidia/nv-embedqa-e5-v5\n",[30,15747,15748,15754,15760,15768,15776],{"__ignoreMap":464},[151,15749,15750,15752],{"class":469,"line":470},[151,15751,15063],{"class":14368},[151,15753,14372],{"class":503},[151,15755,15756,15758],{"class":469,"line":488},[151,15757,15071],{"class":14368},[151,15759,14372],{"class":503},[151,15761,15762,15764,15766],{"class":469,"line":500},[151,15763,14394],{"class":14368},[151,15765,6208],{"class":503},[151,15767,14745],{"class":481},[151,15769,15770,15772,15774],{"class":469,"line":509},[151,15771,14750],{"class":14368},[151,15773,6208],{"class":503},[151,15775,15092],{"class":481},[151,15777,15778,15780,15782],{"class":469,"line":517},[151,15779,14760],{"class":14368},[151,15781,6208],{"class":503},[151,15783,15102],{"class":481},[11,15785,15786],{},"I would have run into rate limits if I was using the hosted version, so being able to run this locally was important for my use case.",[736,15788,15789],{"id":15113},[30,15790,15113],{},[11,15792,15793],{},"The workflow is the main \"application\" part of the config file. It is the entrypoint for your application. In my case, the workflow invokes a LangGraph that does my simulation. First it loads data from my memory backend and when I'm using the interactive mode it gathers information from the request like path parameters so it knows what data fetch from memory (like the case id and session id).",[11,15795,15796],{},"My mediation workflow code is a little bit messy. I tried to keep all of my prompting logic in separate files for simplicity. The trickiest part for me was serializing data between different formats: langgraph state, YAML files and Redis memory. I'm happy to have something now that is functional, but there are a lot of improvements and further refactoring that would make the code easier to read and maintain.",[11,15798,15799],{},"That wraps up the tour of my main config file for mediation simulator! I also had another smaller config file for case generation. Here's a quick look at that:",[459,15801,15803],{"className":14359,"code":15802,"language":14361,"meta":464,"style":464},"general:\n  use_uvloop: true\n  telemetry:\n    enabled: false\n    tracing:\n      phoenix:\n        _type: phoenix\n        endpoint: http://localhost:6006/v1/traces\n        project: mediation-simulator\n\nllms:\n  nim_llm:\n    _type: nim\n    base_url: http://192.168.5.96:1234/v1\n    model_name: qwen3-8b\n    max_tokens: 10000\n    temperature: 0.7\n  # nim_llm:\n  #   _type: nim\n  #   model_name: meta/llama-3.1-70b-instruct\n  #   max_tokens: 10000\n  #   temperature: 0.7\n\nmemory:\n  redis_memory:\n    _type: redis_memory\n    connection_url: redis://localhost:6379/0\n\nworkflow:\n  _type: case_generation\n  llm_name: nim_llm\n  data_dir: ./data\n",[30,15804,15805,15811,15819,15825,15833,15839,15845,15853,15861,15870,15874,15880,15886,15894,15902,15910,15918,15926,15931,15936,15941,15946,15951,15955,15961,15967,15975,15983,15987,15993,16002,16011],{"__ignoreMap":464},[151,15806,15807,15809],{"class":469,"line":470},[151,15808,14369],{"class":14368},[151,15810,14372],{"class":503},[151,15812,15813,15815,15817],{"class":469,"line":488},[151,15814,14377],{"class":14368},[151,15816,6208],{"class":503},[151,15818,14382],{"class":477},[151,15820,15821,15823],{"class":469,"line":500},[151,15822,14575],{"class":14368},[151,15824,14372],{"class":503},[151,15826,15827,15829,15831],{"class":469,"line":509},[151,15828,14582],{"class":14368},[151,15830,6208],{"class":503},[151,15832,14587],{"class":477},[151,15834,15835,15837],{"class":469,"line":517},[151,15836,14592],{"class":14368},[151,15838,14372],{"class":503},[151,15840,15841,15843],{"class":469,"line":534},[151,15842,14599],{"class":14368},[151,15844,14372],{"class":503},[151,15846,15847,15849,15851],{"class":469,"line":1413},[151,15848,14606],{"class":14368},[151,15850,6208],{"class":503},[151,15852,14611],{"class":481},[151,15854,15855,15857,15859],{"class":469,"line":1418},[151,15856,14616],{"class":14368},[151,15858,6208],{"class":503},[151,15860,14621],{"class":481},[151,15862,15863,15865,15867],{"class":469,"line":2462},[151,15864,14626],{"class":14368},[151,15866,6208],{"class":503},[151,15868,15869],{"class":481},"mediation-simulator\n",[151,15871,15872],{"class":469,"line":2471},[151,15873,1090],{"emptyLinePlaceholder":609},[151,15875,15876,15878],{"class":469,"line":2480},[151,15877,14727],{"class":14368},[151,15879,14372],{"class":503},[151,15881,15882,15884],{"class":469,"line":2489},[151,15883,14734],{"class":14368},[151,15885,14372],{"class":503},[151,15887,15888,15890,15892],{"class":469,"line":2497},[151,15889,14394],{"class":14368},[151,15891,6208],{"class":503},[151,15893,14745],{"class":481},[151,15895,15896,15898,15900],{"class":469,"line":3140},[151,15897,14750],{"class":14368},[151,15899,6208],{"class":503},[151,15901,14755],{"class":481},[151,15903,15904,15906,15908],{"class":469,"line":3149},[151,15905,14760],{"class":14368},[151,15907,6208],{"class":503},[151,15909,14765],{"class":481},[151,15911,15912,15914,15916],{"class":469,"line":3158},[151,15913,14770],{"class":14368},[151,15915,6208],{"class":503},[151,15917,14775],{"class":477},[151,15919,15920,15922,15924],{"class":469,"line":3167},[151,15921,14780],{"class":14368},[151,15923,6208],{"class":503},[151,15925,14785],{"class":477},[151,15927,15928],{"class":469,"line":3175},[151,15929,15930],{"class":1527},"  # nim_llm:\n",[151,15932,15933],{"class":469,"line":3184},[151,15934,15935],{"class":1527},"  #   _type: nim\n",[151,15937,15938],{"class":469,"line":3193},[151,15939,15940],{"class":1527},"  #   model_name: meta/llama-3.1-70b-instruct\n",[151,15942,15943],{"class":469,"line":3720},[151,15944,15945],{"class":1527},"  #   max_tokens: 10000\n",[151,15947,15948],{"class":469,"line":3729},[151,15949,15950],{"class":1527},"  #   temperature: 0.7\n",[151,15952,15953],{"class":469,"line":3735},[151,15954,1090],{"emptyLinePlaceholder":609},[151,15956,15957,15959],{"class":469,"line":3745},[151,15958,14841],{"class":14368},[151,15960,14372],{"class":503},[151,15962,15963,15965],{"class":469,"line":3754},[151,15964,14848],{"class":14368},[151,15966,14372],{"class":503},[151,15968,15969,15971,15973],{"class":469,"line":3760},[151,15970,14394],{"class":14368},[151,15972,6208],{"class":503},[151,15974,14859],{"class":481},[151,15976,15977,15979,15981],{"class":469,"line":3773},[151,15978,14864],{"class":14368},[151,15980,6208],{"class":503},[151,15982,14869],{"class":481},[151,15984,15985],{"class":469,"line":3782},[151,15986,1090],{"emptyLinePlaceholder":609},[151,15988,15989,15991],{"class":469,"line":3791},[151,15990,15113],{"class":14368},[151,15992,14372],{"class":503},[151,15994,15995,15997,15999],{"class":469,"line":3803},[151,15996,15121],{"class":14368},[151,15998,6208],{"class":503},[151,16000,16001],{"class":481},"case_generation\n",[151,16003,16004,16007,16009],{"class":469,"line":3811},[151,16005,16006],{"class":14368},"  llm_name",[151,16008,6208],{"class":503},[151,16010,14915],{"class":481},[151,16012,16013,16015,16017],{"class":469,"line":3820},[151,16014,15142],{"class":14368},[151,16016,6208],{"class":503},[151,16018,15147],{"class":481},[11,16020,16021],{},"I added a lot of logging to my workflow in order to keep an eye on how the workflows progressed. Here's a sample of the logs from the CLI invocation of the mediation simulator program:",[459,16023,16026],{"className":16024,"code":16025,"language":997},[995],"12:08:48 mediation.register INFO   ⚖️ [MEDIATOR]: Mediator node called\n12:08:56 mediation.register INFO   👤 [CLERK] Starting clerk node - Phase: JOINT_DISCUSSION_INFO_GATHERING, Turn: 3\n12:08:56 mediation.register INFO   🤔 [CLERK] Using LLM to decide next speaker\n12:09:12 mediation.register INFO   🎯 [CLERK] LLM selected next speaker: RESPONDING_PARTY\n12:09:12 mediation.register INFO   📊 [CLERK] Updated counters - Turn: 4, Phase turns: 4\n12:09:12 mediation.register INFO   🌚 Responding party node called\n12:09:20 mediation.register INFO   👤 [CLERK] Starting clerk node - Phase: JOINT_DISCUSSION_INFO_GATHERING, Turn: 4\n12:09:20 mediation.register INFO   🤔 [CLERK] Using LLM to decide next speaker\n12:09:23 mediation.register INFO   🎯 [CLERK] LLM selected next speaker: REQUESTING_PARTY\n12:09:23 mediation.register INFO   📊 [CLERK] Updated counters - Turn: 5, Phase turns: 5\n12:09:23 mediation.register INFO   🌝 Requesting party node called\n12:09:31 mediation.register INFO   👤 [CLERK] Starting clerk node - Phase: JOINT_DISCUSSION_INFO_GATHERING, Turn: 5\n12:09:31 mediation.register INFO   ⏰ [CLERK] Max turns (5) reached for current phase\n12:09:31 mediation.register INFO   🔄 [CLERK] Transitioning from joint discussion to negotiation\n12:09:31 mediation.register INFO   👨‍⚖️ [CLERK] Mediator will start the new phase\n12:09:31 mediation.register INFO   ⚖️ [MEDIATOR]: Mediator node called\n12:09:40 mediation.register INFO   👤 [CLERK] Starting clerk node - Phase: NEGOTIATION_BARGAINING, Turn: 5\n12:09:40 mediation.register INFO   🤔 [CLERK] Using LLM to decide next speaker\n12:09:48 mediation.register INFO   🎯 [CLERK] LLM selected next speaker: REQUESTING_PARTY\n12:09:48 mediation.register INFO   📊 [CLERK] Updated counters - Turn: 6, Phase turns: 1\n12:09:48 mediation.register INFO   🌝 Requesting party node called\n12:09:56 mediation.register INFO   👤 [CLERK] Starting clerk node - Phase: NEGOTIATION_BARGAINING, Turn: 6\n12:09:56 mediation.register INFO   🤔 [CLERK] Using LLM to decide next speaker\n12:09:58 mediation.register INFO   🎯 [CLERK] LLM selected next speaker: RESPONDING_PARTY\n12:09:58 mediation.register INFO   📊 [CLERK] Updated counters - Turn: 7, Phase turns: 2\n",[30,16027,16025],{"__ignoreMap":464},[11,16029,16030],{},"Config files can be as simple or as complex as they need to be depending on your workflow.",[11,16032,16033],{},"The AgentIQ Toolkit is doing something really valuable by bringing patterns and components from these (and other) frameworks into a cohesive system. This allows for some truly interesting and powerful combinations. The examples provided are excellent and taught me a lot about new patterns for building agentic workflows. ReWOO agents, for instance, were a new concept for me, as was seeing how to effectively combine LangGraph with LlamaIndex.",[11,16035,16036],{},"What I particularly appreciated was:",[76,16038,16039,16045,16051],{},[79,16040,16041,16044],{},[15,16042,16043],{},"Standardized Patterns:"," AgentIQ promotes good practices for crucial aspects of AI development, like evaluations and telemetry/tracing.",[79,16046,16047,16050],{},[15,16048,16049],{},"YAML Configuration:"," I really like using YAML for configuring development environments, similar to Docker Compose. It standardizes things and vastly improves readability. The way AgentIQ allows registering functions that can be included in workflows via YAML config files is a great example of this.",[79,16052,16053,16056],{},[15,16054,16055],{},"A Learning Goldmine:"," Working through as many examples as possible was incredibly beneficial. Reading the code helped me grasp the patterns underpinning AgentIQ. You are pretty much guaranteed to learn something new!",[11,16058,16059],{},"I'm so glad I took the plunge and got my feet wet with the AgentIQ Toolkit. It's been a fantastic learning resource.",[736,16061,16063],{"id":16062},"mediation-competitions-but-for-llms","Mediation Competitions, But for LLMs?",[11,16065,16066],{},"One of the fun, forward-looking ideas this project sparks is the concept of \"mediation competitions, but for LLMs.\" Imagine pitting different LLMs against each other, representing the requesting and responding parties, to see how they fare in these complex negotiation scenarios!",[736,16068,16070],{"id":16069},"whats-next","What's Next?",[11,16072,16073],{},"This hackathon project has been an incredible learning journey. Building Mediation Simulator has not only been a fun technical challenge but has also opened my eyes to the potential of AI agents in simulating complex human interactions. I'm excited to continue refining it and exploring more possibilities with the AgentIQ Toolkit!",[589,16075,16076],{},"html pre.shiki code .s5clZ, html code.shiki .s5clZ{--shiki-default:#22863A;--shiki-dark:#85E89D;--shiki-sepia:#F92672}html pre.shiki code .sMOD_, html code.shiki .sMOD_{--shiki-default:#24292E;--shiki-dark:#E1E4E8;--shiki-sepia:#F8F8F2}html pre.shiki code .s7F3e, html code.shiki .s7F3e{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#AE81FF}html pre.shiki code .sstjo, html code.shiki .sstjo{--shiki-default:#032F62;--shiki-dark:#9ECBFF;--shiki-sepia:#E6DB74}html pre.shiki code .s8-w5, html code.shiki .s8-w5{--shiki-default:#6A737D;--shiki-dark:#6A737D;--shiki-sepia:#88846F}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html .sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html.sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html pre.shiki code .srTi1, html code.shiki .srTi1{--shiki-default:#6F42C1;--shiki-dark:#B392F0;--shiki-sepia:#A6E22E}html pre.shiki code .sTHNf, html code.shiki .sTHNf{--shiki-default:#E36209;--shiki-default-font-style:inherit;--shiki-dark:#FFAB70;--shiki-dark-font-style:inherit;--shiki-sepia:#FD971F;--shiki-sepia-font-style:italic}html pre.shiki code .sC2Qs, html code.shiki .sC2Qs{--shiki-default:#D73A49;--shiki-dark:#F97583;--shiki-sepia:#F92672}html pre.shiki code .sq6CD, html code.shiki .sq6CD{--shiki-default:#D73A49;--shiki-default-font-style:inherit;--shiki-dark:#F97583;--shiki-dark-font-style:inherit;--shiki-sepia:#66D9EF;--shiki-sepia-font-style:italic}html pre.shiki code .so59x, html code.shiki .so59x{--shiki-default:#24292E;--shiki-default-font-style:inherit;--shiki-dark:#E1E4E8;--shiki-dark-font-style:inherit;--shiki-sepia:#FD971F;--shiki-sepia-font-style:italic}html pre.shiki code .sz2Vg, html code.shiki .sz2Vg{--shiki-default:#6F42C1;--shiki-default-text-decoration:inherit;--shiki-dark:#B392F0;--shiki-dark-text-decoration:inherit;--shiki-sepia:#A6E22E;--shiki-sepia-text-decoration:underline}html pre.shiki code .s30JN, html code.shiki .s30JN{--shiki-default:#6F42C1;--shiki-default-font-style:inherit;--shiki-default-text-decoration:inherit;--shiki-dark:#B392F0;--shiki-dark-font-style:inherit;--shiki-dark-text-decoration:inherit;--shiki-sepia:#A6E22E;--shiki-sepia-font-style:italic;--shiki-sepia-text-decoration:underline}html pre.shiki code .sTrkL, html code.shiki .sTrkL{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#66D9EF}html pre.shiki code .sP7S_, html code.shiki .sP7S_{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#FD971F}html pre.shiki code .s-m8C, html code.shiki .s-m8C{--shiki-default:#005CC5;--shiki-default-font-style:inherit;--shiki-dark:#79B8FF;--shiki-dark-font-style:inherit;--shiki-sepia:#66D9EF;--shiki-sepia-font-style:italic}",{"title":464,"searchDepth":488,"depth":488,"links":16078},[16079,16085],{"id":14093,"depth":488,"text":14094,"children":16080},[16081,16082,16083,16084],{"id":14123,"depth":500,"text":14124},{"id":14183,"depth":500,"text":14184},{"id":14236,"depth":500,"text":14237},{"id":14327,"depth":500,"text":14328},{"id":14349,"depth":488,"text":14350,"children":16086},[16087,16088,16089,16090,16091,16092,16093,16094,16095],{"id":14369,"depth":500,"text":14369},{"id":14640,"depth":500,"text":14640},{"id":14727,"depth":500,"text":14727},{"id":14841,"depth":500,"text":14841},{"id":14878,"depth":500,"text":14878},{"id":15063,"depth":500,"text":15063},{"id":15113,"depth":500,"text":15113},{"id":16062,"depth":500,"text":16063},{"id":16069,"depth":500,"text":16070},"2025-05-27","Mediation Simulator is an AI application designed to help legal students and professionals build dispute resolution skills through simulated mediation sessions",[16099],{"link":14084,"site":11126},"/static/mediation-simulator/mediation_simulator_title_image.png",{},"/2025/05/27/mediation-simulator-project-for-nvidia-agent-intelligence-toolkit",{"title":14059,"description":16097},"2025/05/27/mediation-simulator-project-for-nvidia-agent-intelligence-toolkit",[11133,614,1822,615,5404,16106],"mediation","Ik32HJu8f6DN54uoOQPNoW5vjlBJcyithwC6tPOaySU",{"id":16109,"title":16110,"body":16111,"comments":609,"date":19390,"description":19391,"draft":602,"extension":605,"external":19392,"image":16157,"meta":19396,"navigation":609,"path":19397,"seo":19398,"stem":19399,"tags":19400,"__hash__":19406},"blog/2024/10/09/redlm-ai-application-for-studying-chinese-literature-redology-nvidia-llama-index-developer-contest.md","RedLM: My submission for the NVIDIA and LlamaIndex Developer Contest",{"type":8,"value":16112,"toc":19352},[16113,16117,16120,16123,16127,16130,16145,16149,16152,16158,16161,16172,16176,16179,16184,16192,16195,16199,16202,16205,16212,16216,16219,16225,16228,16231,16237,16250,16256,16259,16356,16359,16425,16428,16445,16462,16466,16469,16475,16482,16487,16490,16496,16503,16510,16837,16841,16844,17299,17306,17310,17313,17319,17330,17336,17345,17349,17352,17505,17515,17673,17679,17682,17692,17696,17699,17708,17722,17728,17731,17734,17737,17743,17746,17773,17783,17800,17803,17832,17835,17838,17843,17846,17851,17854,17858,17861,17864,17870,17873,17898,17904,17907,17910,17916,17919,17951,17954,18126,18129,18132,18135,18138,18366,18369,18373,18376,18379,18389,18395,18401,18410,18415,18420,18423,18428,18433,18437,18440,18517,18523,18532,18548,18802,18808,18812,18818,18835,18841,18847,18850,18853,18859,18862,18867,18870,18876,18880,18886,18891,18900,18906,18909,18913,18916,18922,18925,18930,18933,18937,18943,18956,18959,18968,18974,18981,18987,18993,18998,19001,19005,19008,19016,19035,19044,19050,19053,19059,19062,19067,19070,19079,19090,19095,19105,19115,19122,19127,19131,19134,19137,19143,19146,19150,19156,19159,19162,19181,19184,19190,19193,19199,19202,19206,19209,19214,19217,19223,19244,19247,19250,19256,19259,19262,19274,19280,19288,19294,19297,19303,19312,19317,19320,19326,19329,19332,19338,19344,19349],[56,16114,16116],{"id":16115},"tldr","tl;dr",[11,16118,16119],{},"RedLM is a new way to study art and literature powered by artificial intelligence. It is an application that applies LLMs to the study of one of China's most famous literary works: Dream of the Red Chamber. It uses leading language and vision models from Chinese AI groups including Alibaba's Qwen, Baichuan Intelligence Technology and 01.AI. RedLM uses tools, techniques and services from NVIDIA and LlamaIndex including NVIDIA NIMs, Retrieval Augmented Generation and Multi-Modal RAG with vision language models. This project is my submission for the NVIDIA and LlamaIndex Developer Contest.",[11,16121,16122],{},"This article will cover how I built the project, challenges I faced and some of the lessons I learned while working with NVIDIA and LlamaIndex technologies.",[736,16124,16126],{"id":16125},"links","Links",[16128,16129],"red-lm-tweet",{},[76,16131,16132,16138],{},[79,16133,16134],{},[20,16135,14086],{"href":16136,"rel":16137},"https://x.com/briancaffey/status/1855186768452321330",[24],[79,16139,16140],{},[20,16141,16144],{"href":16142,"rel":16143},"https://github.com/briancaffey/RedLM",[24],"RedLM GitHub repository",[56,16146,16148],{"id":16147},"what-is-redlm","What is RedLM?",[11,16150,16151],{},"RedLM is a combination of the word \"Red\" and LM, an abbreviation for \"language model\". Dream of the Red Chamber is such an important book in Chinese literature that it has its own field of study called 红学 (literally \"the study of red\"), or Redology. So, RedLM is an application that uses language models for the study of Redology.",[11,16153,16154],{},[2718,16155],{"alt":16156,"src":16157},"RedLM","/static/redlm/title.png",[11,16159,16160],{},"In this project I focused on three applications of language models:",[700,16162,16163,16166,16169],{},[79,16164,16165],{},"Summary and translation of the source text",[79,16167,16168],{},"A Q&A bot that can answer questions about the book providing references to the specific paragraphs used to give answers",[79,16170,16171],{},"An image-based Q&A bot that can answer questions about sections of paintings that depict scenes from each of the book's chapters.",[56,16173,16175],{"id":16174},"notebooklm","NotebookLM",[11,16177,16178],{},"I used this article to create a \"Deep Dive\" podcast episode for RedLM using Google's NotebookLM.",[11,16180,16181],{},[2718,16182],{"alt":16175,"src":16183},"/static/redlm/notebooklm.png",[11,16185,16186,16187,643],{},"You can ",[20,16188,16191],{"href":16189,"rel":16190},"https://x.com/briancaffey/status/1855186771409244491",[24],"listen to this podcast episode here on 𝕏",[16193,16194],"red-lm-deep-dive-video",{},[56,16196,16198],{"id":16197},"how-i-built-redlm","How I built RedLM",[11,16200,16201],{},"RedLM consists of two parts: a web UI built with Vue 3 using the Nuxt Framework and a backend API built with Python, FastAPI and LlamaIndex. There are lots of great tools for building full-stack AI applications such as Gradio and Streamlit, but I wanted to build with the web tools that I'm most familiar with and that provide the most flexibility. These frameworks (Nuxt and FastAPI) are simple and effective and they allowed me to develop quickly.",[11,16203,16204],{},"Most of the code for this project was written by AI. I used OpenAI's ChatGPT 4o, Anthropic's Claude 3.5 Sonnet and 01.AI's Yi-1.5-9B-Chat model. In my development process with AI, I prompted for one logical piece of the application at a time, such as one API route, one Vue component, one pinia store or one utility function, for example. In this article I'll share some of the prompts I used in my development workflow.",[11,16206,16207,16208,16211],{},"This project embraces a hybrid AI inference model, meaning that the AI inference can be done either on local RTX PCs or using NVIDIA's Cloud APIs from ",[30,16209,16210],{},"build.nvidia.com"," depending on configuration via environment variables. I used PCs with NVIDIA GeForce RTX 4090 GPUs to do inference with language and vision models, and with a change of configuration, I was able to do similar inference using NVIDIA's API endpoints. This allowed me to develop the project both on powerful RTX desktop workstations and Mac laptops.",[56,16213,16215],{"id":16214},"translating-dream-of-the-red-chamber-with-tensorrt-llm","Translating Dream of the Red Chamber with TensorRT-LLM",[11,16217,16218],{},"Translation is often mentioned as one of the capabilities of bilingual LLMs from China. I wanted to try translating this book from Chinese to English, but I also wanted to better understand the meaning of the original text written in vernacular Chinese. Written vernacular Chinese is essentially a form of Chinese that closely resembles the way Chinese was spoken in imperial China by common people. The use of vernacular Chinese (Baihua) in literary works marked a significant cultural shift that started to make literature and education more accessible. Before the emergence of written vernacular Chinese, Chinese literature was dominated by Classical Chinese (Wenyanwen) which is a more concise, ambiguous and specialized form of language that assumes an understanding of ancient texts and Confucian classics. The difference between vernacular Chinese and modern Mandarin Chinese is somewhat analogous to the difference between Shakespearean English (Early Modern English) and Modern English.",[11,16220,16221],{},[2718,16222],{"alt":16223,"src":16224},"Baihua, Mandarin and English","/static/redlm/translations.png",[11,16226,16227],{},"Chinese large language models are well versed in Classical Chinese, written Chinese vernacular and modern Mandarin Chinese. I decided to rewrite the original vernacular text in simple, modern Mandarin Chinese and then using this new modern Mandarin version, translate the story into English.",[11,16229,16230],{},"Dream of the Red Chamber is a large book. It is composed of over 800,000 Chinese characters, using 4303 unique Chinese characters. It has 120 chapters and a total of 3996 paragraphs. Here is a histogram showing the number of characters per paragraph.",[11,16232,16233],{},[2718,16234],{"alt":16235,"src":16236},"Paragraph lengths","/static/redlm/paragraphs.png",[11,16238,16239,16240,16245,16246,16249],{},"I rented a large multi-GPU instance from AWS using some of the credits I get as a member of the AWS Community Builders program. The g5.12xlarge instance I selected has 4 A10G Tensor Core GPUs. The TensorRT-LLM LLM API is a relatively new part of the TensorRT-LLM library. It provides a very simple, high-level interface for doing inference. Following the ",[20,16241,16244],{"href":16242,"rel":16243},"https://nvidia.github.io/TensorRT-LLM/llm-api-examples/llm_inference_distributed.html",[24],"LLM Generate Distributed example"," from the TensorRT-LLM documentation, I was able to translate the entire book into simple Mandarin and then from Mandarin into English in about an hour and 15 minutes. The ",[30,16247,16248],{},"tensor_parallel_size"," option in the LLM API allows for distributed inference, this meant that up to 4 paragraphs could be translated at the same time on different GPUs on the same EC2 instance.",[459,16251,16254],{"className":16252,"code":16253,"language":997},[995],"Translating: data/book/22.json\nProcessed requests: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 38/38 [00:15\u003C00:00,  2.41it/s]\nProcessed requests: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 38/38 [00:24\u003C00:00,  1.54it/s]\nTranslated: data/book/22.json\nTranslating: data/book/114.json\nProcessed requests: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:11\u003C00:00,  1.81it/s]\nProcessed requests: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:12\u003C00:00,  1.58it/s]\nTranslated: data/book/114.json\n[TensorRT-LLM][INFO] Refreshed the MPI local session\n[TensorRT-LLM][INFO] Refreshed the MPI local session\n[TensorRT-LLM][INFO] Refreshed the MPI local session\n[TensorRT-LLM][INFO] Refreshed the MPI local session\n\nreal    74m1.578s\nuser    0m45.936s\nsys 0m36.283s\n",[30,16255,16253],{"__ignoreMap":464},[11,16257,16258],{},"Getting good results required a bit of experimentation with parameters. The LLM API makes this very easy. The following code configures settings and builds the inference engine that can be used for doing completions:",[459,16260,16262],{"className":13136,"code":16261,"language":12886,"meta":464,"style":464},"sampling_params = SamplingParams(temperature=0.7, top_p=0.95, max_tokens=256)\nbuild_config = BuildConfig(max_seq_len=2048)\nllm = LLM(model=MODEL, build_config=build_config, tensor_parallel_size=4)\n",[30,16263,16264,16302,16321],{"__ignoreMap":464},[151,16265,16266,16269,16271,16274,16277,16279,16282,16284,16286,16288,16291,16293,16295,16297,16300],{"class":469,"line":470},[151,16267,16268],{"class":503},"sampling_params ",[151,16270,1876],{"class":1869},[151,16272,16273],{"class":503}," SamplingParams(",[151,16275,16276],{"class":15210},"temperature",[151,16278,1876],{"class":1869},[151,16280,16281],{"class":477},"0.7",[151,16283,106],{"class":503},[151,16285,8509],{"class":15210},[151,16287,1876],{"class":1869},[151,16289,16290],{"class":477},"0.95",[151,16292,106],{"class":503},[151,16294,8532],{"class":15210},[151,16296,1876],{"class":1869},[151,16298,16299],{"class":477},"256",[151,16301,3640],{"class":503},[151,16303,16304,16307,16309,16312,16315,16317,16319],{"class":469,"line":488},[151,16305,16306],{"class":503},"build_config ",[151,16308,1876],{"class":1869},[151,16310,16311],{"class":503}," BuildConfig(",[151,16313,16314],{"class":15210},"max_seq_len",[151,16316,1876],{"class":1869},[151,16318,309],{"class":477},[151,16320,3640],{"class":503},[151,16322,16323,16326,16328,16331,16333,16335,16338,16340,16343,16345,16348,16350,16352,16354],{"class":469,"line":500},[151,16324,16325],{"class":503},"llm ",[151,16327,1876],{"class":1869},[151,16329,16330],{"class":503}," LLM(",[151,16332,8340],{"class":15210},[151,16334,1876],{"class":1869},[151,16336,16337],{"class":477},"MODEL",[151,16339,106],{"class":503},[151,16341,16342],{"class":15210},"build_config",[151,16344,1876],{"class":1869},[151,16346,16347],{"class":503},"build_config, ",[151,16349,16248],{"class":15210},[151,16351,1876],{"class":1869},[151,16353,9187],{"class":477},[151,16355,3640],{"class":503},[11,16357,16358],{},"I used the following prompts to rewrite each paragraph of the original text in simple, modern Mandarin Chinese:",[459,16360,16362],{"className":13136,"code":16361,"language":12886,"meta":464,"style":464},"bai_prompts = [\n    # Here are examples of how to rewrite Chinese vernacular into simple modern Mandarin.\\n\\nChinese vernacular:\\n\\n{p}\\n\\nSimple modern Mandarin\n    f\"以下是如何将中国白话改写为简单的现代普通话的示例。\\n\\n中文白话：\\n\\n{p}\\n\\n简单的现代普通话：\\n\\n\"\n    for p in flat_bai\n]\n",[30,16363,16364,16373,16378,16407,16421],{"__ignoreMap":464},[151,16365,16366,16369,16371],{"class":469,"line":470},[151,16367,16368],{"class":503},"bai_prompts ",[151,16370,1876],{"class":1869},[151,16372,13149],{"class":503},[151,16374,16375],{"class":469,"line":488},[151,16376,16377],{"class":1527},"    # Here are examples of how to rewrite Chinese vernacular into simple modern Mandarin.\\n\\nChinese vernacular:\\n\\n{p}\\n\\nSimple modern Mandarin\n",[151,16379,16380,16383,16386,16388,16391,16394,16396,16399,16402,16404],{"class":469,"line":500},[151,16381,16382],{"class":12347},"    f",[151,16384,16385],{"class":481},"\"以下是如何将中国白话改写为简单的现代普通话的示例。",[151,16387,8049],{"class":477},[151,16389,16390],{"class":481},"中文白话：",[151,16392,16393],{"class":477},"\\n\\n{",[151,16395,11],{"class":503},[151,16397,16398],{"class":477},"}\\n\\n",[151,16400,16401],{"class":481},"简单的现代普通话：",[151,16403,8049],{"class":477},[151,16405,16406],{"class":481},"\"\n",[151,16408,16409,16412,16415,16418],{"class":469,"line":509},[151,16410,16411],{"class":1869},"    for",[151,16413,16414],{"class":503}," p ",[151,16416,16417],{"class":1869},"in",[151,16419,16420],{"class":503}," flat_bai\n",[151,16422,16423],{"class":469,"line":517},[151,16424,3691],{"class":503},[11,16426,16427],{},"It was difficult to get good results consistently. Here are some observations I had:",[76,16429,16430,16433,16436,16439,16442],{},[79,16431,16432],{},"Some of the translated paragraphs were perfect",[79,16434,16435],{},"Some translated paragraphs would randomly hallucinate the same phrase over and over again",[79,16437,16438],{},"Some requests to translate text to English would reply in Mandarin Chinese rather than in English",[79,16440,16441],{},"Sometimes I would even see computer code generated when asking for a translation",[79,16443,16444],{},"The names of characters were sometimes translated inconsistently, sometimes literally and sometimes using differing versions of pinyin, the Romanization system for transcribing the sounds of Mandarin Chinese",[11,16446,16447,16448,16451,16452,10744,16455,16458,16459,16461],{},"I found that ChatGPT 4o could handle any Chinese translation task flawlessly, but the ",[30,16449,16450],{},"Qwen2-7B"," model I used had mixed results! The change that I made that seemed to have the biggest impact on translation quality was setting ",[30,16453,16454],{},"*max_tokens*=256",[30,16456,16457],{},"SamplingParams",". I probably could have used a dynamic value for ",[30,16460,8532],{}," based on the size of the current paragraph being translated. I also would have like to set up side-by-side comparisons of translations using different sized models, but rather than spend time and AWS credits on optimizing translation with TensorRT-LLM, I wanted to focus on the main part of this project: retrieval augmented generation (RAG) with LlamaIndex.",[56,16463,16465],{"id":16464},"building-qa-bots-with-rag-using-llamaindex","Building Q&A bots with RAG using LlamaIndex",[11,16467,16468],{},"My primary objective with this project was to implement a simple chat bot that responds to questions about the book with references to the specific paragraphs used in the response. The following shows images of the UI I built with one of the examples I included in the video I made for this project.",[11,16470,16471],{},[2718,16472],{"alt":16473,"src":16474},"RAG Example","/static/redlm/rag_example.png",[11,16476,16477,16478,16481],{},"I haven't read that much of the book before working on this project, but I have read a lot ",[51,16479,16480],{},"about"," this book's characters, major themes and plot. This Q&A bot was a very interesting entrypoint to explore specific passages of the book starting with questions coming from my knowledge about the book. The question in the screenshots above is: \"What does Jia Baoyu's father think about him?\" The response includes references to paragraphs where Jia Zheng (Baoyu's father) is discussing his son. I was pretty amazed that the RAG query was able to pull out these two paragraphs.",[11,16483,16484],{},[51,16485,16486],{},"In Dream of the Red Chamber, the relationship between protagonist Jia Baoyu and his father, Jia Zheng, is complex and fraught with tension. Jia Zheng, a strict, traditional Confucian patriarch, embodies values of discipline, scholarly rigor, and duty. He expects his son to excel in his studies and uphold the family's honor by pursuing an official career in government. Baoyu, however, is sensitive, imaginative, and inclined toward poetry and the company of women, especially his cousins Lin Daiyu and Xue Baochai. This preference clashes with Jia Zheng's expectations, leading to frequent misunderstandings and disappointment.",[11,16488,16489],{},"By default, LlamaIndex uses cosine similarity as the distance metric for finding the vectors representing the documents (paragraphs) that are \"closest\" to the vector representing the user query. This is the central mechanism by which RAG works. LlamaIndex provides an abstraction of this process, hiding the implementation details and allowing rapid development of retrieval systems.",[11,16491,16492],{},[2718,16493],{"alt":16494,"src":16495},"Cosine Similarity","/static/redlm/cosine_similarity.png",[11,16497,16498,16499],{},"Source: ",[20,16500,16501],{"href":16501,"rel":16502},"https://medium.com/@kbdhunga/a-beginners-guide-to-similarity-search-vector-indexing-part-one-9cf5e9171976",[24],[11,16504,16505,16506,16509],{},"Here is some of the code I wrote for the text-based Q&A bot using LlamaIndex's ",[30,16507,16508],{},"CustomQueryEngine"," class to fetch the nodes from which I get the referenced paragraph text, chapter number and paragraph number.",[459,16511,16513],{"className":13136,"code":16512,"language":12886,"meta":464,"style":464},"class QAndAQueryEngine(CustomQueryEngine):\n    \"\"\"RAG Completion Query Engine optimized for Q&A\"\"\"\n\n    retriever: BaseRetriever\n    response_synthesizer: BaseSynthesizer\n    llm: OpenAILike\n    qa_prompt: PromptTemplate\n\n    def custom_query(self, query_str: str):\n        nodes = self.retriever.retrieve(query_str)\n        metadata = []\n        # Collect the metadata into a list of dicts so that it can be sent to UI for references\n        for node in nodes:\n            metadata_dict = {}\n            node_metadata = node.node.metadata\n            metadata_dict[\"content\"] = node.node.text\n            metadata_dict[\"chapter\"] = int(node_metadata.get(\"chapter\"))\n            metadata_dict[\"paragraph\"] = int(node_metadata.get(\"paragraph\"))\n\n            metadata.append(metadata_dict)\n\n        context_str = \"\\n\\n\".join([n.node.get_content() for n in nodes])\n        response = self.llm.chat(\n            [\n                ChatMessage(\n                    role=\"user\",\n                    content=q_and_a_prompt.format( # the English and Chinese prompt templates are discussed below\n                        context_str=context_str, query_str=query_str\n                    ),\n                )\n            ]\n        )\n\n        return response, metadata\n",[30,16514,16515,16529,16534,16538,16543,16548,16553,16558,16562,16585,16597,16607,16612,16625,16635,16645,16660,16681,16700,16704,16709,16713,16741,16753,16758,16763,16775,16788,16805,16810,16815,16820,16825,16829],{"__ignoreMap":464},[151,16516,16517,16520,16523,16525,16527],{"class":469,"line":470},[151,16518,16519],{"class":12347},"class",[151,16521,16522],{"class":15254}," QAndAQueryEngine",[151,16524,12386],{"class":503},[151,16526,16508],{"class":15260},[151,16528,15264],{"class":503},[151,16530,16531],{"class":469,"line":488},[151,16532,16533],{"class":481},"    \"\"\"RAG Completion Query Engine optimized for Q&A\"\"\"\n",[151,16535,16536],{"class":469,"line":500},[151,16537,1090],{"emptyLinePlaceholder":609},[151,16539,16540],{"class":469,"line":509},[151,16541,16542],{"class":503},"    retriever: BaseRetriever\n",[151,16544,16545],{"class":469,"line":517},[151,16546,16547],{"class":503},"    response_synthesizer: BaseSynthesizer\n",[151,16549,16550],{"class":469,"line":534},[151,16551,16552],{"class":503},"    llm: OpenAILike\n",[151,16554,16555],{"class":469,"line":1413},[151,16556,16557],{"class":503},"    qa_prompt: PromptTemplate\n",[151,16559,16560],{"class":469,"line":1418},[151,16561,1090],{"emptyLinePlaceholder":609},[151,16563,16564,16567,16570,16572,16574,16576,16579,16581,16583],{"class":469,"line":2462},[151,16565,16566],{"class":12347},"    def",[151,16568,16569],{"class":473}," custom_query",[151,16571,12386],{"class":503},[151,16573,15277],{"class":15232},[151,16575,106],{"class":503},[151,16577,16578],{"class":15232},"query_str",[151,16580,6208],{"class":503},[151,16582,15343],{"class":6205},[151,16584,15264],{"class":503},[151,16586,16587,16590,16592,16594],{"class":469,"line":2471},[151,16588,16589],{"class":503},"        nodes ",[151,16591,1876],{"class":1869},[151,16593,15451],{"class":15289},[151,16595,16596],{"class":503},".retriever.retrieve(query_str)\n",[151,16598,16599,16602,16604],{"class":469,"line":2480},[151,16600,16601],{"class":503},"        metadata ",[151,16603,1876],{"class":1869},[151,16605,16606],{"class":503}," []\n",[151,16608,16609],{"class":469,"line":2489},[151,16610,16611],{"class":1527},"        # Collect the metadata into a list of dicts so that it can be sent to UI for references\n",[151,16613,16614,16617,16620,16622],{"class":469,"line":2497},[151,16615,16616],{"class":1869},"        for",[151,16618,16619],{"class":503}," node ",[151,16621,16417],{"class":1869},[151,16623,16624],{"class":503}," nodes:\n",[151,16626,16627,16630,16632],{"class":469,"line":3140},[151,16628,16629],{"class":503},"            metadata_dict ",[151,16631,1876],{"class":1869},[151,16633,16634],{"class":503}," {}\n",[151,16636,16637,16640,16642],{"class":469,"line":3149},[151,16638,16639],{"class":503},"            node_metadata ",[151,16641,1876],{"class":1869},[151,16643,16644],{"class":503}," node.node.metadata\n",[151,16646,16647,16650,16652,16655,16657],{"class":469,"line":3158},[151,16648,16649],{"class":503},"            metadata_dict[",[151,16651,9808],{"class":481},[151,16653,16654],{"class":503},"] ",[151,16656,1876],{"class":1869},[151,16658,16659],{"class":503}," node.node.text\n",[151,16661,16662,16664,16667,16669,16671,16674,16677,16679],{"class":469,"line":3167},[151,16663,16649],{"class":503},[151,16665,16666],{"class":481},"\"chapter\"",[151,16668,16654],{"class":503},[151,16670,1876],{"class":1869},[151,16672,16673],{"class":6205}," int",[151,16675,16676],{"class":503},"(node_metadata.get(",[151,16678,16666],{"class":481},[151,16680,12451],{"class":503},[151,16682,16683,16685,16688,16690,16692,16694,16696,16698],{"class":469,"line":3175},[151,16684,16649],{"class":503},[151,16686,16687],{"class":481},"\"paragraph\"",[151,16689,16654],{"class":503},[151,16691,1876],{"class":1869},[151,16693,16673],{"class":6205},[151,16695,16676],{"class":503},[151,16697,16687],{"class":481},[151,16699,12451],{"class":503},[151,16701,16702],{"class":469,"line":3184},[151,16703,1090],{"emptyLinePlaceholder":609},[151,16705,16706],{"class":469,"line":3193},[151,16707,16708],{"class":503},"            metadata.append(metadata_dict)\n",[151,16710,16711],{"class":469,"line":3720},[151,16712,1090],{"emptyLinePlaceholder":609},[151,16714,16715,16718,16720,16723,16725,16727,16730,16733,16736,16738],{"class":469,"line":3729},[151,16716,16717],{"class":503},"        context_str ",[151,16719,1876],{"class":1869},[151,16721,16722],{"class":481}," \"",[151,16724,8049],{"class":477},[151,16726,8592],{"class":481},[151,16728,16729],{"class":503},".join([n.node.get_content() ",[151,16731,16732],{"class":1869},"for",[151,16734,16735],{"class":503}," n ",[151,16737,16417],{"class":1869},[151,16739,16740],{"class":503}," nodes])\n",[151,16742,16743,16746,16748,16750],{"class":469,"line":3735},[151,16744,16745],{"class":503},"        response ",[151,16747,1876],{"class":1869},[151,16749,15451],{"class":15289},[151,16751,16752],{"class":503},".llm.chat(\n",[151,16754,16755],{"class":469,"line":3745},[151,16756,16757],{"class":503},"            [\n",[151,16759,16760],{"class":469,"line":3754},[151,16761,16762],{"class":503},"                ChatMessage(\n",[151,16764,16765,16768,16770,16773],{"class":469,"line":3760},[151,16766,16767],{"class":15210},"                    role",[151,16769,1876],{"class":1869},[151,16771,16772],{"class":481},"\"user\"",[151,16774,9417],{"class":503},[151,16776,16777,16780,16782,16785],{"class":469,"line":3773},[151,16778,16779],{"class":15210},"                    content",[151,16781,1876],{"class":1869},[151,16783,16784],{"class":503},"q_and_a_prompt.format( ",[151,16786,16787],{"class":1527},"# the English and Chinese prompt templates are discussed below\n",[151,16789,16790,16793,16795,16798,16800,16802],{"class":469,"line":3782},[151,16791,16792],{"class":15210},"                        context_str",[151,16794,1876],{"class":1869},[151,16796,16797],{"class":503},"context_str, ",[151,16799,16578],{"class":15210},[151,16801,1876],{"class":1869},[151,16803,16804],{"class":503},"query_str\n",[151,16806,16807],{"class":469,"line":3791},[151,16808,16809],{"class":503},"                    ),\n",[151,16811,16812],{"class":469,"line":3803},[151,16813,16814],{"class":503},"                )\n",[151,16816,16817],{"class":469,"line":3811},[151,16818,16819],{"class":503},"            ]\n",[151,16821,16822],{"class":469,"line":3820},[151,16823,16824],{"class":503},"        )\n",[151,16826,16827],{"class":469,"line":7084},[151,16828,1090],{"emptyLinePlaceholder":609},[151,16830,16831,16834],{"class":469,"line":7148},[151,16832,16833],{"class":1869},"        return",[151,16835,16836],{"class":503}," response, metadata\n",[736,16838,16840],{"id":16839},"indexing-the-book-data","Indexing the book data",[11,16842,16843],{},"In the indexing process, embedding models are used to translate chunks of text (paragraphs) into high-dimensional vectors that represent the relationships between the tokens in a chunk of text. These are the vectors stored in the \"Vector Database\" used by LlamaIndex. The chapter number, paragraph number and version (original, Mandarin Chinese and English) of each paragraph are added to the database entry as metadata during the indexing step which runs via a script before starting the FastAPI server. Here's how I indexed the original text and translations with LlamaIndex:",[459,16845,16847],{"className":13136,"code":16846,"language":12886,"meta":464,"style":464},"from llama_index.core import Document, VectorStoreIndex\nfrom llama_index.embeddings.huggingface import HuggingFaceEmbedding\n\nen_embedding_model = HuggingFaceEmbedding(model_name=\"BAAI/bge-small-en-v1.5\")\nzh_embedding_model = HuggingFaceEmbedding(model_name=\"BAAI/bge-small-zh-v1.5\")\n\ndef persist_index():\n    documents = []\n    for chapter in range(1, 121):\n        with open(f\"data/book/{chapter}.json\", \"r\") as f:\n            data = json.load(f)\n            paragraphs = data[\"paragraphs\"]\n\n        for i, p in enumerate(paragraphs):\n            for lang in [\"original\", \"chinese\", \"english\"]:\n                document = Document(\n                    text=p[lang],\n                    metadata={\n                        \"chapter\": str(chapter),\n                        \"paragraph\": str(i),\n                        \"language\": lang,\n                    },\n                    metadata_seperator=\"::\",\n                    metadata_template=\"{key}=>{value}\",\n                    text_template=\"Metadata: {metadata_str}\\n-----\\nContent: {content}\",\n                    embed_model=(\n                        en_embedding_model if lang == \"english\" else zh_embedding_model\n                    ),\n                )\n                documents.append(document)\n\n    index = VectorStoreIndex.from_documents(documents)\n    index.storage_context.persist(persist_dir=\"storage\")\n\nif __name__ == \"__main__\":\n    persist_index()\n",[30,16848,16849,16863,16875,16879,16899,16917,16921,16932,16941,16963,17002,17012,17027,17031,17046,17074,17084,17094,17103,17115,17127,17135,17140,17152,17174,17202,17211,17233,17237,17241,17246,17250,17260,17275,17279,17294],{"__ignoreMap":464},[151,16850,16851,16854,16857,16860],{"class":469,"line":470},[151,16852,16853],{"class":1869},"from",[151,16855,16856],{"class":503}," llama_index.core ",[151,16858,16859],{"class":1869},"import",[151,16861,16862],{"class":503}," Document, VectorStoreIndex\n",[151,16864,16865,16867,16870,16872],{"class":469,"line":488},[151,16866,16853],{"class":1869},[151,16868,16869],{"class":503}," llama_index.embeddings.huggingface ",[151,16871,16859],{"class":1869},[151,16873,16874],{"class":503}," HuggingFaceEmbedding\n",[151,16876,16877],{"class":469,"line":500},[151,16878,1090],{"emptyLinePlaceholder":609},[151,16880,16881,16884,16886,16889,16892,16894,16897],{"class":469,"line":509},[151,16882,16883],{"class":503},"en_embedding_model ",[151,16885,1876],{"class":1869},[151,16887,16888],{"class":503}," HuggingFaceEmbedding(",[151,16890,16891],{"class":15210},"model_name",[151,16893,1876],{"class":1869},[151,16895,16896],{"class":481},"\"BAAI/bge-small-en-v1.5\"",[151,16898,3640],{"class":503},[151,16900,16901,16904,16906,16908,16910,16912,16915],{"class":469,"line":517},[151,16902,16903],{"class":503},"zh_embedding_model ",[151,16905,1876],{"class":1869},[151,16907,16888],{"class":503},[151,16909,16891],{"class":15210},[151,16911,1876],{"class":1869},[151,16913,16914],{"class":481},"\"BAAI/bge-small-zh-v1.5\"",[151,16916,3640],{"class":503},[151,16918,16919],{"class":469,"line":534},[151,16920,1090],{"emptyLinePlaceholder":609},[151,16922,16923,16926,16929],{"class":469,"line":1413},[151,16924,16925],{"class":12347},"def",[151,16927,16928],{"class":473}," persist_index",[151,16930,16931],{"class":503},"():\n",[151,16933,16934,16937,16939],{"class":469,"line":1418},[151,16935,16936],{"class":503},"    documents ",[151,16938,1876],{"class":1869},[151,16940,16606],{"class":503},[151,16942,16943,16945,16948,16950,16952,16954,16956,16958,16961],{"class":469,"line":2462},[151,16944,16411],{"class":1869},[151,16946,16947],{"class":503}," chapter ",[151,16949,16417],{"class":1869},[151,16951,2793],{"class":2226},[151,16953,12386],{"class":503},[151,16955,6760],{"class":477},[151,16957,106],{"class":503},[151,16959,16960],{"class":477},"121",[151,16962,15264],{"class":503},[151,16964,16965,16968,16971,16973,16975,16978,16980,16983,16985,16988,16990,16993,16996,16999],{"class":469,"line":2471},[151,16966,16967],{"class":1869},"        with",[151,16969,16970],{"class":2226}," open",[151,16972,12386],{"class":503},[151,16974,13214],{"class":12347},[151,16976,16977],{"class":481},"\"data/book/",[151,16979,5729],{"class":477},[151,16981,16982],{"class":503},"chapter",[151,16984,2001],{"class":477},[151,16986,16987],{"class":481},".json\"",[151,16989,106],{"class":503},[151,16991,16992],{"class":481},"\"r\"",[151,16994,16995],{"class":503},") ",[151,16997,16998],{"class":1869},"as",[151,17000,17001],{"class":503}," f:\n",[151,17003,17004,17007,17009],{"class":469,"line":2480},[151,17005,17006],{"class":503},"            data ",[151,17008,1876],{"class":1869},[151,17010,17011],{"class":503}," json.load(f)\n",[151,17013,17014,17017,17019,17022,17025],{"class":469,"line":2489},[151,17015,17016],{"class":503},"            paragraphs ",[151,17018,1876],{"class":1869},[151,17020,17021],{"class":503}," data[",[151,17023,17024],{"class":481},"\"paragraphs\"",[151,17026,3691],{"class":503},[151,17028,17029],{"class":469,"line":2497},[151,17030,1090],{"emptyLinePlaceholder":609},[151,17032,17033,17035,17038,17040,17043],{"class":469,"line":3140},[151,17034,16616],{"class":1869},[151,17036,17037],{"class":503}," i, p ",[151,17039,16417],{"class":1869},[151,17041,17042],{"class":2226}," enumerate",[151,17044,17045],{"class":503},"(paragraphs):\n",[151,17047,17048,17051,17054,17056,17058,17061,17063,17066,17068,17071],{"class":469,"line":3149},[151,17049,17050],{"class":1869},"            for",[151,17052,17053],{"class":503}," lang ",[151,17055,16417],{"class":1869},[151,17057,6604],{"class":503},[151,17059,17060],{"class":481},"\"original\"",[151,17062,106],{"class":503},[151,17064,17065],{"class":481},"\"chinese\"",[151,17067,106],{"class":503},[151,17069,17070],{"class":481},"\"english\"",[151,17072,17073],{"class":503},"]:\n",[151,17075,17076,17079,17081],{"class":469,"line":3158},[151,17077,17078],{"class":503},"                document ",[151,17080,1876],{"class":1869},[151,17082,17083],{"class":503}," Document(\n",[151,17085,17086,17089,17091],{"class":469,"line":3167},[151,17087,17088],{"class":15210},"                    text",[151,17090,1876],{"class":1869},[151,17092,17093],{"class":503},"p[lang],\n",[151,17095,17096,17099,17101],{"class":469,"line":3175},[151,17097,17098],{"class":15210},"                    metadata",[151,17100,1876],{"class":1869},[151,17102,12966],{"class":503},[151,17104,17105,17108,17110,17112],{"class":469,"line":3184},[151,17106,17107],{"class":481},"                        \"chapter\"",[151,17109,6208],{"class":503},[151,17111,15343],{"class":6205},[151,17113,17114],{"class":503},"(chapter),\n",[151,17116,17117,17120,17122,17124],{"class":469,"line":3193},[151,17118,17119],{"class":481},"                        \"paragraph\"",[151,17121,6208],{"class":503},[151,17123,15343],{"class":6205},[151,17125,17126],{"class":503},"(i),\n",[151,17128,17129,17132],{"class":469,"line":3720},[151,17130,17131],{"class":481},"                        \"language\"",[151,17133,17134],{"class":503},": lang,\n",[151,17136,17137],{"class":469,"line":3729},[151,17138,17139],{"class":503},"                    },\n",[151,17141,17142,17145,17147,17150],{"class":469,"line":3735},[151,17143,17144],{"class":15210},"                    metadata_seperator",[151,17146,1876],{"class":1869},[151,17148,17149],{"class":481},"\"::\"",[151,17151,9417],{"class":503},[151,17153,17154,17157,17159,17161,17164,17167,17170,17172],{"class":469,"line":3745},[151,17155,17156],{"class":15210},"                    metadata_template",[151,17158,1876],{"class":1869},[151,17160,8592],{"class":481},[151,17162,17163],{"class":477},"{key}",[151,17165,17166],{"class":481},"=>",[151,17168,17169],{"class":477},"{value}",[151,17171,8592],{"class":481},[151,17173,9417],{"class":503},[151,17175,17176,17179,17181,17184,17187,17190,17192,17195,17198,17200],{"class":469,"line":3754},[151,17177,17178],{"class":15210},"                    text_template",[151,17180,1876],{"class":1869},[151,17182,17183],{"class":481},"\"Metadata: ",[151,17185,17186],{"class":477},"{metadata_str}\\n",[151,17188,17189],{"class":481},"-----",[151,17191,8043],{"class":477},[151,17193,17194],{"class":481},"Content: ",[151,17196,17197],{"class":477},"{content}",[151,17199,8592],{"class":481},[151,17201,9417],{"class":503},[151,17203,17204,17207,17209],{"class":469,"line":3760},[151,17205,17206],{"class":15210},"                    embed_model",[151,17208,1876],{"class":1869},[151,17210,15410],{"class":503},[151,17212,17213,17216,17219,17221,17224,17227,17230],{"class":469,"line":3773},[151,17214,17215],{"class":503},"                        en_embedding_model ",[151,17217,17218],{"class":1869},"if",[151,17220,17053],{"class":503},[151,17222,17223],{"class":1869},"==",[151,17225,17226],{"class":481}," \"english\"",[151,17228,17229],{"class":1869}," else",[151,17231,17232],{"class":503}," zh_embedding_model\n",[151,17234,17235],{"class":469,"line":3782},[151,17236,16809],{"class":503},[151,17238,17239],{"class":469,"line":3791},[151,17240,16814],{"class":503},[151,17242,17243],{"class":469,"line":3803},[151,17244,17245],{"class":503},"                documents.append(document)\n",[151,17247,17248],{"class":469,"line":3811},[151,17249,1090],{"emptyLinePlaceholder":609},[151,17251,17252,17255,17257],{"class":469,"line":3820},[151,17253,17254],{"class":503},"    index ",[151,17256,1876],{"class":1869},[151,17258,17259],{"class":503}," VectorStoreIndex.from_documents(documents)\n",[151,17261,17262,17265,17268,17270,17273],{"class":469,"line":7084},[151,17263,17264],{"class":503},"    index.storage_context.persist(",[151,17266,17267],{"class":15210},"persist_dir",[151,17269,1876],{"class":1869},[151,17271,17272],{"class":481},"\"storage\"",[151,17274,3640],{"class":503},[151,17276,17277],{"class":469,"line":7148},[151,17278,1090],{"emptyLinePlaceholder":609},[151,17280,17281,17283,17286,17289,17292],{"class":469,"line":7211},[151,17282,17218],{"class":1869},[151,17284,17285],{"class":12360}," __name__",[151,17287,17288],{"class":1869}," ==",[151,17290,17291],{"class":481}," \"__main__\"",[151,17293,14372],{"class":503},[151,17295,17296],{"class":469,"line":7273},[151,17297,17298],{"class":503},"    persist_index()\n",[11,17300,17301,17302,17305],{},"For the embedding models, I used the small BAAI General Embedding models (BGE) for English and Chinese. BAAI is the Beijing Academy of Artificial Intelligence, and I learned about this organization through some of the examples on the LlamaIndex site that use BAAI embeddings. There are multi-lingual embedding models (e.g. ",[30,17303,17304],{},"BAAI/bge-m3","), but setting the embedding model on a per-document basis is possible and in some cases it might be preferable to using a single embedding model for all documents.",[736,17307,17309],{"id":17308},"milvus-vector-database","Milvus Vector Database",[11,17311,17312],{},"I did most of the development for this project using the in-memory VectorIndexStore provided by LlamaIndex. This worked well, but making any changes to the FastAPI server required the data to be reloaded into memory which took several seconds each time. This can really hinder a good development flow, so I looked into using an external service for the vector database instead of running it in memory.",[11,17314,17315],{},[2718,17316],{"alt":17317,"src":17318},"Vector Database Options","/static/redlm/vectordbs.png",[11,17320,17321,17322,17329],{},"There are a LOT of options to consider when picking a vector database for a RAG application. Milvus has a highly decoupled architecture, it is fully open source and I had seen it in some examples in the ",[20,17323,17326],{"href":17324,"rel":17325},"https://github.com/NVIDIA/GenerativeAIExamples/tree/main/RAG/examples/advanced_rag/multimodal_rag",[24],[30,17327,17328],{},"NVIDIA/GenerativeAIExamples"," repo, so I decided to give it a try.",[11,17331,17332],{},[2718,17333],{"alt":17334,"src":17335},"Milvus Vector Database Architecture","/static/redlm/milvus.png",[11,17337,17338,17339,17344],{},"Using the ",[20,17340,17343],{"href":17341,"rel":17342},"https://milvus.io/docs/v2.0.x/install_standalone-docker.md",[24],"Milvus docker compose example"," I was able to set up an external vector database based on etcd and minio. Milvus also provides a Helm chart for running their vector database, this would be helpful if I was going to be running everything in Kubernetes (inference, vector database and application containers).",[736,17346,17348],{"id":17347},"other-examples-of-rag-with-english-questions","Other examples of RAG with English questions",[11,17350,17351],{},"One interesting design question I faced was how to support answering questions in both English and Chinese. I initially built the Q&A bot with only Chinese language support. Later, I added a simple helper function to determine if the input text is Chinese:",[459,17353,17355],{"className":13136,"code":17354,"language":12886,"meta":464,"style":464},"def is_chinese_text(text: str) -> bool:\n    \"\"\"\n    This is a simple helper function that is used to determine which prompt to use\n    depending on the language of the original user query\n    \"\"\"\n    chinese_count = sum(1 for char in text if '\\u4e00' \u003C= char \u003C= '\\u9fff')\n    english_count = sum(1 for char in text if 'a' \u003C= char.lower() \u003C= 'z')\n\n    return chinese_count > english_count\n",[30,17356,17357,17380,17385,17390,17395,17399,17450,17488,17492],{"__ignoreMap":464},[151,17358,17359,17361,17364,17366,17368,17370,17372,17375,17378],{"class":469,"line":470},[151,17360,16925],{"class":12347},[151,17362,17363],{"class":473}," is_chinese_text",[151,17365,12386],{"class":503},[151,17367,997],{"class":15232},[151,17369,6208],{"class":503},[151,17371,15343],{"class":6205},[151,17373,17374],{"class":503},") -> ",[151,17376,17377],{"class":6205},"bool",[151,17379,14372],{"class":503},[151,17381,17382],{"class":469,"line":488},[151,17383,17384],{"class":481},"    \"\"\"\n",[151,17386,17387],{"class":469,"line":500},[151,17388,17389],{"class":481},"    This is a simple helper function that is used to determine which prompt to use\n",[151,17391,17392],{"class":469,"line":509},[151,17393,17394],{"class":481},"    depending on the language of the original user query\n",[151,17396,17397],{"class":469,"line":517},[151,17398,17384],{"class":481},[151,17400,17401,17404,17406,17409,17411,17413,17415,17418,17420,17423,17425,17428,17431,17433,17436,17438,17441,17443,17446,17448],{"class":469,"line":534},[151,17402,17403],{"class":503},"    chinese_count ",[151,17405,1876],{"class":1869},[151,17407,17408],{"class":2226}," sum",[151,17410,12386],{"class":503},[151,17412,6760],{"class":477},[151,17414,2235],{"class":1869},[151,17416,17417],{"class":503}," char ",[151,17419,16417],{"class":1869},[151,17421,17422],{"class":503}," text ",[151,17424,17218],{"class":1869},[151,17426,17427],{"class":481}," '",[151,17429,17430],{"class":477},"\\u4e00",[151,17432,13223],{"class":481},[151,17434,17435],{"class":1869}," \u003C=",[151,17437,17417],{"class":503},[151,17439,17440],{"class":1869},"\u003C=",[151,17442,17427],{"class":481},[151,17444,17445],{"class":477},"\\u9fff",[151,17447,13223],{"class":481},[151,17449,3640],{"class":503},[151,17451,17452,17455,17457,17459,17461,17463,17465,17467,17469,17471,17473,17476,17478,17481,17483,17486],{"class":469,"line":1413},[151,17453,17454],{"class":503},"    english_count ",[151,17456,1876],{"class":1869},[151,17458,17408],{"class":2226},[151,17460,12386],{"class":503},[151,17462,6760],{"class":477},[151,17464,2235],{"class":1869},[151,17466,17417],{"class":503},[151,17468,16417],{"class":1869},[151,17470,17422],{"class":503},[151,17472,17218],{"class":1869},[151,17474,17475],{"class":481}," 'a'",[151,17477,17435],{"class":1869},[151,17479,17480],{"class":503}," char.lower() ",[151,17482,17440],{"class":1869},[151,17484,17485],{"class":481}," 'z'",[151,17487,3640],{"class":503},[151,17489,17490],{"class":469,"line":1418},[151,17491,1090],{"emptyLinePlaceholder":609},[151,17493,17494,17497,17500,17502],{"class":469,"line":2462},[151,17495,17496],{"class":1869},"    return",[151,17498,17499],{"class":503}," chinese_count ",[151,17501,3663],{"class":1869},[151,17503,17504],{"class":503}," english_count\n",[11,17506,17507,17508,17510,17511,17514],{},"This boolean value would then be used in the ",[30,17509,16508],{}," to use either the Chinese or English ",[30,17512,17513],{},"PromptTemplate",". This allowed the Q&A bot to answer questions in either Chinese or English, and it does not require translating back and forth between Chinese and English. However, this method relies on high-quality translations, so I don't expect English language questions to be answered as accurately as Chinese language questions. Here are the Chinese and English prompts that I used for the text-based Q&A bot, as well as some examples of the Q&A bot answering questions in English. The referenced materials include paragraphs from the English translation.",[459,17516,17518],{"className":13136,"code":17517,"language":12886,"meta":464,"style":464},"# Chinese prompt for text-based Q&A bot\nq_and_a_prompt = PromptTemplate(\n    \"这是相关的参考资料：\\n\"\n    \"---------------------\\n\"\n    \"{context_str}\\n\" # context_str contains Chinese paragraphs retrieved via RAG query\n    \"---------------------\\n\"\n    \"根据上述的参考资料，回答下面的问题\\n\"\n    \"问题：{user_question}\\n\"\n)\n\n# English prompt for text-based Q&A bot\nq_and_a_prompt_english = PromptTemplate(\n    \"This is some related reference material:\\n\"\n    \"---------------------\\n\"\n    \"{context_str}\\n\" # context_str contains English paragraphs retrieved via RAG query\n    \"---------------------\\n\"\n    \"Based on the above material, answer the following question:\\n\"\n    \"Question: {user_question}\\n\"\n)\n",[30,17519,17520,17525,17535,17544,17553,17566,17574,17583,17593,17597,17601,17606,17615,17624,17632,17643,17651,17660,17669],{"__ignoreMap":464},[151,17521,17522],{"class":469,"line":470},[151,17523,17524],{"class":1527},"# Chinese prompt for text-based Q&A bot\n",[151,17526,17527,17530,17532],{"class":469,"line":488},[151,17528,17529],{"class":503},"q_and_a_prompt ",[151,17531,1876],{"class":1869},[151,17533,17534],{"class":503}," PromptTemplate(\n",[151,17536,17537,17540,17542],{"class":469,"line":500},[151,17538,17539],{"class":481},"    \"这是相关的参考资料：",[151,17541,8043],{"class":477},[151,17543,16406],{"class":481},[151,17545,17546,17549,17551],{"class":469,"line":509},[151,17547,17548],{"class":481},"    \"---------------------",[151,17550,8043],{"class":477},[151,17552,16406],{"class":481},[151,17554,17555,17558,17561,17563],{"class":469,"line":517},[151,17556,17557],{"class":481},"    \"",[151,17559,17560],{"class":477},"{context_str}\\n",[151,17562,8592],{"class":481},[151,17564,17565],{"class":1527}," # context_str contains Chinese paragraphs retrieved via RAG query\n",[151,17567,17568,17570,17572],{"class":469,"line":534},[151,17569,17548],{"class":481},[151,17571,8043],{"class":477},[151,17573,16406],{"class":481},[151,17575,17576,17579,17581],{"class":469,"line":1413},[151,17577,17578],{"class":481},"    \"根据上述的参考资料，回答下面的问题",[151,17580,8043],{"class":477},[151,17582,16406],{"class":481},[151,17584,17585,17588,17591],{"class":469,"line":1418},[151,17586,17587],{"class":481},"    \"问题：",[151,17589,17590],{"class":477},"{user_question}\\n",[151,17592,16406],{"class":481},[151,17594,17595],{"class":469,"line":2462},[151,17596,3640],{"class":503},[151,17598,17599],{"class":469,"line":2471},[151,17600,1090],{"emptyLinePlaceholder":609},[151,17602,17603],{"class":469,"line":2480},[151,17604,17605],{"class":1527},"# English prompt for text-based Q&A bot\n",[151,17607,17608,17611,17613],{"class":469,"line":2489},[151,17609,17610],{"class":503},"q_and_a_prompt_english ",[151,17612,1876],{"class":1869},[151,17614,17534],{"class":503},[151,17616,17617,17620,17622],{"class":469,"line":2497},[151,17618,17619],{"class":481},"    \"This is some related reference material:",[151,17621,8043],{"class":477},[151,17623,16406],{"class":481},[151,17625,17626,17628,17630],{"class":469,"line":3140},[151,17627,17548],{"class":481},[151,17629,8043],{"class":477},[151,17631,16406],{"class":481},[151,17633,17634,17636,17638,17640],{"class":469,"line":3149},[151,17635,17557],{"class":481},[151,17637,17560],{"class":477},[151,17639,8592],{"class":481},[151,17641,17642],{"class":1527}," # context_str contains English paragraphs retrieved via RAG query\n",[151,17644,17645,17647,17649],{"class":469,"line":3158},[151,17646,17548],{"class":481},[151,17648,8043],{"class":477},[151,17650,16406],{"class":481},[151,17652,17653,17656,17658],{"class":469,"line":3167},[151,17654,17655],{"class":481},"    \"Based on the above material, answer the following question:",[151,17657,8043],{"class":477},[151,17659,16406],{"class":481},[151,17661,17662,17665,17667],{"class":469,"line":3175},[151,17663,17664],{"class":481},"    \"Question: ",[151,17666,17590],{"class":477},[151,17668,16406],{"class":481},[151,17670,17671],{"class":469,"line":3184},[151,17672,3640],{"class":503},[11,17674,17675],{},[2718,17676],{"alt":17677,"src":17678},"Multi-modal Q&A example 1","/static/redlm/qa_example_01.png",[11,17680,17681],{},"Asking random questions like this one is a fun way to explore the many scenes of Dream of the Red Chamber.",[11,17683,17684,17688],{},[2718,17685],{"alt":17686,"src":17687},"RAG Flower Pedal Example","/static/redlm/qa_example_flower_pedals.png",[2718,17689],{"alt":17690,"src":17691},"RAG Flower Pedal Example with Reference","/static/redlm/qa_example_flower_pedals_a.png",[56,17693,17695],{"id":17694},"redlm-rag-evaluation","RedLM RAG Evaluation",[11,17697,17698],{},"Examinations have long been a cornerstone of Chinese society, shaping individual aspirations, cultural values, and even government structures. This legacy began with the imperial civil service exams, kējǔ (科举), established during the Sui and Tang dynasties, and carries through in Modern China with the gaokao (高考) college entrance examination, both of which have allowed for unprecedented meritocratic routes to power and prestige. Given how widely this novel is studied in China, I was not surprised to find a wealth of examination questions written for students studying Dream of the Red Chamber.",[11,17700,17701,17702,17707],{},"I used ",[20,17703,17706],{"href":17704,"rel":17705},"https://www.examcoo.com/editor/do/view/id/246401",[24],"a set of 1000 multiple choice questions about Dream of the Red Chamber on examcoo.com"," to evaluate the effectiveness of the RAG system I built with LlamaIndex. I wrote a script to parse the questions from the website HTML using ChatGPT (parsing HTML is one of my favorite use cases of LLMs!) I filtered the list of 1000 questions down to 877 questions based on the following criteria:",[76,17709,17710,17716],{},[79,17711,17712,17715],{},[15,17713,17714],{},"Four answer choices",": some of the questions had more than four answer choices. I filtered questions with more than four answer choices to keep the evaluation simple. This would allow me to assume that random answer choices would have a 25% chance of being correct.",[79,17717,17718,17721],{},[15,17719,17720],{},"Only one answer",": For some questions the correct answer required selecting multiple answer choices. This would also help keep the evaluation logic simple.",[11,17723,17724],{},[2718,17725],{"alt":17726,"src":17727},"Multiple Choice Questions from Dream of the Red Chamber Test","/static/redlm/hlm_mcq.png",[11,17729,17730],{},"Multiple choice questions from a Dream of the Red Chamber test (examcoo.com)",[11,17732,17733],{},"To run the evaluation I set up two scripts. The first script would prompt the LLM to answer the question without any additional information from the RAG system. This served as a baseline to see how well the LLM could do at answering multiple choice questions about the book. The script simply checks to see if the LLM response contains the letter (A, B, C or D) of the correct answer and keeps track of the number of questions answered correctly.",[11,17735,17736],{},"Another script was used to take the test using large language models with RAG. In this script, the prompt sent to the LLM included relevant paragraphs from the book based on how similar the query is to each paragraph in the book based on the cosine similarity metric mentioned earlier.",[11,17738,17739],{},[2718,17740],{"alt":17741,"src":17742},"RAG evaluation","/static/redlm/rag_eval.png",[11,17744,17745],{},"Here are some results and other observations from this experiment:",[76,17747,17748,17751,17754,17760,17763,17770],{},[79,17749,17750],{},"LLMs alone scored in the mid 30% range (36%)",[79,17752,17753],{},"LLMs using retrieval augmented generation with the set of questions score in the mid 40% range (44%)",[79,17755,17756,17757,17759],{},"I used the completion API rather than the chat API and set the ",[30,17758,8532],{}," to 16. This was done to ensure that the LLM only gave a short response with a valid answer choice rather than giving a long response with an explanation.",[79,17761,17762],{},"The evaluation took longer for LLM + RAG test because of the time required for making the RAG query and the longer prompt (including both the original multiple-choice question and the referenced paragraphs).",[79,17764,17765,17766,17769],{},"I used the ",[30,17767,17768],{},"01-ai/Yi-1.5-9B-Chat"," model for this test, but I probably should have used the base model rather than the chat model.",[79,17771,17772],{},"Some questions would not be capable of being answered by RAG. For example, some of the questions are about film renditions of the novel. Most of the questions seemed relevant to the content of the book, so I didn't bother to filter out the questions that were not directly related to the book's content.",[11,17774,17775,17776,17779,17780,643],{},"Here is an example of a question that the LLM test script answered ",[51,17777,17778],{},"incorrectly"," and the LLM + RAG test script answered ",[15,17781,17782],{},"correctly",[210,17784,17785,17788,17791,17794,17797],{},[11,17786,17787],{},"秦钟的父亲是如何死的？",[11,17789,17790],{},"A、外感风寒、风毒之症",[11,17792,17793],{},"B、被智能儿气死的",[11,17795,17796],{},"C、生气引发旧病加重",[11,17798,17799],{},"D、生气而诱发中风而死",[11,17801,17802],{},"Translation:",[210,17804,17805,17808,17814,17820,17826],{},[11,17806,17807],{},"How did Qin Zhong's father die?",[11,17809,17810,17813],{},[15,17811,17812],{},"A."," He caught a cold and developed wind-related illnesses.",[11,17815,17816,17819],{},[15,17817,17818],{},"B."," He was angered to death by Zhineng'er (a character).",[11,17821,17822,17825],{},[15,17823,17824],{},"C."," His old illness worsened due to anger.",[11,17827,17828,17831],{},[15,17829,17830],{},"D."," He had a stroke induced by anger and died.",[11,17833,17834],{},"Here is the paragraphs that the RAG query returned along with the English translation:",[11,17836,17837],{},"Original",[210,17839,17840],{},[11,17841,17842],{},"荣两处上下内外人等莫不欢天喜地，独有宝玉置若罔闻。你道什么缘故？原来近日水月庵的智能私逃入城，来找秦钟，不意被秦邦业知觉，将智能逐出，将秦钟打了一顿，自己气的老病发了，三五日便呜呼哀哉了。秦钟本自怯弱，又带病未痊，受了笞杖，今见老父气死，悔痛无及，又添了许多病症。因此，宝玉心中怅怅不乐。虽有元春晋封之事，那解得他的愁闷？贾母等如何谢恩，如何回家，亲友如何来庆贺，宁荣两府近日如何热闹，众人如何得意，独他一个皆视有如无，毫不介意。因此，众人嘲他越发呆了。",[11,17844,17845],{},"English",[210,17847,17848],{},[11,17849,17850],{},"Everyone in the Rong and Ning households, both inside and outside, were extremely happy, except for Baoyu, who seemed indifferent. Do you want to know why? It turns out that recently, the nun Zhineng from Shuiyue Temple secretly ran into the city to find Qin Zhong. Unexpectedly, she was discovered by Qin Zhong's father, Qin Banger. Qin Banger not only drove Zhineng away but also gave Qin Zhong a beating. This made Qin Banger so angry that his old illness relapsed, and within three to five days, he passed away. Qin Zhong had always been weak and hadn't fully recovered from a previous illness. After being beaten and seeing his father die in anger, he was overwhelmed with regret and sorrow, which worsened his condition. As a result, Baoyu felt very melancholic. Although the promotion of Yuan Chun to imperial concubine was a joyful event, it couldn't alleviate the gloom in his heart. While Grandmother Jia and others were busy expressing their gratitude and returning home, and relatives and friends came to celebrate, and the Rong and Ning households were bustling with excitement, Baoyu alone remained completely indifferent to it all. Consequently, everyone started to mock him for becoming more and more absent-minded.",[11,17852,17853],{},"The correct answer for this question is C.",[56,17855,17857],{"id":17856},"multi-modal-rag-for-visual-reasoning","Multi-modal RAG for visual reasoning",[11,17859,17860],{},"Qwen2-VL is a new AI model that was released in late August 2024. Qwen is the name of Alibaba's AI Lab, and it is an abbreviation of the Chinese characters: 千问 (\"qian wen\", meaning 1000 questions). VL stands for vision-language, meaning that the model is capable of understanding both text and images. I had tested out the previous version of Qwen's vision-language model and was very impressed by how it could accurately describe the contents of images and also answer general questions about images.",[11,17862,17863],{},"Sun Wen was a Qing-era painter who spent 36 years of his life creating a series of 230 paintings capturing scenes from Dream of the Red Chamber. The paintings are incredibly detailed and often contain repeated figures in a temporal sequence. If you asked a Qwen-VL model to describe one of the images, it might return lengthy description that doesn't fully capture the full detail of the scene. It might also be difficult for a language model to \"focus\" on a portion of the whole image.",[11,17865,17866],{},[2718,17867],{"alt":17868,"src":17869},"Dream of the Red Chamber Painting 131","/static/redlm/painting_131.png",[11,17871,17872],{},"This sparked the idea to create a feature where users can click and drag over an image to select part of a painting, then ask questions specifically about the selected portion. I knew that while this could be achieved with tools like HTML canvas, writing it on my own would be quite time-consuming. It took me just a few minutes to write out the prompt, and Claude 3.5 Sonnet generated a perfect prototype of this feature in under a minute. Here's the prompt I used:",[210,17874,17875,17878,17881,17892,17895],{},[11,17876,17877],{},"I'm going to describe a Vue component and I want you to write it using Vue 3 to the best of your ability.",[11,17879,17880],{},"write a simple single-file vue component using Vue 3 that does the following:",[76,17882,17883,17886,17889],{},[79,17884,17885],{},"displays an image",[79,17887,17888],{},"allows the users to click and drag to select a subsection of the image",[79,17890,17891],{},"the subsection of the image is saved as a base64-encoded data url to a variable that is displayed below the image",[11,17893,17894],{},"The solution should make use of HTML canvas. When you click down on the image you begin selecting the subsection. You then move the mouse to make your subsection on the image, and when you mouse-up the subsection is selected and the data url is updated. Then the subsection is displayed at the very bottom of the page as a \"preview\" image using the base 64 image string as the image source.",[11,17896,17897],{},"The selection box should be a dashed red line",[11,17899,17900],{},[2718,17901],{"alt":17902,"src":17903},"RedLM Image Q&A","/static/redlm/image-qa.png",[11,17905,17906],{},"This shows the final result of the UI I built for the image Q&A feature in RedLM. It uses a similar chat layout that the text-based Q&A feature uses, with the addition of the image preview included in the chat log. The user query in this example just says \"Please describe the contents of the image\". This was the first image that I tested when building the image Q&A feature to see if the correct passage can be referenced based on the description of an image. This pulled the exact passage and the answer provides details about what happened (a fire broke out) where it happened (at the Gourd Temple) and why it happened (a Monk accidentally set an oil pot on fire).",[11,17908,17909],{},"Here is a diagram showing the overall flow of data in the image Q&A feature:",[11,17911,17912],{},[2718,17913],{"alt":17914,"src":17915},"Diagram of RedLM Image Q&A with RAG and Vision Language Models","/static/redlm/redlm.drawio.png",[11,17917,17918],{},"This flow chart shows how the image Q&A feature works.",[700,17920,17921,17928,17939,17942,17948],{},[79,17922,17923,17924,17927],{},"The user selects part of an image and writes a question. This data is then sent to the RedLM API as a post request to the ",[30,17925,17926],{},"/mm-q-and-a"," endpoint (multi-modal Q&A).",[79,17929,17930,17931,17934,17935,17938],{},"Vision language models are used to get a description of the image. Depending on the application configuration, this query can use models such as ",[30,17932,17933],{},"Qwen/Qwen2-VL-2B-Instruct"," on RTX PCs or using the NVIDIA API Catalog using larger models such as ",[30,17936,17937],{},"meta/llama-3.2-90b-vision-instruct",". Not all vision language models have the same interface, so I added some logic to handle different model formats.",[79,17940,17941],{},"The image description is used to fetch relevant documents from the Vector Database",[79,17943,17944,17945,17947],{},"The full prompt with the image description and relevant documents is sent to the LLM. Again, inference for this step is done either with RTX PCs or using models from the ",[30,17946,16210],{}," API catalog.",[79,17949,17950],{},"The response from the LLM is sent back to the browser and is displayed to the user as a chat message.",[11,17952,17953],{},"Here is the prompt I used for the image Q&A feature:",[459,17955,17957],{"className":13136,"code":17956,"language":12886,"meta":464,"style":464},"# Chinese prompt for image-based Q&A bot\nmm_q_and_a_prompt = PromptTemplate(\n    \"这是书中相关的内容：\\n\"\n    \"{context_str}\\n\"\n    \"---------------------\\n\"\n    \"下面是场景的描述：\\n\"\n    \"---------------------\\n\"\n    \"{image_description}\\n\"\n    \"---------------------\\n\"\n    \"根据上述的信息，尽量解释上说的场景和书的关系。\"\n)\n\n# English prompt for image-based Q&A bot\nmm_q_and_a_prompt_english = PromptTemplate(\n    \"Here is relevant content from the book:\\n\"\n    \"{context_str}\\n\"\n    \"---------------------\\n\"\n    \"Below is the description of a scene:\\n\"\n    \"---------------------\\n\"\n    \"{image_description}\\n\"\n    \"---------------------\\n\"\n    \"Based on the information provided above, try to explain the relationship between the described scene and the book content.\"\n)\n",[30,17958,17959,17964,17973,17982,17990,17998,18007,18015,18024,18032,18037,18041,18045,18050,18059,18068,18076,18084,18093,18101,18109,18117,18122],{"__ignoreMap":464},[151,17960,17961],{"class":469,"line":470},[151,17962,17963],{"class":1527},"# Chinese prompt for image-based Q&A bot\n",[151,17965,17966,17969,17971],{"class":469,"line":488},[151,17967,17968],{"class":503},"mm_q_and_a_prompt ",[151,17970,1876],{"class":1869},[151,17972,17534],{"class":503},[151,17974,17975,17978,17980],{"class":469,"line":500},[151,17976,17977],{"class":481},"    \"这是书中相关的内容：",[151,17979,8043],{"class":477},[151,17981,16406],{"class":481},[151,17983,17984,17986,17988],{"class":469,"line":509},[151,17985,17557],{"class":481},[151,17987,17560],{"class":477},[151,17989,16406],{"class":481},[151,17991,17992,17994,17996],{"class":469,"line":517},[151,17993,17548],{"class":481},[151,17995,8043],{"class":477},[151,17997,16406],{"class":481},[151,17999,18000,18003,18005],{"class":469,"line":534},[151,18001,18002],{"class":481},"    \"下面是场景的描述：",[151,18004,8043],{"class":477},[151,18006,16406],{"class":481},[151,18008,18009,18011,18013],{"class":469,"line":1413},[151,18010,17548],{"class":481},[151,18012,8043],{"class":477},[151,18014,16406],{"class":481},[151,18016,18017,18019,18022],{"class":469,"line":1418},[151,18018,17557],{"class":481},[151,18020,18021],{"class":477},"{image_description}\\n",[151,18023,16406],{"class":481},[151,18025,18026,18028,18030],{"class":469,"line":2462},[151,18027,17548],{"class":481},[151,18029,8043],{"class":477},[151,18031,16406],{"class":481},[151,18033,18034],{"class":469,"line":2471},[151,18035,18036],{"class":481},"    \"根据上述的信息，尽量解释上说的场景和书的关系。\"\n",[151,18038,18039],{"class":469,"line":2480},[151,18040,3640],{"class":503},[151,18042,18043],{"class":469,"line":2489},[151,18044,1090],{"emptyLinePlaceholder":609},[151,18046,18047],{"class":469,"line":2497},[151,18048,18049],{"class":1527},"# English prompt for image-based Q&A bot\n",[151,18051,18052,18055,18057],{"class":469,"line":3140},[151,18053,18054],{"class":503},"mm_q_and_a_prompt_english ",[151,18056,1876],{"class":1869},[151,18058,17534],{"class":503},[151,18060,18061,18064,18066],{"class":469,"line":3149},[151,18062,18063],{"class":481},"    \"Here is relevant content from the book:",[151,18065,8043],{"class":477},[151,18067,16406],{"class":481},[151,18069,18070,18072,18074],{"class":469,"line":3158},[151,18071,17557],{"class":481},[151,18073,17560],{"class":477},[151,18075,16406],{"class":481},[151,18077,18078,18080,18082],{"class":469,"line":3167},[151,18079,17548],{"class":481},[151,18081,8043],{"class":477},[151,18083,16406],{"class":481},[151,18085,18086,18089,18091],{"class":469,"line":3175},[151,18087,18088],{"class":481},"    \"Below is the description of a scene:",[151,18090,8043],{"class":477},[151,18092,16406],{"class":481},[151,18094,18095,18097,18099],{"class":469,"line":3184},[151,18096,17548],{"class":481},[151,18098,8043],{"class":477},[151,18100,16406],{"class":481},[151,18102,18103,18105,18107],{"class":469,"line":3193},[151,18104,17557],{"class":481},[151,18106,18021],{"class":477},[151,18108,16406],{"class":481},[151,18110,18111,18113,18115],{"class":469,"line":3720},[151,18112,17548],{"class":481},[151,18114,8043],{"class":477},[151,18116,16406],{"class":481},[151,18118,18119],{"class":469,"line":3729},[151,18120,18121],{"class":481},"    \"Based on the information provided above, try to explain the relationship between the described scene and the book content.\"\n",[151,18123,18124],{"class":469,"line":3735},[151,18125,3640],{"class":503},[11,18127,18128],{},"The prompt engineering for this feature was tricky. I was able to get some awesome results that would give me detailed and accurate responses, and then sometimes the LLM would seem confused about my query and tell me that there was no relationship between the scene description and the book content. Sometimes it would give me an accurate description of the scene, but then proceed to tell me that the book content is not related to the scene at all.",[11,18130,18131],{},"There is another important concept from LlamaIndex that I used to build the image Q&A feature: metadata filtering. Metadata filtering is an important concept in RAG systems  because it helps you focus your query on relevant documents in a precise way. A very simple example might be a RAG system that indexes news articles and stores the associated date as metadata. You could allow a user to set a date range for their query and only include articles that match the given date range.",[11,18133,18134],{},"For my image Q&A system, I have a mapping between the paintings and their associated chapters. When I ask a question about a painting, I want to use the description of the image to find similar paragraphs, but only the paragraphs that occur in the painting's associated chapter. What I ended up doing was filtering the entire index before making the query. The alternative would be filtering the returned nodes after making the query, but this would have the possibility of not returning any nodes.",[11,18136,18137],{},"Here's what some of the metadata filtering code looks like:",[459,18139,18141],{"className":13136,"code":18140,"language":12886,"meta":464,"style":464},"# main.py\n# filter by chapters associated with the queried image\nfilters = MetadataFilters(\n    filters=[ExactMatchFilter(key=\"chapter\", value=str(req_data.chapter))]\n)\nquery_engine = get_query_engine_for_multi_modal(filters)\n\n# rag.py\n# utility function that returns the query engine use for image Q&A\n# the index is filtered to only include nodes associated with the image being queried\ndef get_query_engine_for_multi_modal(filters):\n    retriever = index.as_retriever(filters=filters)\n    synthesizer = get_response_synthesizer(response_mode=\"compact\")\n    try:\n        query_engine = QAndAQueryEngine(\n            retriever=retriever,\n            response_synthesizer=synthesizer,\n            llm=model,\n            qa_prompt=mm_q_and_a_prompt,\n        )\n    except Exception as e:\n        print(e)\n    return query_engine\n",[30,18142,18143,18148,18153,18163,18192,18196,18206,18210,18215,18220,18225,18239,18256,18276,18283,18293,18303,18313,18323,18333,18337,18351,18359],{"__ignoreMap":464},[151,18144,18145],{"class":469,"line":470},[151,18146,18147],{"class":1527},"# main.py\n",[151,18149,18150],{"class":469,"line":488},[151,18151,18152],{"class":1527},"# filter by chapters associated with the queried image\n",[151,18154,18155,18158,18160],{"class":469,"line":500},[151,18156,18157],{"class":503},"filters ",[151,18159,1876],{"class":1869},[151,18161,18162],{"class":503}," MetadataFilters(\n",[151,18164,18165,18168,18170,18173,18176,18178,18180,18182,18185,18187,18189],{"class":469,"line":509},[151,18166,18167],{"class":15210},"    filters",[151,18169,1876],{"class":1869},[151,18171,18172],{"class":503},"[ExactMatchFilter(",[151,18174,18175],{"class":15210},"key",[151,18177,1876],{"class":1869},[151,18179,16666],{"class":481},[151,18181,106],{"class":503},[151,18183,18184],{"class":15210},"value",[151,18186,1876],{"class":1869},[151,18188,15343],{"class":6205},[151,18190,18191],{"class":503},"(req_data.chapter))]\n",[151,18193,18194],{"class":469,"line":517},[151,18195,3640],{"class":503},[151,18197,18198,18201,18203],{"class":469,"line":534},[151,18199,18200],{"class":503},"query_engine ",[151,18202,1876],{"class":1869},[151,18204,18205],{"class":503}," get_query_engine_for_multi_modal(filters)\n",[151,18207,18208],{"class":469,"line":1413},[151,18209,1090],{"emptyLinePlaceholder":609},[151,18211,18212],{"class":469,"line":1418},[151,18213,18214],{"class":1527},"# rag.py\n",[151,18216,18217],{"class":469,"line":2462},[151,18218,18219],{"class":1527},"# utility function that returns the query engine use for image Q&A\n",[151,18221,18222],{"class":469,"line":2471},[151,18223,18224],{"class":1527},"# the index is filtered to only include nodes associated with the image being queried\n",[151,18226,18227,18229,18232,18234,18237],{"class":469,"line":2480},[151,18228,16925],{"class":12347},[151,18230,18231],{"class":473}," get_query_engine_for_multi_modal",[151,18233,12386],{"class":503},[151,18235,18236],{"class":15232},"filters",[151,18238,15264],{"class":503},[151,18240,18241,18244,18246,18249,18251,18253],{"class":469,"line":2489},[151,18242,18243],{"class":503},"    retriever ",[151,18245,1876],{"class":1869},[151,18247,18248],{"class":503}," index.as_retriever(",[151,18250,18236],{"class":15210},[151,18252,1876],{"class":1869},[151,18254,18255],{"class":503},"filters)\n",[151,18257,18258,18261,18263,18266,18269,18271,18274],{"class":469,"line":2497},[151,18259,18260],{"class":503},"    synthesizer ",[151,18262,1876],{"class":1869},[151,18264,18265],{"class":503}," get_response_synthesizer(",[151,18267,18268],{"class":15210},"response_mode",[151,18270,1876],{"class":1869},[151,18272,18273],{"class":481},"\"compact\"",[151,18275,3640],{"class":503},[151,18277,18278,18281],{"class":469,"line":3140},[151,18279,18280],{"class":1869},"    try",[151,18282,14372],{"class":503},[151,18284,18285,18288,18290],{"class":469,"line":3149},[151,18286,18287],{"class":503},"        query_engine ",[151,18289,1876],{"class":1869},[151,18291,18292],{"class":503}," QAndAQueryEngine(\n",[151,18294,18295,18298,18300],{"class":469,"line":3158},[151,18296,18297],{"class":15210},"            retriever",[151,18299,1876],{"class":1869},[151,18301,18302],{"class":503},"retriever,\n",[151,18304,18305,18308,18310],{"class":469,"line":3167},[151,18306,18307],{"class":15210},"            response_synthesizer",[151,18309,1876],{"class":1869},[151,18311,18312],{"class":503},"synthesizer,\n",[151,18314,18315,18318,18320],{"class":469,"line":3175},[151,18316,18317],{"class":15210},"            llm",[151,18319,1876],{"class":1869},[151,18321,18322],{"class":503},"model,\n",[151,18324,18325,18328,18330],{"class":469,"line":3184},[151,18326,18327],{"class":15210},"            qa_prompt",[151,18329,1876],{"class":1869},[151,18331,18332],{"class":503},"mm_q_and_a_prompt,\n",[151,18334,18335],{"class":469,"line":3193},[151,18336,16824],{"class":503},[151,18338,18339,18342,18345,18348],{"class":469,"line":3720},[151,18340,18341],{"class":1869},"    except",[151,18343,18344],{"class":6205}," Exception",[151,18346,18347],{"class":1869}," as",[151,18349,18350],{"class":503}," e:\n",[151,18352,18353,18356],{"class":469,"line":3729},[151,18354,18355],{"class":2226},"        print",[151,18357,18358],{"class":503},"(e)\n",[151,18360,18361,18363],{"class":469,"line":3735},[151,18362,17496],{"class":1869},[151,18364,18365],{"class":503}," query_engine\n",[11,18367,18368],{},"This seemed to work well for my use case, but it might not be a best practice, and it might not be efficient at a bigger scale.",[736,18370,18372],{"id":18371},"multi-modal-qa-examples","Multi-modal Q&A examples",[11,18374,18375],{},"Here are some more examples of results from different types of questions from the multi-modal Q&A bot.",[11,18377,18378],{},"The response to the following query did a good job of combining information gathered from the image description and image from related passages.",[11,18380,18381,18385],{},[2718,18382],{"alt":18383,"src":18384},"Multi-modal Q&A example 2","/static/redlm/qa_example_02.png",[2718,18386],{"alt":18387,"src":18388},"Multi-modal Q&A example 3","/static/redlm/qa_example_03.png",[11,18390,18391],{},[2718,18392],{"alt":18393,"src":18394},"Q&A Example with Carriage","/static/redlm/qa_example_carriage.png",[11,18396,18397],{},[2718,18398],{"alt":18399,"src":18400},"Ou Guan Example","/static/redlm/qa_example_ou_guan.png",[11,18402,18403,18404,18409],{},"This is one of my favorite examples of the RedLM image Q&A bot in action. The query here in Chinese says: \"What are these two people doing\"? The answer incorporates a description of what is happening in the story (Jia Baoyu comes across Ou Guan while visiting a temple) and also describes the significance of what is happening (",[20,18405,18408],{"href":18406,"rel":18407},"https://en.wikipedia.org/wiki/Joss_paper",[24],"burning paper money as a form of Chinese ancestral worship","). But this answer is not perfect and it demonstrates some of the difficulties I had with the prompt engineering for this project. The first part makes a lot of sense:",[210,18411,18412],{},[11,18413,18414],{},"这段参考资料来自中国古典小说《红楼梦》，讲述了贾宝玉在探访寺庙时发现藕官在河边烧纸钱的情景。藕官是贾府中的一个戏子，她的行为可能代表了对去世亲人的哀悼或祭奥。贾宝玉询问她烧纸的原因，藕官透露了自己有私下的情感问题，并且提到了芳官和蕊官，暗示了她们之间可能存在某种秘密或情感联系。",[210,18416,18417],{},[11,18418,18419],{},"This reference is from the classical Chinese novel Dream of the Red Chamber, describing a scene where Jia Baoyu discovers Ou Guan burning paper money by the river during his visit to a temple. Ou Guan is a performer in the Jia household, and her actions might represent mourning or honoring deceased loved ones. Jia Baoyu asks her why she is burning the paper, and Ou Guan reveals that she has private emotional issues, mentioning Fang Guan and Rui Guan, hinting that there might be some kind of secret or emotional connection among them.",[11,18421,18422],{},"But the second part seems to contradict the first part of the response:",[210,18424,18425],{},[11,18426,18427],{},"相比之下，你描述的画作场景与《红楼梦》中的描述不同。画作描绘的是两个人在河边的场景，左边的人跪在地上，似乎在烧东西，右边的人站在旁边，似乎在观察或指导。根据你提供的描述，无法确定画中人物的具体行为，如可能是烧水、煮饭、烧药等。因此，这幅画可能是一幅现实主义作品，描绘了日常生活中的一幕。它可能没有像《红楼梦》中的情节那样深刻的情感或秘密背景，而是单纯地表现了人与人之间的关系和行为。此外，画中的背景、人物的表情和姿态等细节也会对画面的意义产生影响，需要结合具体的画作进行分析。",[210,18429,18430],{},[11,18431,18432],{},"In contrast, the scene you described in the painting differs from the depiction in Dream of the Red Chamber. The painting shows two people by the river: the person on the left is kneeling on the ground, seemingly burning something, while the person on the right stands nearby, possibly observing or guiding. Based on your description, it's unclear what specific action the figures are engaged in, such as possibly boiling water, cooking, or burning herbs. Therefore, this painting might be a realist work, portraying a moment from everyday life. It may not have the profound emotional or secretive background found in the storyline of Dream of the Red Chamber, instead simply highlighting the relationships and actions between individuals. Additionally, details such as the background, expressions, and postures of the figures in the painting would also influence the scene's meaning and would require analysis based on the specific artwork.",[56,18434,18436],{"id":18435},"llamaindex-developer-experience","LlamaIndex Developer Experience",[11,18438,18439],{},"Overall, I found the LlamaIndex documentation to be very helpful. Before using LlamaIndex for this project I had used LangChain to build a RAG POC, but I didn't get very good results. I love how the LlamaIndex documentation has a 5-line starter example for building a RAG system:",[459,18441,18443],{"className":13136,"code":18442,"language":12886,"meta":464,"style":464},"from llama_index.core import VectorStoreIndex, SimpleDirectoryReader\n\ndocuments = SimpleDirectoryReader(\"data\").load_data()\nindex = VectorStoreIndex.from_documents(documents)\nquery_engine = index.as_query_engine()\nresponse = query_engine.query(\"Some question about the data should go here\")\nprint(response)\n",[30,18444,18445,18456,18460,18476,18485,18494,18509],{"__ignoreMap":464},[151,18446,18447,18449,18451,18453],{"class":469,"line":470},[151,18448,16853],{"class":1869},[151,18450,16856],{"class":503},[151,18452,16859],{"class":1869},[151,18454,18455],{"class":503}," VectorStoreIndex, SimpleDirectoryReader\n",[151,18457,18458],{"class":469,"line":488},[151,18459,1090],{"emptyLinePlaceholder":609},[151,18461,18462,18465,18467,18470,18473],{"class":469,"line":500},[151,18463,18464],{"class":503},"documents ",[151,18466,1876],{"class":1869},[151,18468,18469],{"class":503}," SimpleDirectoryReader(",[151,18471,18472],{"class":481},"\"data\"",[151,18474,18475],{"class":503},").load_data()\n",[151,18477,18478,18481,18483],{"class":469,"line":509},[151,18479,18480],{"class":503},"index ",[151,18482,1876],{"class":1869},[151,18484,17259],{"class":503},[151,18486,18487,18489,18491],{"class":469,"line":517},[151,18488,18200],{"class":503},[151,18490,1876],{"class":1869},[151,18492,18493],{"class":503}," index.as_query_engine()\n",[151,18495,18496,18499,18501,18504,18507],{"class":469,"line":534},[151,18497,18498],{"class":503},"response ",[151,18500,1876],{"class":1869},[151,18502,18503],{"class":503}," query_engine.query(",[151,18505,18506],{"class":481},"\"Some question about the data should go here\"",[151,18508,3640],{"class":503},[151,18510,18511,18514],{"class":469,"line":1413},[151,18512,18513],{"class":2226},"print",[151,18515,18516],{"class":503},"(response)\n",[11,18518,16498,18519],{},[20,18520,18521],{"href":18521,"rel":18522},"https://docs.llamaindex.ai/en/stable/#getting-started",[24],[11,18524,18525,18526,18531],{},"I was able to expand this simple example to implement the text and image Q&A bots for RedLM fairly easily. The application I built is somewhat similar to the ",[20,18527,18530],{"href":18528,"rel":18529},"https://docs.llamaindex.ai/en/stable/understanding/putting_it_all_together/apps/fullstack_app_guide/",[24],"Full-Stack Web App with LLamaIndex"," included in their documentation.",[11,18533,18534,18535,18537,18538,18543,18544,18547],{},"Most of the early development I did on this project used the ",[30,18536,16508],{},". Later I tried using ",[20,18539,18542],{"href":18540,"rel":18541},"https://docs.llamaindex.ai/en/stable/module_guides/workflow/",[24],"LlamaIndex Workflows"," to better organize the logic in the text and image-based Q&A bots. The same workflow ",[30,18545,18546],{},"RAGWorkflow"," is used to handle requests for both the text and image Q&A bot queries. Workflows also work seamlessly with asynchronous Python frameworks like FastAPI. Here's the API endpoint for the multimodal image-Q&A bot using a LlamaIndex Workflow:",[459,18549,18551],{"className":13136,"code":18550,"language":12886,"meta":464,"style":464},"@app.post(\"/mm-q-and-a\")\nasync def mm_q_and_a_workflow(req_data: MultiModalRequest):\n    \"\"\"\n    This function handles Multimodal Q&A bot requests using a LlamaIndex workflow\n    \"\"\"\n    try:\n        # parse data from request object\n        image_b64 = req_data.image\n        prompt = req_data.prompt\n        chapter = req_data.chapter\n\n        # setup LlamaIndex Workflow and run it with data from request\n        w = RAGWorkflow(timeout=None)\n        result = await w.run(query=prompt, image_data=image_b64, chapter_number=chapter)\n\n        # return response\n        return QAQueryResponse(\n            response=result[\"response\"].message.content,\n            metadata=result[\"metadata\"],\n            image_desc=result[\"image_description\"],\n        )\n    except Exception as e:\n        raise HTTPException(status_code=500, detail=str(e))\n",[30,18552,18553,18565,18582,18586,18591,18595,18601,18606,18616,18626,18636,18640,18645,18664,18700,18704,18709,18716,18732,18747,18761,18765,18775],{"__ignoreMap":464},[151,18554,18555,18558,18560,18563],{"class":469,"line":470},[151,18556,18557],{"class":473},"@app.post",[151,18559,12386],{"class":503},[151,18561,18562],{"class":481},"\"/mm-q-and-a\"",[151,18564,3640],{"class":503},[151,18566,18567,18569,18571,18574,18576,18579],{"class":469,"line":488},[151,18568,15221],{"class":12347},[151,18570,15224],{"class":12347},[151,18572,18573],{"class":473}," mm_q_and_a_workflow",[151,18575,12386],{"class":503},[151,18577,18578],{"class":15232},"req_data",[151,18580,18581],{"class":503},": MultiModalRequest):\n",[151,18583,18584],{"class":469,"line":500},[151,18585,17384],{"class":481},[151,18587,18588],{"class":469,"line":509},[151,18589,18590],{"class":481},"    This function handles Multimodal Q&A bot requests using a LlamaIndex workflow\n",[151,18592,18593],{"class":469,"line":517},[151,18594,17384],{"class":481},[151,18596,18597,18599],{"class":469,"line":534},[151,18598,18280],{"class":1869},[151,18600,14372],{"class":503},[151,18602,18603],{"class":469,"line":1413},[151,18604,18605],{"class":1527},"        # parse data from request object\n",[151,18607,18608,18611,18613],{"class":469,"line":1418},[151,18609,18610],{"class":503},"        image_b64 ",[151,18612,1876],{"class":1869},[151,18614,18615],{"class":503}," req_data.image\n",[151,18617,18618,18621,18623],{"class":469,"line":2462},[151,18619,18620],{"class":503},"        prompt ",[151,18622,1876],{"class":1869},[151,18624,18625],{"class":503}," req_data.prompt\n",[151,18627,18628,18631,18633],{"class":469,"line":2471},[151,18629,18630],{"class":503},"        chapter ",[151,18632,1876],{"class":1869},[151,18634,18635],{"class":503}," req_data.chapter\n",[151,18637,18638],{"class":469,"line":2480},[151,18639,1090],{"emptyLinePlaceholder":609},[151,18641,18642],{"class":469,"line":2489},[151,18643,18644],{"class":1527},"        # setup LlamaIndex Workflow and run it with data from request\n",[151,18646,18647,18650,18652,18655,18658,18660,18662],{"class":469,"line":2497},[151,18648,18649],{"class":503},"        w ",[151,18651,1876],{"class":1869},[151,18653,18654],{"class":503}," RAGWorkflow(",[151,18656,18657],{"class":15210},"timeout",[151,18659,1876],{"class":1869},[151,18661,15437],{"class":477},[151,18663,3640],{"class":503},[151,18665,18666,18669,18671,18673,18676,18679,18681,18684,18687,18689,18692,18695,18697],{"class":469,"line":3140},[151,18667,18668],{"class":503},"        result ",[151,18670,1876],{"class":1869},[151,18672,12369],{"class":1869},[151,18674,18675],{"class":503}," w.run(",[151,18677,18678],{"class":15210},"query",[151,18680,1876],{"class":1869},[151,18682,18683],{"class":503},"prompt, ",[151,18685,18686],{"class":15210},"image_data",[151,18688,1876],{"class":1869},[151,18690,18691],{"class":503},"image_b64, ",[151,18693,18694],{"class":15210},"chapter_number",[151,18696,1876],{"class":1869},[151,18698,18699],{"class":503},"chapter)\n",[151,18701,18702],{"class":469,"line":3149},[151,18703,1090],{"emptyLinePlaceholder":609},[151,18705,18706],{"class":469,"line":3158},[151,18707,18708],{"class":1527},"        # return response\n",[151,18710,18711,18713],{"class":469,"line":3167},[151,18712,16833],{"class":1869},[151,18714,18715],{"class":503}," QAQueryResponse(\n",[151,18717,18718,18721,18723,18726,18729],{"class":469,"line":3175},[151,18719,18720],{"class":15210},"            response",[151,18722,1876],{"class":1869},[151,18724,18725],{"class":503},"result[",[151,18727,18728],{"class":481},"\"response\"",[151,18730,18731],{"class":503},"].message.content,\n",[151,18733,18734,18737,18739,18741,18744],{"class":469,"line":3184},[151,18735,18736],{"class":15210},"            metadata",[151,18738,1876],{"class":1869},[151,18740,18725],{"class":503},[151,18742,18743],{"class":481},"\"metadata\"",[151,18745,18746],{"class":503},"],\n",[151,18748,18749,18752,18754,18756,18759],{"class":469,"line":3193},[151,18750,18751],{"class":15210},"            image_desc",[151,18753,1876],{"class":1869},[151,18755,18725],{"class":503},[151,18757,18758],{"class":481},"\"image_description\"",[151,18760,18746],{"class":503},[151,18762,18763],{"class":469,"line":3720},[151,18764,16824],{"class":503},[151,18766,18767,18769,18771,18773],{"class":469,"line":3729},[151,18768,18341],{"class":1869},[151,18770,18344],{"class":6205},[151,18772,18347],{"class":1869},[151,18774,18350],{"class":503},[151,18776,18777,18780,18783,18786,18788,18790,18792,18795,18797,18799],{"class":469,"line":3735},[151,18778,18779],{"class":1869},"        raise",[151,18781,18782],{"class":503}," HTTPException(",[151,18784,18785],{"class":15210},"status_code",[151,18787,1876],{"class":1869},[151,18789,12208],{"class":477},[151,18791,106],{"class":503},[151,18793,18794],{"class":15210},"detail",[151,18796,1876],{"class":1869},[151,18798,15343],{"class":6205},[151,18800,18801],{"class":503},"(e))\n",[11,18803,18804,18805,18807],{},"Using LlamaIndex Workflows also helped me add additional logic in a maintainable and standardized way. For example, I expanded the ",[30,18806,18546],{}," logic to include LLM-based re-ranking in order to ensure retrieval of the most relevant documents for my chatbot queries. This technique increases request latency, but this is an acceptable tradeoff for an application like RedLM.",[736,18809,18811],{"id":18810},"llmrerank","LLMRerank",[11,18813,18814,18815,18817],{},"LLM Rerank was an interesting technique to try out, and LlamaIndex provides ",[30,18816,18811],{}," to make the implementation as simple as possible. Here's my understanding of how it works:",[76,18819,18820,18823,18826,18832],{},[79,18821,18822],{},"LLMRerank searches in the vector database for a high number of documents that are relevant to your query. This is done using cosine similarity, which essentially compares the vectors that represent the query and the documents.",[79,18824,18825],{},"Next, LLMRerank goes through a process of assigning a numerical score to each document to score relevancy. It does this via a special prompt that requests relevancy score for each document in batches.",[79,18827,18828,18829,18831],{},"For example, I configured ",[30,18830,18811],{}," to initially fetch 4 documents from the vector database based on cosine similarity. Then in batches of 2, relevancy scores are assigned. Finally, the top 2 most relevant documents based on the LLM-given scores are used to make the RAG query.",[79,18833,18834],{},"Adding LLMRerank can require a number of additional queries based on how you configure the batch size and the number of documents you would like to compare. This will increase latency for your application and use more resources to make the extra calls.",[11,18836,18837,18838,18840],{},"Here's an example LLM query that ",[30,18839,18811],{}," uses to do assign scores:",[11,18842,18843],{},[2718,18844],{"alt":18845,"src":18846},"LLMRerank Prompt","/static/redlm/llmrerank_prompt.png",[11,18848,18849],{},"Here are logs from my application showing what happens inside the workflow.",[11,18851,18852],{},"Application for text-base Q&A query:",[459,18854,18857],{"className":18855,"code":18856,"language":997},[995],"INFO:     💬Request for Q&A chatbot: query='宝玉和谁打架？'\nINFO:     🔀Routing Workflow to next step\nINFO:     💬Routing to QueryEvent\nINFO:     🧮Query the vector database with: 宝玉和谁打架？\nINFO:     🖥️Using in-memory embedding database\nINFO:     ⏳Loading index from storage directory...\nINFO:     ✅Finished loading index.\nINFO:     📐Retrieved 4 nodes.\nINFO:     🔀Doing LLMRerank\nINFO:     ℹ️ Chat Model Info:\nINFO:     🟩Using NVIDIA Cloud API for inference\nINFO:     🔘Chat Model: baichuan-inc/baichuan2-13b-chat\nINFO:     🔢Reranked nodes to 2\nINFO:     🤖Doing inference step\nINFO:     ⚙️ Getting query engine..\nINFO:     🔎Getting response from custom query engine\nINFO:     💬Text-based Q&A query\nINFO:     🀄Text is Chinese\nINFO:     Using nodes from workflow...\nINFO:     🔏Formatting prompt\nINFO:     Prompt is\n\n这是相关的参考资料：\n---------------------\n宝玉从来没有经历过这样的痛苦。起初，他觉得被打得很痛，乱喊乱叫。后来，他的气变得虚弱，声音变得嘶哑，无法说话。众门客见他被打得很惨，赶上来恳求他停下来。贾政不肯听，说：\"你们知道他干了什么坏事，还能饶他吗？平时都是你们这些人把他带坏了，现在到了这步田地，你们还来劝他。明天，如果他杀父弑君，你们才不劝吗？\"\n\n宝玉从来没有经历过这样的痛苦。起初，他觉得打得很痛，乱喊乱叫。后来，他的气变得虚弱，声音变得嘶哑，无法说话。众门客见他被打得很惨，赶上来恳求他停下来。贾政不肯听，说：\"你们知道他干了什么坏事，还能饶他吗？平时都是你们这些人把他带坏了，现在到了这步田地，你们还来劝他。明天，如果他杀父弑君，你们才不劝吗？\"\n---------------------\n根据上述的参考资料，回答下面的问题\n问题：宝玉和谁打架？\n\nResponse...\n宝玉和贾政打架。\n",[30,18858,18856],{"__ignoreMap":464},[11,18860,18861],{},"My question here was basically asking \"Who gets in a fight with Baoyu?\" The reply says that his father, Jiazheng, gets in a fight with Baoyu, and the documents that are used here very similar, differing by only one character. One of the documents is supposed to be and English translation, but in fact there was a failure in the translation for this paragraph and it \"translated\" the Chinese by simply repeating it. A translation of this paragraph using GPT 4o describes a tense scene between protagonist Jia Baoyu and his father Jia Zheng:",[210,18863,18864],{},[11,18865,18866],{},"Baoyu had never endured such agony before. At first, he felt the pain intensely and cried out loudly. Later, his breath grew weak, his voice turned hoarse, and he couldn't speak. The attendants, seeing how severely he was being beaten, rushed forward to plead for him to stop. Jia Zheng refused to listen, saying, \"Do you know the misdeeds he's committed, and still you want to spare him? Normally, it's you people who lead him astray, and now that it's come to this, you still try to persuade him? Tomorrow, if he were to commit patricide or treason, would you still not advise him?\"",[11,18868,18869],{},"Another benefit of LlamaIndex workflows is the ability to create visualizations of each step, the branches between them and the overall flow of events and the functions that accept/emit them as arguments/return values. It took a little bit of getting used to the patterns used to create workflows, but the documentation for Workflows provided a good starting point that I could adapt for my application. Here's a visualization of the LlamaIndex Workflow that is used by the image and text-based Q&A bots:",[11,18871,18872],{},[2718,18873],{"alt":18874,"src":18875},"RedLM RAG Workflow","/static/redlm/rag_workflow.png",[736,18877,18879],{"id":18878},"observability-and-tracing-with-langfuse","Observability and Tracing with Langfuse",[11,18881,18882,18883,18885],{},"It is never too soon to add observability and tracing to a RAG application! I learned this the hard way after doing some refactoring of prompts and ",[30,18884,16508],{}," logic.",[210,18887,18888],{},[11,18889,18890],{},"Langfuse is an open source LLM engineering platform to help teams collaboratively debug, analyze and iterate on their LLM Applications. With the Langfuse integration, you can seamlessly track and monitor performance, traces, and metrics of your LlamaIndex application. Detailed traces of the LlamaIndex context augmentation and the LLM querying processes are captured and can be inspected directly in the Langfuse UI.",[11,18892,18893,18894,18899],{},"LlamaIndex supports lots of different observability and tracing solutions. I tried using ",[20,18895,18898],{"href":18896,"rel":18897},"https://langfuse.com/",[24],"Langfuse"," (YC W23) which is an open-source option that has a self hosted option.",[11,18901,18902],{},[2718,18903],{"alt":18904,"src":18905},"Langfuse tracing for RedLM","/static/redlm/langfuse.png",[11,18907,18908],{},"Langfuse came in handy when debugging the prompts for the image-based Q&A bot. This screenshot shows a trace of a multi-modal Q&A bot query about the fire at the Gourd Temple that occurs in Chapter 1 of the book.",[56,18910,18912],{"id":18911},"nvidia-inference-stack-tensorrt-llm-and-buildnvidiacom","NVIDIA inference stack (TensorRT-LLM and build.nvidia.com)",[11,18914,18915],{},"The LLM API for TensorRT-LLM is a very nice developer experience compared with my earlier attempts with manually building inference engines. The roadmap for TensorRT-LLM looks promising, I'm looking forward to support for an OpenAI Compatible API and more models. NVIDIA NIMs using TensorRT-LLM are an easy way to run models as OpenAI compatible API servers, but the selection of models is still pretty limited. vLLM provides a strong alternative with a wide range of support models. NVIDIA NIMs for LLMs build on vLLM libraries and the TensorRT-LLM library, so it is helpful to have an understanding of both of these libraries to stay on the bleeding edge of performant inference engines.",[11,18917,18918],{},[2718,18919],{"alt":18920,"src":18921},"trt-llm-roadmap","/static/redlm/trt-llm-roadmap.png",[11,18923,18924],{},"The NVIDIA API catalog is a great way to test a variety of different models, especially large models that cannot fit into consumer hardware like RTX PCs or high-end MacBooks. I got to try out the new meta/llama-3.2-90b-vision-instruct model in my project by simply changing a value in my .env file, this is a great developer experience!",[11,18926,18927],{},[2718,18928],{"alt":16210,"src":18929},"/static/redlm/build.nvidia.com.png",[11,18931,18932],{},"The NVIDIA API catalog doesn't have every model in every size, however. For example, it has the qwen/qwen2-7b-instruct model, but doesn't have the qwen/qwen2-7b-instruct model. Also, only some of the models are labeled as \"Run Anywhere\"; a lot of the models say \"Self-Hosted API Coming Soon\" meaning that they can't be downloaded an run locally as a container. To get around this, I ran inferences services locally using both vLLM's vllm/vllm-openai container and my own container running Qwen and other services.",[56,18934,18936],{"id":18935},"my-local-inference-stack-rtx","My local inference stack (RTX)",[11,18938,18939],{},[2718,18940],{"alt":18941,"src":18942},"RTX PCs","/static/redlm/rtxpcs.png",[11,18944,18945,18946,187,18949,18952,18953,18955],{},"Two of the RTX PCs in my home network: ",[30,18947,18948],{},"a1",[30,18950,18951],{},"a3",". ",[30,18954,18948],{}," was the first PC I built by myself and was the beginning of my GeForce journey. Luckily I built it with an over-provisioned PSU, so it can use a 4090 FE card! The front panel doesn't fit, however.",[11,18957,18958],{},"One limitation of the NVIDIA API catalog is the number of free credits given for a trial account. Using 1 credit per API call, I would use up the 1000 credits very quickly when running scripts like translation or the RAG evaluation. The same would be true with rate limits of the OpenAI API. That's why running LLMs locally is still an important part of the development cycle for this type of project.",[11,18960,18961,18962,18967],{},"This project primarily uses two models: a large language model and a vision language models. Running the Yi-1.5-9B-Chat model from ",[20,18963,18966],{"href":18964,"rel":18965},"http://01.AI",[24],"01.AI"," takes up just about all of the GPU memory on one of my RTX 4090 PCs, so I had to run the vision model on another PC. In a previous project, I used Kubernetes to manage lots of different inference services: LLMs, ComfyUI, ChatTTS and MusicGen for making AI videos and I found it to a nice way to manage different containerized inference services.",[459,18969,18972],{"className":18970,"code":18971,"language":997},[995],"brian@a3:~$ microk8s kubectl get no -o wide\nNAME   STATUS   ROLES    AGE    VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIME\na1     Ready    \u003Cnone>   4d4h   v1.30.5   192.168.5.182   \u003Cnone>        Ubuntu 24.04.1 LTS   6.8.0-45-generic   containerd://1.6.28\na2     Ready    \u003Cnone>   11d    v1.30.5   192.168.5.96    \u003Cnone>        Ubuntu 24.04.1 LTS   6.8.0-45-generic   containerd://1.6.28\na3     Ready    \u003Cnone>   11d    v1.30.5   192.168.5.173   \u003Cnone>        Ubuntu 24.04.1 LTS   6.8.0-45-generic   containerd://1.6.28\n",[30,18973,18971],{"__ignoreMap":464},[11,18975,18976,18977,18980],{},"In the RedLM GitHub repo I included kubernetes manifests that show how to run the LLM and VLM across two different computers. I used Kustomize as a way to replace dynamic values in the YAML files for different resources. The kubernetes set up is experimental; the LLM and VLM can more reliably be run with ",[30,18978,18979],{},"docker run"," commands.",[11,18982,18983],{},[2718,18984],{"alt":18985,"src":18986},"k8s dashboard for local inference services","/static/redlm/k8s-dashboard.png",[11,18988,18989,18990,18992],{},"I had a lot of driver issues when trying to get kubernetes to run the vLLM container for the Yi LLM. I struggled with the following error message when trying to run the ",[30,18991,618],{}," LLM service:",[210,18994,18995],{},[11,18996,18997],{},"RuntimeError: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 804: forward compatibility was attempted on non supported HW",[11,18999,19000],{},"I tried uninstalling and reinstalling different versions of the NVIDIA drivers and CUDA but kept seeing the same message once the server would try to start up in the vLLM container logs. Rebooting my PC didn't work either. I saw a recommendation to turn off secure boot in BIOS. I didn't turn it on, but having nothing else to try I went into the BIOS settings and found that there were some keys configured in the secure boot section. After I deleted these keys and reboot, everything seemed to work normally. I'm not sure why my PC was in secure boot mode, though!",[56,19002,19004],{"id":19003},"ai-models-used-in-this-project","AI Models used in this project",[11,19006,19007],{},"I selected LLMs that run efficiently on RTX PCs, are available in the NVIDIA API catalog, and offer strong bilingual support in Chinese and English, ensuring compatibility, performance, and linguistic flexibility. Here are the models that I ended up using with RedLM:",[736,19009,19011,187,19013],{"id":19010},"_01-aiyi-15-9b-chat-and-nvidiayi-large",[30,19012,17768],{},[30,19014,19015],{},"nvidia/yi-large",[11,19017,17701,19018,19020,19021,19026,19027,19030,19031,19034],{},[30,19019,17768],{}," for most of the LLM inference while developing RedLM on my RTX PCs. ",[20,19022,19025],{"href":19023,"rel":19024},"https://github.com/01-ai/Yi",[24],"This model family"," performs well on both Chinese and English benchmarks, and has a variety of model sizes. I was able to try using the ",[30,19028,19029],{},"01-ai/yi-large"," model from the NVIDIA API catalog when using remote cloud inference. I used the ",[30,19032,19033],{},"vllm/vllm-openai:latest"," container to run this locally.",[11,19036,19037,19038,19043],{},"There are also vision models in the Yi series, such as ",[20,19039,19042],{"href":19040,"rel":19041},"https://huggingface.co/01-ai/Yi-VL-34B",[24],"01-ai/Yi-VL-34B",", but I didn't use these models in my project.",[736,19045,19047],{"id":19046},"baichuan-incbaichuan2-13b-chat",[30,19048,19049],{},"baichuan-inc/baichuan2-13b-chat",[11,19051,19052],{},"This model is available in the NVIDIA API catalog, and it was the main model I used when testing remote inference. It performs well in a variety of tasks and scores highly on the the Chinese Massive Multitask Language Understanding (CMMLU) benchmark.",[736,19054,19056],{"id":19055},"qwenqwen2-7b",[30,19057,19058],{},"Qwen/Qwen2-7B",[11,19060,19061],{},"This model was used for summary and translation of the source text. It was supported by the TensorRT-LLM LLM API and I didn't have any issues building the TensorRT-LLM model with it on the EC2 instance used to do the completion inference for translations.",[736,19063,19065],{"id":19064},"qwenqwen2-vl-2b-instruct",[30,19066,17933],{},[11,19068,19069],{},"This was the vision language model (VLM) that I used locally when developing on RTX. I was impressed at how well it could describe images given the small parameter count of the model (2 billion parameters). The small size of this model made it easy to run in my RTX PC cluster.",[11,19071,19072,19073,19078],{},"There is ",[20,19074,19077],{"href":19075,"rel":19076},"https://github.com/NVIDIA/TensorRT-LLM/issues/2183",[24],"an open GitHub issue for TensorRT-LLM support for Qwen2-VL"," at the time of writing.",[11,19080,19081,19082,19085,19086,19089],{},"I wrote a simple FastAPI server using the Hugging Face ",[30,19083,19084],{},"transformers"," library based on example code from this model's documentation (see ",[30,19087,19088],{},"services/qwen2-vl"," in the RedLM GitHub repo for more details). I packaged this service into a container in order to run it in my local kubernetes cluster along with other inference services.",[736,19091,19093],{"id":19092},"metallama-32-90b-vision-instruct",[30,19094,17937],{},[11,19096,19097,19098,19101,19102,19104],{},"This model came out while I was working on the project, and I decided to use it instead of the ",[30,19099,19100],{},"adept/fuyu-8b"," model that was previously one of the only vision language models in the NVIDIA API catalog. The ",[30,19103,17937],{}," model has strong Chinese language skills, so it was a good model to use when doing remote inference for the image Q&A bot.",[736,19106,19108],{"id":19107},"nvidianvlm-d-72b",[20,19109,19112],{"href":19110,"rel":19111},"https://huggingface.co/nvidia/NVLM-D-72B",[24],[30,19113,19114],{},"nvidia/NVLM-D-72B",[11,19116,19117,19118,19121],{},"I didn't use this model in my project, but it came out recently and looks awesome! Hopefully this model will be available on the NVIDIA API catalog soon. It is trained on the ",[30,19119,19120],{},"Qwen2-72B-Instruct"," text-only model, so it likely also has very strong support for Chinese language.",[210,19123,19124],{},[11,19125,19126],{},"Today (September 17th, 2024), we introduce NVLM 1.0, a family of frontier-class multimodal large language models (LLMs) that achieve state-of-the-art results on vision-language tasks, rivaling the leading proprietary models (e.g., GPT-4o) and open-access models (e.g., Llama 3-V 405B and InternVL 2). Remarkably, NVLM 1.0 shows improved text-only performance over its LLM backbone after multimodal training.",[56,19128,19130],{"id":19129},"the-success-of-black-myth-wukong","The success of Black Myth: Wukong",[11,19132,19133],{},"I originally got the idea to build this project after seeing the release of Black Myth: Wukong. This game is a blockbuster success from a Chinese developer that tells the story of the Monkey King's adventure in the Journey West universe. Journey West (西游记) is another one of the \"Four Great Works\" of Chinese literature. It tells the story of the legendary pilgrimage of the monk Xuanzang (also known as Tang Sanzang) to India, accompanied by his three disciples-Sun Wukong (the Monkey King), Zhu Bajie (Pigsy), and Sha Wujing (Sandy). The group travels from China to India to retrieve sacred Buddhist scriptures, facing numerous challenges, demons, and supernatural beings along the way.",[11,19135,19136],{},"The novel blends elements of adventure, mythology, and spiritual allegory, with Sun Wukong's mischievous nature and extraordinary powers adding humor and excitement. Through their journey, the characters grow and overcome personal flaws, ultimately achieving enlightenment and spiritual success. The video game adaptation has set world records for numbers of concurrent players, and it has rewritten the narrative around what is possible with single-player, offline games in the gaming industry.",[11,19138,19139],{},[2718,19140],{"alt":19141,"src":19142},"Black Myth: Wukong","/static/redlm/wukong.png",[11,19144,19145],{},"Three renditions of Journey West: Songokū (The Monkey King) polychrome woodblock (surimono) (1824) by Yashima Gakutei (1786-1868), Black Myth: Wukong video game by Game Science (2024), Journey to the West TV series by CCTV (1982-2000)",[56,19147,19149],{"id":19148},"redlm-video","RedLM video",[11,19151,19152],{},[20,19153,19155],{"href":16136,"rel":19154},[24],"Watch the RedLM video on 𝕏",[19157,19158],"red-lm-video",{},[11,19160,19161],{},"I created the video for this project using Blender.The Blender sequencer editor is a great non-linear video editing tool for simple video projects like this one. I used the following formula to create the project video for RedLM:",[700,19163,19164,19172,19175,19178],{},[79,19165,19166,19167],{},"Background music: I used the AI music generation service called Suno with the prompt \"mystical strange traditional Chinese music from the Qing Dynasty\". Here's the link to my Suno playlist called \"Qing Dynasty Music\" where you can find the original song and some other good songs that I generated using this prompt. My ",[20,19168,19171],{"href":19169,"rel":19170},"https://suno.com/playlist/863ea0dd-1921-467c-8b69-16dbd126d966",[24],"Qing Dynasty Music Playlist on Suno",[79,19173,19174],{},"Outline: For this project, the main sections are the introduction, then explaining each part with a short demo: translation, text-based Q&A, evaluation for text-based Q&A, image-based Q&A, and finally a short outro. I wrote an outline and then ChatGPT helped with filling out the content.",[79,19176,19177],{},"Narration: I used ElevenLabs to narrate the main part of the video using a clone of my voice using the ElevenLabs Voice Lab. The Chinese voices were generated on my computer with an open-source text-to-speech model called ChatTTS.",[79,19179,19180],{},"Images and videos: I gathered images and screen captures of different parts of the project including code snippets, paintings of the book, flow diagrams and screen recordings of the application.",[11,19182,19183],{},"The video is composed of different \"strips\". The green strips represent the music and voice clips. Red strips are images and yellow strips are videos. Here is what the final cut of the video looks like in Blender's Sequencer view:",[11,19185,19186],{},[2718,19187],{"alt":19188,"src":19189},"Blender Sequence Editor","/static/redlm/blender_sequence_editor.png",[11,19191,19192],{},"ChatTTS is one of the most impressive open-source models I have seen for generating conversational speech with prosodic elements (pausing, laughter, etc.) It is developed by a Chinese company called 2noise. Earlier this year I made a small contribution to this project with an API example using FastAPI to show how to run a standalone API using the model. Another example in this project provides a comprehensive example application built with gradio:",[11,19194,19195],{},[2718,19196],{"alt":19197,"src":19198},"ChatTTS UI","/static/redlm/chattts_ui.png",[11,19200,19201],{},"I was planning on streaming the narration audio for Q&A answers using my ChatTTS API service, but I didn't get around to doing this. Instead, I just used the Gradio application to generate the Chinese narration for Q&A and image Q&A examples included in the video.",[736,19203,19205],{"id":19204},"redlm-deep-dive-video-with-notebooklm","RedLM Deep Dive video with NotebookLM",[11,19207,19208],{},"NotebookLM is a new application from Google that is a truly magical application of retrieval augmented generation.",[210,19210,19211],{},[11,19212,19213],{},"NotebookLM is a research and note-taking online tool developed by Google Labs that uses artificial intelligence, specifically Google Gemini, to assist users in interacting with their documents. It can generate summaries, explanations, and answers based on content uploaded by users.",[11,19215,19216],{},"I used NotebookLM to generate a \"Deep Dive\" podcast episode using only this article. I was pretty impressed with what it was able to produce, and I wanted to share it as part of this project, so I used Blender and some Python scripts to put together a simple and engaging visualization.",[11,19218,19219],{},[2718,19220],{"alt":19221,"src":19222},"Deep Dive video in Blender","/static/redlm/deep_dive_blender.png",[11,19224,19225,19226,19229,19230,19237,19238,19243],{},"The ",[30,19227,19228],{},"openai/whisper-base"," model was used to get time stamps for the start and end of each spoken word using Automated Speech Recognition (ASR). A speaker segmentation library called ",[20,19231,19234],{"href":19232,"rel":19233},"https://github.com/pyannote/pyannote-audio",[24],[30,19235,19236],{},"pyannote/audio"," was used to perform speaker diarization. This is an interesting algorithm that can segment any number of distinct speakers in an audio recording using a series of models and a discrete-time stochastic process known as the ",[20,19239,19242],{"href":19240,"rel":19241},"https://en.wikipedia.org/wiki/Chinese_restaurant_process",[24],"Chinese restaurant process",". This gave a list of time intervals with a speaker ID, and I used the intervals to attribute a speaker ID to each word. Then I segmented the audio into two files using this data and used the files to generate audio waveforms using Blender's geometry nodes. Another script was used to animate each word of as it is spoken in one of two positions for each speaker.",[56,19245,19246],{"id":11724},"Final thoughts",[11,19248,19249],{},"I'm glad to have had the opportunity to join three NVIDIA developer contests this year. I like the idea of a \"developer contest\" that takes place over several weeks compared to hackathons that take place over just a few days. Having more time allows you to learn about a new tool or framework at a deeper level and think about how to apply it in a creative project.",[11,19251,19252],{},[2718,19253],{"alt":19254,"src":19255},"NVIDIA and LlamaIndex Contest","/static/redlm/llama-contest-og.jpg",[11,19257,19258],{},"I also like how this contest is not team based. Working on this project I was able to do a lot of high-level thinking, write out features as detailed prompts, and then delegate the code writing to LLMs as if I was giving tasks to teammates.",[11,19260,19261],{},"NVIDIA's contests are \"global developer contests\", but the contests so far are not open to developers in India and China. This is probably due to local rules and regulations governing how contests, prizes and taxes work. It is too bad; I would love to see what types of applications would come from participants in these countries. Also, there are also a lot of really interesting developments happening in the LLM space in both China and India!",[11,19263,19264,19265,19273],{},"The LLMs I used in this project were developed by leading Chinese AI companies, and they are competitive with LLMs from Western countries on LLM benchmarks despite having access to fewer GPU resources. ",[20,19266,19269,19270],{"href":19267,"rel":19268},"https://qwenlm.github.io/blog/qwen2.5-coder-family/",[24],"Qwen recently released a new model called ",[30,19271,19272],{},"Qwen2.5-Coder-32B"," that has outperfomed leading models at coding tasks.",[11,19275,19276],{},[2718,19277],{"alt":19278,"src":19279},"Qwen coder model","/static/redlm/qwen_coder.png",[11,19281,19282,19287],{},[20,19283,19286],{"href":19284,"rel":19285},"https://www.youtube.com/watch?v=UitJxc9LE60",[24],"Kaifu Lee mentioned in a Bloomberg interview"," that the scarcity of GPU resources in China will force Chinese engineers to innovate in new ways to gain an advantage. One example of this we saw recently was when Chinese hardware hackers doubled the usable memory of the RTX 4090D (a variant of the RTX 4090 card with lower processing power to comply with US export regulations for China - the D stands for Dragon, apparently!)",[11,19289,19290],{},[2718,19291],{"alt":19292,"src":19293},"RTX 4090D 48GB","/static/redlm/RTX4090D.jpg",[11,19295,19296],{},"NVIDIA recently concluded its AI Summit in Mumbai. I was intrigued by the fact that Hindi has unique challenges that have have limited the development of Hindi LLMs compared to the development of English and Chinese LLMs. In a conversation with Jensen Huang, Indian industrial titan and CEO of Reliance Industries Mukesh Ambani spoke about his aspirations and ambition for India to overcome these challenges and develop a Hindi LLM. In a viral moment Mukesh Ambani shared that through devotion to attaining knowledge through the Hindu Goddess of knowledge Sarawati, India will be met by the Goddess of prosperity, Lakshmi.",[11,19298,19299],{},[2718,19300],{"alt":19301,"src":19302},"Mukesh Ambani","/static/redlm/mukesh_ambani.png",[11,19304,19305,19306,19311],{},"NVIDIA recently released a small language model for Hindi at the AI Summit in Mumbai called  ",[20,19307,19310],{"href":19308,"rel":19309},"https://indiaai.gov.in/article/nvidia-unveils-nemotron-4-mini-hindi-4b-ai-for-india-s-500-million-hindi-speakers",[24],"Nemotron-4-Mini-Hindi-4B",". Hindi LLMs could enable applications to explore important works of literature from India. I don't know that much about India literature, but a comparable work of literature in size and cultural significance might be the Ramayana.",[11,19313,19314],{},[51,19315,19316],{},"The Ramayana is an ancient Indian epic that tells the story of Prince Rama's heroic quest to rescue his wife, Sita, who has been kidnapped by the demon king Ravana. Set in a world of gods, demons, and celestial beings, the story explores themes of duty, loyalty, and the triumph of good over evil. Guided by wisdom, strength, and the support of devoted allies like Hanuman, the monkey god, and his brother Lakshmana, Rama's journey is a deeply spiritual tale, celebrated for its poetic beauty and moral depth. The Ramayana continues to inspire and captivate audiences across cultures.",[11,19318,19319],{},"The Ramayana story journeyed to Thailand centuries ago, transforming into the Ramakien, a Thai adaptation that retains the essence of the original Indian epic while adding distinctive Thai cultural elements. Introduced through trade, diplomacy, and cultural exchange between India and Southeast Asia, the story became deeply woven into Thailand's art, literature, and performance traditions. Thai kings, particularly King Rama I, adapted and documented the Ramakien, giving it a prominent place in Thai history. Lavishly detailed murals surrounding the Temple of the Emerald Buddha in Bangkok's Grand Palace depict the Ramakien in over 178 panels that totaling over 2 kilometers in length. On a recent visit to the Grand Palace, I imagined having an application that could link the detailed murals to elements of the story in Hindi, Thai, English, Chinese or any language.",[11,19321,19322],{},[2718,19323],{"alt":19324,"src":19325},"Ramakien murals surrounding Temple of the Emerald Buddha","/static/redlm/ramakien.png",[11,19327,19328],{},"The Dream of the Red Chamber, originally titled The Story of the Stone, is one of China's greatest literary works and a masterpiece of world literature. The novel begins with a frame story centered on a magical stone, left over from the Chinese creation myth where the goddess Nuwa mends the heavens. Longing to experience the human world, the sentient stone persuades a Buddhist monk and a Taoist priest to reincarnate it as a boy. This boy, Baoyu, is born into a wealthy and influential family-a character partly based on the author, Cao Xueqin, and his own aristocratic upbringing. Through Baoyu's life, friendships, and romantic relationships, the novel delves into his family's gradual decline, mirroring the instability of China's own noble families in the late Qing dynasty. The story also portrays the era's customs, social structures, and beliefs, offering readers a richly detailed exploration of life in Qing China.",[11,19330,19331],{},"It was a lot of fun to work on this project with tools from LlamaIndex and NVIDIA. With AI technology, GPUs are now essentially sentient stones, and I was able to share this important touchstone of the human experience with my computers using LlamaIndex and open source language models. In turn, RedLM shared with me delightful insights into world of Dream of the Red Chamber.",[11,19333,19334],{},[2718,19335],{"alt":19336,"src":19337},"Story of a Stone","/static/redlm/stone_story.png",[11,19339,19340],{},[2718,19341],{"alt":19342,"src":19343},"Story of a Stone Analysis","/static/redlm/stone_story_analysis.png",[210,19345,19346],{},[11,19347,19348],{},"This scene describes a piece of traditional Chinese painting, depicting two elderly figures conversing amidst mountains and rivers. The painting likely visually represents the scene from the book where a monk and a Taoist are chatting at the foot of Qinggeng Peak. The two elderly figures in the painting may represent the monk and Taoist from the book, discussing their discovery of a bright and pristine stone, and planning to take it to a bustling, splendid place for a happy life. The painting's elements-mountains, peaks, flowing water, trees, and rocks-might echo the book's descriptions, illustrating the natural environment at the base of Qinggeng Peak where the monk and Taoist reside. The painting's tranquil and harmonious atmosphere may also align with the storyline, expressing the monk and Taoist's care for the stone and their wish for it to live a happy life. In summary, this painted scene might be an artistic portrayal of the story between the monk, the Taoist, and the stone from the book, using visual elements and ambiance to convey the narrative and themes within the story.",[589,19350,19351],{},"html pre.shiki code .sMOD_, html code.shiki .sMOD_{--shiki-default:#24292E;--shiki-dark:#E1E4E8;--shiki-sepia:#F8F8F2}html pre.shiki code .sC2Qs, html code.shiki .sC2Qs{--shiki-default:#D73A49;--shiki-dark:#F97583;--shiki-sepia:#F92672}html pre.shiki code .sTHNf, html code.shiki .sTHNf{--shiki-default:#E36209;--shiki-default-font-style:inherit;--shiki-dark:#FFAB70;--shiki-dark-font-style:inherit;--shiki-sepia:#FD971F;--shiki-sepia-font-style:italic}html pre.shiki code .s7F3e, html code.shiki .s7F3e{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#AE81FF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html .sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html.sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html pre.shiki code .s8-w5, html code.shiki .s8-w5{--shiki-default:#6A737D;--shiki-dark:#6A737D;--shiki-sepia:#88846F}html pre.shiki code .sq6CD, html code.shiki .sq6CD{--shiki-default:#D73A49;--shiki-default-font-style:inherit;--shiki-dark:#F97583;--shiki-dark-font-style:inherit;--shiki-sepia:#66D9EF;--shiki-sepia-font-style:italic}html pre.shiki code .sstjo, html code.shiki .sstjo{--shiki-default:#032F62;--shiki-dark:#9ECBFF;--shiki-sepia:#E6DB74}html pre.shiki code .sz2Vg, html code.shiki .sz2Vg{--shiki-default:#6F42C1;--shiki-default-text-decoration:inherit;--shiki-dark:#B392F0;--shiki-dark-text-decoration:inherit;--shiki-sepia:#A6E22E;--shiki-sepia-text-decoration:underline}html pre.shiki code .s30JN, html code.shiki .s30JN{--shiki-default:#6F42C1;--shiki-default-font-style:inherit;--shiki-default-text-decoration:inherit;--shiki-dark:#B392F0;--shiki-dark-font-style:inherit;--shiki-dark-text-decoration:inherit;--shiki-sepia:#A6E22E;--shiki-sepia-font-style:italic;--shiki-sepia-text-decoration:underline}html pre.shiki code .srTi1, html code.shiki .srTi1{--shiki-default:#6F42C1;--shiki-dark:#B392F0;--shiki-sepia:#A6E22E}html pre.shiki code .so59x, html code.shiki .so59x{--shiki-default:#24292E;--shiki-default-font-style:inherit;--shiki-dark:#E1E4E8;--shiki-dark-font-style:inherit;--shiki-sepia:#FD971F;--shiki-sepia-font-style:italic}html pre.shiki code .s-m8C, html code.shiki .s-m8C{--shiki-default:#005CC5;--shiki-default-font-style:inherit;--shiki-dark:#79B8FF;--shiki-dark-font-style:inherit;--shiki-sepia:#66D9EF;--shiki-sepia-font-style:italic}html pre.shiki code .sP7S_, html code.shiki .sP7S_{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#FD971F}html pre.shiki code .sTrkL, html code.shiki .sTrkL{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#66D9EF}html pre.shiki code .sXSQT, html code.shiki .sXSQT{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#F8F8F2}",{"title":464,"searchDepth":488,"depth":488,"links":19353},[19354,19357,19358,19359,19360,19361,19366,19367,19370,19374,19375,19376,19385,19386,19389],{"id":16115,"depth":488,"text":16116,"children":19355},[19356],{"id":16125,"depth":500,"text":16126},{"id":16147,"depth":488,"text":16148},{"id":16174,"depth":488,"text":16175},{"id":16197,"depth":488,"text":16198},{"id":16214,"depth":488,"text":16215},{"id":16464,"depth":488,"text":16465,"children":19362},[19363,19364,19365],{"id":16839,"depth":500,"text":16840},{"id":17308,"depth":500,"text":17309},{"id":17347,"depth":500,"text":17348},{"id":17694,"depth":488,"text":17695},{"id":17856,"depth":488,"text":17857,"children":19368},[19369],{"id":18371,"depth":500,"text":18372},{"id":18435,"depth":488,"text":18436,"children":19371},[19372,19373],{"id":18810,"depth":500,"text":18811},{"id":18878,"depth":500,"text":18879},{"id":18911,"depth":488,"text":18912},{"id":18935,"depth":488,"text":18936},{"id":19003,"depth":488,"text":19004,"children":19377},[19378,19380,19381,19382,19383,19384],{"id":19010,"depth":500,"text":19379},"01-ai/Yi-1.5-9B-Chat and nvidia/yi-large",{"id":19046,"depth":500,"text":19049},{"id":19055,"depth":500,"text":19058},{"id":19064,"depth":500,"text":17933},{"id":19092,"depth":500,"text":17937},{"id":19107,"depth":500,"text":19114},{"id":19129,"depth":488,"text":19130},{"id":19148,"depth":488,"text":19149,"children":19387},[19388],{"id":19204,"depth":500,"text":19205},{"id":11724,"depth":488,"text":19246},"2024-11-09","RedLM is an AI-powered application for the study of China's greatest classical novel: Dream of the Red Chamber",[19393,19394],{"link":16136,"site":11126},{"link":19395,"site":10715},"https://dev.to/briancaffey/redlm-my-submission-for-the-nvidia-and-llamaindex-developer-contest-1c3k",{},"/2024/10/09/redlm-ai-application-for-studying-chinese-literature-redology-nvidia-llama-index-developer-contest",{"title":16110,"description":19391},"2024/10/09/redlm-ai-application-for-studying-chinese-literature-redology-nvidia-llama-index-developer-contest",[11133,19401,614,615,19402,19403,19404,19405],"llama-index","rag","tensorrt-llm","chinese","redology","tYSvdyki4D4olE0MPsiZ-TdIr1YPsV8_q6sDVyQ3L8g",{"id":19408,"title":19409,"body":19410,"comments":609,"date":21289,"description":21290,"draft":602,"extension":605,"external":606,"image":21291,"meta":21292,"navigation":609,"path":21293,"seo":21294,"stem":21295,"tags":21296,"__hash__":21299},"blog/2024/08/11/upgrading-my-github-pages-blog-to-nuxt-3.md","Upgrading my GitHub Pages blog to Nuxt 3",{"type":8,"value":19411,"toc":21258},[19412,19416,19421,19431,19434,19437,19443,19447,19451,19457,19611,19615,19618,19786,19797,19803,19809,19971,19982,20138,20165,20174,20332,20335,20339,20350,20364,20370,20374,20382,20388,20391,20395,20401,20403,20406,20712,20727,20730,20736,20742,20764,20769,20773,20777,20789,20792,20795,20851,20854,20858,20861,20913,20917,20927,20934,20937,20941,20944,20950,20977,20981,20984,20988,20991,20995,21006,21008,21011,21013,21016,21020,21028,21032,21035,21039,21043,21055,21061,21180,21184,21187,21196,21200,21211,21238,21241,21245,21252,21255],[56,19413,19415],{"id":19414},"blog-history","Blog history",[11,19417,19418,19420],{},[15,19419,11825],{}," This article was published in August 2024 (~7 months old). The Nuxt versions referenced (Nuxt 3.12.4, Nitro 2.9.7) may have updated since publication. Check official documentation for the latest releases and breaking changes.",[11,19422,19423,19424,19430],{},"My personal website has always lived on GitHub Pages at ",[20,19425,19428],{"href":19426,"rel":19427},"https://briancaffey.github.io",[24],[30,19429,662],{},". The first version was built with the Jekyll framework. I started learning about Vue, and Nuxt seemed like an interesting alternative to Jekyll that would allow me to practice frontend development. In September 2020 I deployed the first version of the new site using Nuxt 2 and Vue 2.",[11,19432,19433],{},"I recently went through the process of upgrading from Nuxt 2 to Nuxt 3. This upgrade path also included an upgrade from Vue 2 to Vue 3. My previous attempts to upgrade this site from Nuxt 2 to Nuxt 3 failed because of error messages that I couldn't work through. This time, with a big help from AI, I got through the entire upgrade and learned a lot in the process. I'm happy to share my new blog that is powered by Vue 3, Nuxt 3 and Nuxt Content v2!",[11,19435,19436],{},"This article will go over the features of my site, how I'm using Nuxt and Vue and some of the changes I had to make when doing the upgrade. Let's go!",[11,19438,19439],{},[2718,19440],{"alt":19441,"src":19442},"New site powered by Nuxt 3, Tailwind and Pinia","/static/nuxt/nuxt3.png",[56,19444,19446],{"id":19445},"features-of-my-blog","Features of my blog",[736,19448,19450],{"id":19449},"modules-and-plugins-overview","Modules and plugins overview",[11,19452,19453,19454,208],{},"Here are the modules and plugins I use on my site defined in ",[30,19455,19456],{},"nuxt.config.js",[459,19458,19462],{"className":19459,"code":19460,"language":19461,"meta":464,"style":464},"language-javascript shiki shiki-themes github-light github-dark monokai","export default defineNuxtConfig({\n  modules: [\n    '@nuxt/content',\n    '@nuxtjs/tailwindcss',\n    '@nuxtjs/i18n',\n    '@nuxtjs/color-mode',\n    '@pinia/nuxt',\n    '@nuxt/eslint',\n    '@nuxtjs/sitemap',\n    '@nuxt/image',\n    'nuxt-gtag',\n    // '@nuxtjs/feed', --> this module is not yet supported in Nuxt 3!\n  ],\n\n  plugins: [\n    '~/plugins/disqus',\n    { src: '~/plugins/apexcharts', mode: 'client' },\n    { src: '~/plugins/drift', mode: 'client' }\n  ]\n})\n","javascript",[30,19463,19464,19477,19482,19489,19496,19503,19510,19517,19524,19531,19538,19545,19550,19554,19558,19563,19570,19587,19601,19606],{"__ignoreMap":464},[151,19465,19466,19468,19471,19474],{"class":469,"line":470},[151,19467,1870],{"class":1869},[151,19469,19470],{"class":1869}," default",[151,19472,19473],{"class":473}," defineNuxtConfig",[151,19475,19476],{"class":503},"({\n",[151,19478,19479],{"class":469,"line":488},[151,19480,19481],{"class":503},"  modules: [\n",[151,19483,19484,19487],{"class":469,"line":500},[151,19485,19486],{"class":481},"    '@nuxt/content'",[151,19488,9417],{"class":503},[151,19490,19491,19494],{"class":469,"line":509},[151,19492,19493],{"class":481},"    '@nuxtjs/tailwindcss'",[151,19495,9417],{"class":503},[151,19497,19498,19501],{"class":469,"line":517},[151,19499,19500],{"class":481},"    '@nuxtjs/i18n'",[151,19502,9417],{"class":503},[151,19504,19505,19508],{"class":469,"line":534},[151,19506,19507],{"class":481},"    '@nuxtjs/color-mode'",[151,19509,9417],{"class":503},[151,19511,19512,19515],{"class":469,"line":1413},[151,19513,19514],{"class":481},"    '@pinia/nuxt'",[151,19516,9417],{"class":503},[151,19518,19519,19522],{"class":469,"line":1418},[151,19520,19521],{"class":481},"    '@nuxt/eslint'",[151,19523,9417],{"class":503},[151,19525,19526,19529],{"class":469,"line":2462},[151,19527,19528],{"class":481},"    '@nuxtjs/sitemap'",[151,19530,9417],{"class":503},[151,19532,19533,19536],{"class":469,"line":2471},[151,19534,19535],{"class":481},"    '@nuxt/image'",[151,19537,9417],{"class":503},[151,19539,19540,19543],{"class":469,"line":2480},[151,19541,19542],{"class":481},"    'nuxt-gtag'",[151,19544,9417],{"class":503},[151,19546,19547],{"class":469,"line":2489},[151,19548,19549],{"class":1527},"    // '@nuxtjs/feed', --> this module is not yet supported in Nuxt 3!\n",[151,19551,19552],{"class":469,"line":2497},[151,19553,9466],{"class":503},[151,19555,19556],{"class":469,"line":3140},[151,19557,1090],{"emptyLinePlaceholder":609},[151,19559,19560],{"class":469,"line":3149},[151,19561,19562],{"class":503},"  plugins: [\n",[151,19564,19565,19568],{"class":469,"line":3158},[151,19566,19567],{"class":481},"    '~/plugins/disqus'",[151,19569,9417],{"class":503},[151,19571,19572,19575,19578,19581,19584],{"class":469,"line":3167},[151,19573,19574],{"class":503},"    { src: ",[151,19576,19577],{"class":481},"'~/plugins/apexcharts'",[151,19579,19580],{"class":503},", mode: ",[151,19582,19583],{"class":481},"'client'",[151,19585,19586],{"class":503}," },\n",[151,19588,19589,19591,19594,19596,19598],{"class":469,"line":3175},[151,19590,19574],{"class":503},[151,19592,19593],{"class":481},"'~/plugins/drift'",[151,19595,19580],{"class":503},[151,19597,19583],{"class":481},[151,19599,19600],{"class":503}," }\n",[151,19602,19603],{"class":469,"line":3184},[151,19604,19605],{"class":503},"  ]\n",[151,19607,19608],{"class":469,"line":3193},[151,19609,19610],{"class":503},"})\n",[736,19612,19614],{"id":19613},"nuxt-content","Nuxt Content",[11,19616,19617],{},"The Nuxt Content module is a powerful git-based CMS. Articles on my site are written in Markdown files that contains frontmatter like the following:",[459,19619,19621],{"className":14359,"code":19620,"language":14361,"meta":464,"style":464},"---\ntitle: \"Upgrading my GitHub Pages blog to Nuxt 3\" # used on the page and in the \u003Chead> metadata\ndate: '2024-08-11'\ndescription: \"An overview of my newly upgraded GitHub Pages blog powered by Nuxt 3\"\nimage: /static/nuxt/nuxt3.png # cover image and og:image + twitter:image\ntags: # tags are used to categorize and navigate content\n  - vue\n  - nuxt\n  - github\n  - pinia\n\ndraft: true # drafts are publicly available but not displayed in the list of blog articles and not indexed\n\nexternal: # a list of external links where the article has been shared or republished\n  - link: https://x.com/briancaffey/status/abc123\n    site: x\n\ncomments: true # shows disqus comments\n---\n",[30,19622,19623,19629,19642,19652,19662,19674,19684,19692,19699,19706,19713,19717,19730,19734,19744,19756,19766,19770,19782],{"__ignoreMap":464},[151,19624,19625],{"class":469,"line":470},[151,19626,19628],{"class":19627},"s_OQ2","---\n",[151,19630,19631,19634,19636,19639],{"class":469,"line":488},[151,19632,19633],{"class":14368},"title",[151,19635,6208],{"class":503},[151,19637,19638],{"class":481},"\"Upgrading my GitHub Pages blog to Nuxt 3\"",[151,19640,19641],{"class":1527}," # used on the page and in the \u003Chead> metadata\n",[151,19643,19644,19647,19649],{"class":469,"line":500},[151,19645,19646],{"class":14368},"date",[151,19648,6208],{"class":503},[151,19650,19651],{"class":481},"'2024-08-11'\n",[151,19653,19654,19657,19659],{"class":469,"line":509},[151,19655,19656],{"class":14368},"description",[151,19658,6208],{"class":503},[151,19660,19661],{"class":481},"\"An overview of my newly upgraded GitHub Pages blog powered by Nuxt 3\"\n",[151,19663,19664,19667,19669,19671],{"class":469,"line":517},[151,19665,19666],{"class":14368},"image",[151,19668,6208],{"class":503},[151,19670,19442],{"class":481},[151,19672,19673],{"class":1527}," # cover image and og:image + twitter:image\n",[151,19675,19676,19679,19681],{"class":469,"line":534},[151,19677,19678],{"class":14368},"tags",[151,19680,6208],{"class":503},[151,19682,19683],{"class":1527},"# tags are used to categorize and navigate content\n",[151,19685,19686,19689],{"class":469,"line":1413},[151,19687,19688],{"class":503},"  - ",[151,19690,19691],{"class":481},"vue\n",[151,19693,19694,19696],{"class":469,"line":1418},[151,19695,19688],{"class":503},[151,19697,19698],{"class":481},"nuxt\n",[151,19700,19701,19703],{"class":469,"line":2462},[151,19702,19688],{"class":503},[151,19704,19705],{"class":481},"github\n",[151,19707,19708,19710],{"class":469,"line":2471},[151,19709,19688],{"class":503},[151,19711,19712],{"class":481},"pinia\n",[151,19714,19715],{"class":469,"line":2480},[151,19716,1090],{"emptyLinePlaceholder":609},[151,19718,19719,19722,19724,19727],{"class":469,"line":2489},[151,19720,19721],{"class":14368},"draft",[151,19723,6208],{"class":503},[151,19725,19726],{"class":477},"true",[151,19728,19729],{"class":1527}," # drafts are publicly available but not displayed in the list of blog articles and not indexed\n",[151,19731,19732],{"class":469,"line":2497},[151,19733,1090],{"emptyLinePlaceholder":609},[151,19735,19736,19739,19741],{"class":469,"line":3140},[151,19737,19738],{"class":14368},"external",[151,19740,6208],{"class":503},[151,19742,19743],{"class":1527},"# a list of external links where the article has been shared or republished\n",[151,19745,19746,19748,19751,19753],{"class":469,"line":3149},[151,19747,19688],{"class":503},[151,19749,19750],{"class":14368},"link",[151,19752,6208],{"class":503},[151,19754,19755],{"class":481},"https://x.com/briancaffey/status/abc123\n",[151,19757,19758,19761,19763],{"class":469,"line":3158},[151,19759,19760],{"class":14368},"    site",[151,19762,6208],{"class":503},[151,19764,19765],{"class":481},"x\n",[151,19767,19768],{"class":469,"line":3167},[151,19769,1090],{"emptyLinePlaceholder":609},[151,19771,19772,19775,19777,19779],{"class":469,"line":3175},[151,19773,19774],{"class":14368},"comments",[151,19776,6208],{"class":503},[151,19778,19726],{"class":477},[151,19780,19781],{"class":1527}," # shows disqus comments\n",[151,19783,19784],{"class":469,"line":3184},[151,19785,19628],{"class":19627},[11,19787,19788,19789,19792,19793,19796],{},"Files for articles are stored in ",[30,19790,19791],{},"/content/[year]/[month]/[day]/[slug].md",", and the URLs for the articles are ",[30,19794,19795],{},"/${year}/${month}/${day}/${slug}",". This URL scheme was used in the Jekyll blog on my GitHub Pages site and kept this URL structure when I switched to Nuxt.",[11,19798,19799,19802],{},[30,19800,19801],{},"contentQuery"," is used for getting content from Nuxt Content. Here's a comparison of the old and new way of fetching data from Nuxt content.",[11,19804,19805,19806,208],{},"Old way using ",[30,19807,19808],{},"asyncData",[459,19810,19814],{"className":19811,"code":19812,"language":19813,"meta":464,"style":464},"language-html shiki shiki-themes github-light github-dark monokai","\u003Cscript>\nexport default {\n  async asyncData ({ $content, params }) {\n    const file = `${params.year}/${params.month}/${params.day}/${params.slug}`\n    const article = await $content(file).fetch()\n    return { article }\n  }\n}\n\u003C/script>\n","html",[30,19815,19816,19825,19834,19856,19924,19946,19953,19958,19962],{"__ignoreMap":464},[151,19817,19818,19820,19823],{"class":469,"line":470},[151,19819,3613],{"class":503},[151,19821,19822],{"class":14368},"script",[151,19824,3742],{"class":503},[151,19826,19827,19829,19831],{"class":469,"line":488},[151,19828,1870],{"class":1869},[151,19830,19470],{"class":1869},[151,19832,19833],{"class":503}," {\n",[151,19835,19836,19839,19842,19845,19848,19850,19853],{"class":469,"line":500},[151,19837,19838],{"class":1869},"  async",[151,19840,19841],{"class":473}," asyncData",[151,19843,19844],{"class":503}," ({ ",[151,19846,19847],{"class":15210},"$content",[151,19849,106],{"class":503},[151,19851,19852],{"class":15210},"params",[151,19854,19855],{"class":503}," }) {\n",[151,19857,19858,19861,19863,19866,19868,19872,19874,19876,19879,19881,19884,19886,19888,19890,19893,19895,19897,19899,19901,19903,19906,19908,19910,19912,19914,19916,19919,19921],{"class":469,"line":509},[151,19859,19860],{"class":12347},"    const",[151,19862,4231],{"class":12360},[151,19864,19865],{"class":1869}," =",[151,19867,2218],{"class":481},[151,19869,19871],{"class":19870},"s1EfO","${",[151,19873,19852],{"class":503},[151,19875,643],{"class":4828},[151,19877,19878],{"class":503},"year",[151,19880,2001],{"class":19870},[151,19882,19883],{"class":481},"/",[151,19885,19871],{"class":19870},[151,19887,19852],{"class":503},[151,19889,643],{"class":4828},[151,19891,19892],{"class":503},"month",[151,19894,2001],{"class":19870},[151,19896,19883],{"class":481},[151,19898,19871],{"class":19870},[151,19900,19852],{"class":503},[151,19902,643],{"class":4828},[151,19904,19905],{"class":503},"day",[151,19907,2001],{"class":19870},[151,19909,19883],{"class":481},[151,19911,19871],{"class":19870},[151,19913,19852],{"class":503},[151,19915,643],{"class":4828},[151,19917,19918],{"class":503},"slug",[151,19920,2001],{"class":19870},[151,19922,19923],{"class":481},"`\n",[151,19925,19926,19928,19931,19933,19935,19938,19941,19944],{"class":469,"line":517},[151,19927,19860],{"class":12347},[151,19929,19930],{"class":12360}," article",[151,19932,19865],{"class":1869},[151,19934,12369],{"class":1869},[151,19936,19937],{"class":473}," $content",[151,19939,19940],{"class":503},"(file).",[151,19942,19943],{"class":473},"fetch",[151,19945,12461],{"class":503},[151,19947,19948,19950],{"class":469,"line":534},[151,19949,17496],{"class":1869},[151,19951,19952],{"class":503}," { article }\n",[151,19954,19955],{"class":469,"line":1413},[151,19956,19957],{"class":503},"  }\n",[151,19959,19960],{"class":469,"line":1418},[151,19961,6274],{"class":503},[151,19963,19964,19967,19969],{"class":469,"line":2462},[151,19965,19966],{"class":503},"\u003C/",[151,19968,19822],{"class":14368},[151,19970,3742],{"class":503},[11,19972,19973,19974,19977,19978,19981],{},"Here's the new way of fetching content using ",[30,19975,19976],{},"useAsyncData"," with ",[30,19979,19980],{},"\u003Cscript setup>"," syntax:",[459,19983,19985],{"className":19811,"code":19984,"language":19813,"meta":464,"style":464},"\u003Cscript setup>\nconst route = useRoute();\nconst { year, month, day, slug } = route.params;\nconst page = `/${year}/${month}/${day}/${slug}`;\nconst { data: article } = await useAsyncData(route.params.slug, () =>\n  queryCollection(page).findOne()\n);\n\u003C/script>\n",[30,19986,19987,19998,20013,20040,20087,20113,20125,20130],{"__ignoreMap":464},[151,19988,19989,19991,19993,19996],{"class":469,"line":470},[151,19990,3613],{"class":503},[151,19992,19822],{"class":14368},[151,19994,19995],{"class":473}," setup",[151,19997,3742],{"class":503},[151,19999,20000,20002,20005,20007,20010],{"class":469,"line":488},[151,20001,12348],{"class":12347},[151,20003,20004],{"class":12360}," route",[151,20006,19865],{"class":1869},[151,20008,20009],{"class":473}," useRoute",[151,20011,20012],{"class":503},"();\n",[151,20014,20015,20017,20019,20021,20023,20025,20027,20029,20031,20033,20035,20037],{"class":469,"line":500},[151,20016,12348],{"class":12347},[151,20018,12351],{"class":503},[151,20020,19878],{"class":12360},[151,20022,106],{"class":503},[151,20024,19892],{"class":12360},[151,20026,106],{"class":503},[151,20028,19905],{"class":12360},[151,20030,106],{"class":503},[151,20032,19918],{"class":12360},[151,20034,12364],{"class":503},[151,20036,1876],{"class":1869},[151,20038,20039],{"class":503}," route.params;\n",[151,20041,20042,20044,20047,20049,20052,20054,20056,20058,20060,20062,20064,20066,20068,20070,20072,20074,20076,20078,20080,20082,20084],{"class":469,"line":509},[151,20043,12348],{"class":12347},[151,20045,20046],{"class":12360}," page",[151,20048,19865],{"class":1869},[151,20050,20051],{"class":481}," `/",[151,20053,19871],{"class":19870},[151,20055,19878],{"class":503},[151,20057,2001],{"class":19870},[151,20059,19883],{"class":481},[151,20061,19871],{"class":19870},[151,20063,19892],{"class":503},[151,20065,2001],{"class":19870},[151,20067,19883],{"class":481},[151,20069,19871],{"class":19870},[151,20071,19905],{"class":503},[151,20073,2001],{"class":19870},[151,20075,19883],{"class":481},[151,20077,19871],{"class":19870},[151,20079,19918],{"class":503},[151,20081,2001],{"class":19870},[151,20083,2798],{"class":481},[151,20085,20086],{"class":503},";\n",[151,20088,20089,20091,20093,20095,20097,20100,20102,20104,20106,20108,20111],{"class":469,"line":517},[151,20090,12348],{"class":12347},[151,20092,12351],{"class":503},[151,20094,12355],{"class":12354},[151,20096,6208],{"class":503},[151,20098,20099],{"class":12360},"article",[151,20101,12364],{"class":503},[151,20103,1876],{"class":1869},[151,20105,12369],{"class":1869},[151,20107,12372],{"class":473},[151,20109,20110],{"class":503},"(route.params.slug, () ",[151,20112,12378],{"class":12347},[151,20114,20115,20117,20120,20123],{"class":469,"line":534},[151,20116,12383],{"class":473},[151,20118,20119],{"class":503},"(page).",[151,20121,20122],{"class":473},"findOne",[151,20124,12461],{"class":503},[151,20126,20127],{"class":469,"line":1413},[151,20128,20129],{"class":503},");\n",[151,20131,20132,20134,20136],{"class":469,"line":1418},[151,20133,19966],{"class":503},[151,20135,19822],{"class":14368},[151,20137,3742],{"class":503},[11,20139,20140,20141,20144,20145,20147,20148,20150,20151,20153,20154,20156,20157,20160,20161,20164],{},"Note: using ",[30,20142,20143],{},"route.param.slug"," as the first argument for ",[30,20146,19976],{}," is important when using ",[30,20149,19976],{},". I initially misunderstood the statement about this needing to be a unique value and I gave it a value of ",[30,20152,20099],{},", but this caused a strange cache issue when I viewed different articles. The problem didn't show when I was developing with ",[30,20155,12175],{},", it only showed up when I built the site and served in locally with ",[30,20158,20159],{},"yarn generate && yarn serve",". Since it is used in a dynamic page ",[30,20162,20163],{},"[slug].vue",", it needs to have a unique value for every individual page for the dynamic route.",[11,20166,20167,20168,20173],{},"I learned about this from ",[20,20169,20172],{"href":20170,"rel":20171},"https://nuxt.com/docs/getting-started/upgrade#shared-prerender-data",[24],"an example"," on the Nuxt 3 site about migration to Nuxt 4:",[459,20175,20177],{"className":19459,"code":20176,"language":19461,"meta":464,"style":464},"// This would be unsafe in a dynamic page (e.g. `[slug].vue`) because the route slug makes a difference\n// to the data fetched, but Nuxt can't know that because it's not reflected in the key.\nconst route = useRoute()\nconst { data } = await useAsyncData(async () => {\n  return await $fetch(`/api/my-page/${route.params.slug}`)\n})\n// Instead, you should use a key that uniquely identifies the data fetched.\nconst { data } = await useAsyncData(route.params.slug, async () => {\n  return await $fetch(`/api/my-page/${route.params.slug}`)\n})\n",[30,20178,20179,20184,20189,20201,20228,20262,20266,20271,20298,20328],{"__ignoreMap":464},[151,20180,20181],{"class":469,"line":470},[151,20182,20183],{"class":1527},"// This would be unsafe in a dynamic page (e.g. `[slug].vue`) because the route slug makes a difference\n",[151,20185,20186],{"class":469,"line":488},[151,20187,20188],{"class":1527},"// to the data fetched, but Nuxt can't know that because it's not reflected in the key.\n",[151,20190,20191,20193,20195,20197,20199],{"class":469,"line":500},[151,20192,12348],{"class":12347},[151,20194,20004],{"class":12360},[151,20196,19865],{"class":1869},[151,20198,20009],{"class":473},[151,20200,12461],{"class":503},[151,20202,20203,20205,20207,20209,20211,20213,20215,20217,20219,20221,20224,20226],{"class":469,"line":509},[151,20204,12348],{"class":12347},[151,20206,12351],{"class":503},[151,20208,12355],{"class":12360},[151,20210,12364],{"class":503},[151,20212,1876],{"class":1869},[151,20214,12369],{"class":1869},[151,20216,12372],{"class":473},[151,20218,12386],{"class":503},[151,20220,15221],{"class":1869},[151,20222,20223],{"class":503}," () ",[151,20225,17166],{"class":12347},[151,20227,19833],{"class":503},[151,20229,20230,20233,20235,20238,20240,20243,20245,20248,20250,20252,20254,20256,20258,20260],{"class":469,"line":517},[151,20231,20232],{"class":1869},"  return",[151,20234,12369],{"class":1869},[151,20236,20237],{"class":473}," $fetch",[151,20239,12386],{"class":503},[151,20241,20242],{"class":481},"`/api/my-page/",[151,20244,19871],{"class":19870},[151,20246,20247],{"class":503},"route",[151,20249,643],{"class":4828},[151,20251,19852],{"class":503},[151,20253,643],{"class":4828},[151,20255,19918],{"class":503},[151,20257,2001],{"class":19870},[151,20259,2798],{"class":481},[151,20261,3640],{"class":503},[151,20263,20264],{"class":469,"line":534},[151,20265,19610],{"class":503},[151,20267,20268],{"class":469,"line":1413},[151,20269,20270],{"class":1527},"// Instead, you should use a key that uniquely identifies the data fetched.\n",[151,20272,20273,20275,20277,20279,20281,20283,20285,20287,20290,20292,20294,20296],{"class":469,"line":1418},[151,20274,12348],{"class":12347},[151,20276,12351],{"class":503},[151,20278,12355],{"class":12360},[151,20280,12364],{"class":503},[151,20282,1876],{"class":1869},[151,20284,12369],{"class":1869},[151,20286,12372],{"class":473},[151,20288,20289],{"class":503},"(route.params.slug, ",[151,20291,15221],{"class":1869},[151,20293,20223],{"class":503},[151,20295,17166],{"class":12347},[151,20297,19833],{"class":503},[151,20299,20300,20302,20304,20306,20308,20310,20312,20314,20316,20318,20320,20322,20324,20326],{"class":469,"line":2462},[151,20301,20232],{"class":1869},[151,20303,12369],{"class":1869},[151,20305,20237],{"class":473},[151,20307,12386],{"class":503},[151,20309,20242],{"class":481},[151,20311,19871],{"class":19870},[151,20313,20247],{"class":503},[151,20315,643],{"class":4828},[151,20317,19852],{"class":503},[151,20319,643],{"class":4828},[151,20321,19918],{"class":503},[151,20323,2001],{"class":19870},[151,20325,2798],{"class":481},[151,20327,3640],{"class":503},[151,20329,20330],{"class":469,"line":2471},[151,20331,19610],{"class":503},[11,20333,20334],{},"Wow, there is already a Nuxt 4 in the works! Hopefully the upgrade process from Nuxt 3 to 4 is easier than Nuxt 2 to 3.",[736,20336,20338],{"id":20337},"development-process","Development process",[76,20340,20341,20345],{},[79,20342,20343],{},[30,20344,12175],{},[79,20346,20347],{},[30,20348,20349],{},"yarn clean && yarn generate && yarn serve",[11,20351,20352,20353,20355,20356,20359,20360,20363],{},"When I want to run the site locally I run ",[30,20354,12175],{}," which is an alias for ",[30,20357,20358],{},"nuxt dev --host",". Adding the ",[30,20361,20362],{},"--host"," option is important for when you want to view the site on a mobile device. It provides a QR code that links directly to the local IP:",[459,20365,20368],{"className":20366,"code":20367,"language":997},[995],"~/git/github/briancaffey.github.io$ yarn dev\nyarn run v1.22.21\n$ nuxt dev --host\nNuxt 3.12.4 with Nitro 2.9.7                                                   8:04:47 AM\n                                                                               8:04:47 AM\n\n              █▀▀▀▀▀▀▀█▀██▀▀███▀█▀▀▀▀▀▀▀█\n              █ █▀▀▀█ ██▄ █▀  ███ █▀▀▀█ █\n              █ █   █ ██▀█▀▄ ▄▄▄█ █   █ █\n              █ ▀▀▀▀▀ █▀█ █ █▀█▀█ ▀▀▀▀▀ █\n              █▀▀█▀▀█▀█▄▀  █▄▄▀▄█▀█████▀█\n              █▀ ▄▀ ▄▀▄▄▄▀▀▄ ▀ ▄ ▀▀  ▄▄▀█\n              █▄▄  ▀▀▀██ ▄  █▀▄▀ ▀█▄▄▄▄ █\n              █ ▀▀▄▀█▀▄█ █▀▀▄█ ▀▀█▄▄▀▀ ▀█\n              █ █▀▄▀▄▀ █▀▀▄▀▀▀█ ▀▀▀ █ ▀▄█\n              █▀▀▀▀▀▀▀█▄ █ ▄▀ ▄ █▀█ ▀█▄▀█\n              █ █▀▀▀█ █▀ ▀ ▀▄▄▀ ▀▀▀ ▀█▄▄█\n              █ █   █ █▄▄█  █▀███▄▄▀▄▀  █\n              █ ▀▀▀▀▀ █ ██ ▄█▀ ▀▄ ▄▄▀▄▄ █\n              ▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀\n\n  ➜ Local:    http://localhost:3000/\n  ➜ Network:  http://192.168.5.98:3000/ [QR code]\n\nℹ Using Tailwind CSS from ~/assets/css/tailwind.css          nuxt:tailwindcss 8:04:49 AM\n  ➜ DevTools: press Shift + Option + D in the browser (v1.3.9)                 8:04:50 AM\n\nℹ Tailwind Viewer: http://localhost:3000/_tailwind/          nuxt:tailwindcss 8:04:50 AM\n✔ Vite client built in 32ms                                                   8:04:51 AM\n✔ Vite server built in 1286ms                                                 8:04:52 AM\n✔ Nuxt Nitro server built in 1895 ms                                                            nitro 8:05:05 AM\nℹ Vite client warmed up in 0ms                                                                        8:05:05 AM\nℹ Vite server warmed up in 2186ms                                                                     8:05:07 AM\n",[30,20369,20367],{"__ignoreMap":464},[736,20371,20373],{"id":20372},"nuxt-dev-tools","Nuxt Dev Tools",[11,20375,20376,20381],{},[20,20377,20380],{"href":20378,"rel":20379},"https://devtools.nuxt.com/guide/getting-started",[24],"Nuxt DevTools"," is amazing! This is one of the best benefits of upgrading to Nuxt 3 for me.",[11,20383,20384],{},[2718,20385],{"alt":20386,"src":20387},"png","/static/nuxt/nuxt_dev_tools.png",[11,20389,20390],{},"It is available in Nuxt 3.9.0 or higher.",[736,20392,20394],{"id":20393},"vue-3-and-script-setup","Vue 3 and Script setup",[11,20396,20397,20398,20400],{},"Vue 3 allows for a much more streamlined single file component syntax with ",[30,20399,19980],{},". For most of the component I did some minimal cleanup and then asked ChatGPT to convert the component script tag to use the new script setup syntax, and that worked very well!",[736,20402,12503],{"id":12502},[11,20404,20405],{},"My blog is update using the following GitHub Action:",[459,20407,20409],{"className":14359,"code":20408,"language":14361,"meta":464,"style":464},"name: github pages\n\non:\n  push:\n    branches:\n      - master\n  pull_request:\n\njobs:\n  deploy:\n    runs-on: ubuntu-latest\n    steps:\n      - uses: actions/checkout@v4\n\n      - name: Setup Node\n        uses: actions/setup-node@v4\n        with:\n          node-version: \"20\"\n\n      - name: Cache dependencies\n        uses: actions/cache@v4\n        with:\n          path: ~/.npm\n          key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}\n          restore-keys: |\n            ${{ runner.os }}-node-\n\n      - run: yarn\n      - run: yarn build\n      - run: yarn lint\n      - run: yarn generate\n\n      - name: deploy\n        uses: peaceiris/actions-gh-pages@v4\n        with:\n          github_token: ${{ secrets.GITHUB_TOKEN }}\n          publish_dir: dist\n",[30,20410,20411,20421,20425,20432,20439,20446,20453,20460,20464,20471,20478,20488,20495,20507,20511,20522,20532,20538,20548,20552,20563,20572,20578,20588,20598,20608,20613,20617,20629,20640,20651,20662,20666,20677,20686,20692,20702],{"__ignoreMap":464},[151,20412,20413,20416,20418],{"class":469,"line":470},[151,20414,20415],{"class":14368},"name",[151,20417,6208],{"class":503},[151,20419,20420],{"class":481},"github pages\n",[151,20422,20423],{"class":469,"line":488},[151,20424,1090],{"emptyLinePlaceholder":609},[151,20426,20427,20430],{"class":469,"line":500},[151,20428,20429],{"class":477},"on",[151,20431,14372],{"class":503},[151,20433,20434,20437],{"class":469,"line":509},[151,20435,20436],{"class":14368},"  push",[151,20438,14372],{"class":503},[151,20440,20441,20444],{"class":469,"line":517},[151,20442,20443],{"class":14368},"    branches",[151,20445,14372],{"class":503},[151,20447,20448,20450],{"class":469,"line":534},[151,20449,14459],{"class":503},[151,20451,20452],{"class":481},"master\n",[151,20454,20455,20458],{"class":469,"line":1413},[151,20456,20457],{"class":14368},"  pull_request",[151,20459,14372],{"class":503},[151,20461,20462],{"class":469,"line":1418},[151,20463,1090],{"emptyLinePlaceholder":609},[151,20465,20466,20469],{"class":469,"line":2462},[151,20467,20468],{"class":14368},"jobs",[151,20470,14372],{"class":503},[151,20472,20473,20476],{"class":469,"line":2471},[151,20474,20475],{"class":14368},"  deploy",[151,20477,14372],{"class":503},[151,20479,20480,20483,20485],{"class":469,"line":2480},[151,20481,20482],{"class":14368},"    runs-on",[151,20484,6208],{"class":503},[151,20486,20487],{"class":481},"ubuntu-latest\n",[151,20489,20490,20493],{"class":469,"line":2489},[151,20491,20492],{"class":14368},"    steps",[151,20494,14372],{"class":503},[151,20496,20497,20499,20502,20504],{"class":469,"line":2497},[151,20498,14459],{"class":503},[151,20500,20501],{"class":14368},"uses",[151,20503,6208],{"class":503},[151,20505,20506],{"class":481},"actions/checkout@v4\n",[151,20508,20509],{"class":469,"line":3140},[151,20510,1090],{"emptyLinePlaceholder":609},[151,20512,20513,20515,20517,20519],{"class":469,"line":3149},[151,20514,14459],{"class":503},[151,20516,20415],{"class":14368},[151,20518,6208],{"class":503},[151,20520,20521],{"class":481},"Setup Node\n",[151,20523,20524,20527,20529],{"class":469,"line":3158},[151,20525,20526],{"class":14368},"        uses",[151,20528,6208],{"class":503},[151,20530,20531],{"class":481},"actions/setup-node@v4\n",[151,20533,20534,20536],{"class":469,"line":3167},[151,20535,16967],{"class":14368},[151,20537,14372],{"class":503},[151,20539,20540,20543,20545],{"class":469,"line":3175},[151,20541,20542],{"class":14368},"          node-version",[151,20544,6208],{"class":503},[151,20546,20547],{"class":481},"\"20\"\n",[151,20549,20550],{"class":469,"line":3184},[151,20551,1090],{"emptyLinePlaceholder":609},[151,20553,20554,20556,20558,20560],{"class":469,"line":3193},[151,20555,14459],{"class":503},[151,20557,20415],{"class":14368},[151,20559,6208],{"class":503},[151,20561,20562],{"class":481},"Cache dependencies\n",[151,20564,20565,20567,20569],{"class":469,"line":3720},[151,20566,20526],{"class":14368},[151,20568,6208],{"class":503},[151,20570,20571],{"class":481},"actions/cache@v4\n",[151,20573,20574,20576],{"class":469,"line":3729},[151,20575,16967],{"class":14368},[151,20577,14372],{"class":503},[151,20579,20580,20583,20585],{"class":469,"line":3735},[151,20581,20582],{"class":14368},"          path",[151,20584,6208],{"class":503},[151,20586,20587],{"class":481},"~/.npm\n",[151,20589,20590,20593,20595],{"class":469,"line":3745},[151,20591,20592],{"class":14368},"          key",[151,20594,6208],{"class":503},[151,20596,20597],{"class":481},"${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}\n",[151,20599,20600,20603,20605],{"class":469,"line":3754},[151,20601,20602],{"class":14368},"          restore-keys",[151,20604,6208],{"class":503},[151,20606,20607],{"class":1869},"|\n",[151,20609,20610],{"class":469,"line":3760},[151,20611,20612],{"class":481},"            ${{ runner.os }}-node-\n",[151,20614,20615],{"class":469,"line":3773},[151,20616,1090],{"emptyLinePlaceholder":609},[151,20618,20619,20621,20624,20626],{"class":469,"line":3782},[151,20620,14459],{"class":503},[151,20622,20623],{"class":14368},"run",[151,20625,6208],{"class":503},[151,20627,20628],{"class":481},"yarn\n",[151,20630,20631,20633,20635,20637],{"class":469,"line":3791},[151,20632,14459],{"class":503},[151,20634,20623],{"class":14368},[151,20636,6208],{"class":503},[151,20638,20639],{"class":481},"yarn build\n",[151,20641,20642,20644,20646,20648],{"class":469,"line":3803},[151,20643,14459],{"class":503},[151,20645,20623],{"class":14368},[151,20647,6208],{"class":503},[151,20649,20650],{"class":481},"yarn lint\n",[151,20652,20653,20655,20657,20659],{"class":469,"line":3811},[151,20654,14459],{"class":503},[151,20656,20623],{"class":14368},[151,20658,6208],{"class":503},[151,20660,20661],{"class":481},"yarn generate\n",[151,20663,20664],{"class":469,"line":3820},[151,20665,1090],{"emptyLinePlaceholder":609},[151,20667,20668,20670,20672,20674],{"class":469,"line":7084},[151,20669,14459],{"class":503},[151,20671,20415],{"class":14368},[151,20673,6208],{"class":503},[151,20675,20676],{"class":481},"deploy\n",[151,20678,20679,20681,20683],{"class":469,"line":7148},[151,20680,20526],{"class":14368},[151,20682,6208],{"class":503},[151,20684,20685],{"class":481},"peaceiris/actions-gh-pages@v4\n",[151,20687,20688,20690],{"class":469,"line":7211},[151,20689,16967],{"class":14368},[151,20691,14372],{"class":503},[151,20693,20694,20697,20699],{"class":469,"line":7273},[151,20695,20696],{"class":14368},"          github_token",[151,20698,6208],{"class":503},[151,20700,20701],{"class":481},"${{ secrets.GITHUB_TOKEN }}\n",[151,20703,20704,20707,20709],{"class":469,"line":7335},[151,20705,20706],{"class":14368},"          publish_dir",[151,20708,6208],{"class":503},[151,20710,20711],{"class":481},"dist\n",[11,20713,20714,20715,129,20718,20721,20722,129,20724,13576],{},"When I upgraded to Nuxt 3, I needed to add the ",[30,20716,20717],{},"yarn build",[30,20719,20720],{},"nuxt build",") step in order to run ",[30,20723,12556],{},[30,20725,20726],{},"eslint .",[11,20728,20729],{},"It takes about 3 minutes to build the site, most of that time is for building the HTML files for each route:",[11,20731,20732],{},[2718,20733],{"alt":20734,"src":20735},"GitHub Action for deploying briancaffey.github.io","/static/nuxt/gha.png",[459,20737,20740],{"className":20738,"code":20739,"language":997},[995],"[log] [nitro]   ├─ /sitemap.xml (26ms)\n[info] [nitro] Prerendered 525 routes in 67.595 seconds\n[success] [nitro] Generated public .output/public\n[success] [nitro] You can preview this build using `npx serve .output/public`\n[success] You can now deploy `.output/public` to any static hosting!\nDone in 81.09s.\n",[30,20741,20739],{"__ignoreMap":464},[11,20743,20744,20745,20748,20749,20752,20753,20756,20757,20760,20761,20763],{},"Shout out to GitHub user ",[30,20746,20747],{},"peaceiris"," for maintaining the ",[30,20750,20751],{},"peaceiris/actions-gh-pages@v4"," GitHub Action. ",[30,20754,20755],{},"dist"," is a symbolic link that links to ",[30,20758,20759],{},".output/public"," where the static build files from ",[30,20762,12471],{}," are stored.",[11,20765,20766],{},[2718,20767],{"alt":20734,"src":20768},"/static/nuxt/gha_deploy.png",[56,20770,20772],{"id":20771},"data-heavy-articles","Data-heavy articles",[736,20774,20776],{"id":20775},"migrating-from-vuex-to-pinia","Migrating from Vuex to Pinia",[11,20778,20779,20780,20782,20783,20788],{},"Most of my blog articles only include text and images. In some articles I include dynamic content through iframes to other projects on my GitHub that are deployed on subdomains of ",[30,20781,662],{},". Another way to add dynamic content it to write Vue components and then embed those directly in the Nuxt Content Markdown files. I wrote ",[20,20784,20787],{"href":20785,"rel":20786},"https://briancaffey.github.io/2021/01/16/i-scraped-analyzed-and-generated-yc-companies-founders-and-work-at-a-startup-job-postings",[24],"an article about data from YC's Work at a Startup jobs page"," and made components that get data from a data store and then render that data using Apex Charts.",[11,20790,20791],{},"Previously I had used Vuex to do this, but I switched to using Pinia which is Vue's new module for managing state. I use LLMs to convert the store module from Vuex to Pinia and also used LLMs to update the components that use the store, and it worked!",[11,20793,20794],{},"Setting up the plugin for Apex Charts looks like this:",[459,20796,20798],{"className":19459,"code":20797,"language":19461,"meta":464,"style":464},"import VueApexCharts from 'vue3-apexcharts'\n\nexport default defineNuxtPlugin(nuxtApp => {\n    nuxtApp.vueApp.use(VueApexCharts)\n});\n",[30,20799,20800,20812,20816,20835,20846],{"__ignoreMap":464},[151,20801,20802,20804,20807,20809],{"class":469,"line":470},[151,20803,16859],{"class":1869},[151,20805,20806],{"class":503}," VueApexCharts ",[151,20808,16853],{"class":1869},[151,20810,20811],{"class":481}," 'vue3-apexcharts'\n",[151,20813,20814],{"class":469,"line":488},[151,20815,1090],{"emptyLinePlaceholder":609},[151,20817,20818,20820,20822,20825,20827,20830,20833],{"class":469,"line":500},[151,20819,1870],{"class":1869},[151,20821,19470],{"class":1869},[151,20823,20824],{"class":473}," defineNuxtPlugin",[151,20826,12386],{"class":503},[151,20828,20829],{"class":15210},"nuxtApp",[151,20831,20832],{"class":12347}," =>",[151,20834,19833],{"class":503},[151,20836,20837,20840,20843],{"class":469,"line":509},[151,20838,20839],{"class":503},"    nuxtApp.vueApp.",[151,20841,20842],{"class":473},"use",[151,20844,20845],{"class":503},"(VueApexCharts)\n",[151,20847,20848],{"class":469,"line":517},[151,20849,20850],{"class":503},"});\n",[56,20852,20853],{"id":11627},"Lessons learned",[736,20855,20857],{"id":20856},"embedding-tweets-in-nuxt-content","Embedding Tweets in Nuxt Content",[11,20859,20860],{},"The upgrade from Nuxt 2 to Nuxt 3 broke some twitter embeds that were working in Nuxt 2 (directly copy and pasting the embed code into a Nuxt Content Markdown file). Here's how I got it working for now:",[76,20862,20863,20874,20892,20902,20905],{},[79,20864,20865,20866,129,20869,748],{},"create a new global component in ",[30,20867,20868],{},"components/content",[20,20870,20873],{"href":20871,"rel":20872},"https://github.com/briancaffey/briancaffey.github.io/blob/master/components/content/aoi/AgentsOfInferenceTweet.vue",[24],"example",[79,20875,20876,20877,20880,20881,20884,20885,20888,20889,748],{},"convert the ",[30,20878,20879],{},"\u003Cscript>"," tag in the embed code to a ",[30,20882,20883],{},"\u003Ccomponent>"," tag and include the ",[30,20886,20887],{},":is=\"'script'\""," (ES Lint will throw an error if you do not have the ",[30,20890,20891],{},"v-bind:is",[79,20893,20894,20895,129,20898,20901],{},"include the component in your Markdown like this: ",[30,20896,20897],{},"\u003CMyComponent>\u003C/MyComponent>",[30,20899,20900],{},"\u003CMyComponent />"," will not work, for me it would not render and also cut off the rest of the content from the page)",[79,20903,20904],{},"The same process works for video embeds from 𝕏",[79,20906,20907,20912],{},[20,20908,20911],{"href":20909,"rel":20910},"https://briancaffey.github.io/2024/06/24/agents-of-inference-speed-of-light-nvidia-langchain-generative-ai-agents-developer-contest-update",[24],"This post on my GitHub Pages blog"," uses an embedded tweet.",[736,20914,20916],{"id":20915},"internationalization-i18n","Internationalization (i18n)",[11,20918,20919,20920,313,20923,20926],{},"I added i18n to my site mostly to learn how it works. Nuxt i18n has different strategies for how different locales are displayed. Previously I used a URL prefix for all locales other than the default locale (English). Switching to other locales would switch from ",[30,20921,20922],{},"/contact-me",[30,20924,20925],{},"/zh/contact-me"," for example.",[11,20928,20929,20930,20933],{},"In this upgrade I switched to the ",[30,20931,20932],{},"no_prefix"," option which instead stores the locale in a cookie. This makes generating my site easier because it does not require generating a locale for each blog tag or blog article.",[11,20935,20936],{},"I currently do not have i18n for the articles on my blog, but I'm hoping to add this in a future update once there is better support for it in Nuxt Content.",[56,20938,20940],{"id":20939},"lighthouse","Lighthouse",[11,20942,20943],{},"I made a number of improvements to the site to get an almost-perfect Lighthouse score for the home page of my site:",[11,20945,20946],{},[2718,20947],{"alt":20948,"src":20949},"Lighthouse results for briancaffey.github.io","/static/nuxt/lighthouse.png",[76,20951,20952,20962,20970],{},[79,20953,20954,20955,20958,20959,748],{},"using ",[30,20956,20957],{},"@nuxt/image"," for optimized image formats (",[30,20960,20961],{},"webp",[79,20963,20964,20965,748],{},"adjust colors for improved contrast (measured using ",[20,20966,20969],{"href":20967,"rel":20968},"https://webaim.org/resources/contrastchecker/",[24],"webaim.org",[79,20971,20972,20973,20976],{},"fixes for ",[30,20974,20975],{},"head"," metadata",[56,20978,20980],{"id":20979},"interactivity","Interactivity",[11,20982,20983],{},"I use a few plugins for interactivity on my site. These plugins needed some slight modifications and upgrades",[736,20985,20987],{"id":20986},"drift","Drift",[11,20989,20990],{},"Drift is a chat box that lets users send me message. When someone sends me a message I can see what page of my website they are on and I can also see their location (based on their IP address). I get messages in the Drift app on my phone. In total I have had 320 conversations since I initially added the plugin a few years ago.",[736,20992,20994],{"id":20993},"mailchimp-email-list","MailChimp email list",[11,20996,20997,20998,21001,21002,21005],{},"I have an email list of 55 people that I manage through MailChimp. Occasionally I send out emails about new articles on my blog and other updates. It is a fun way to practice email marketing! It uses a global component in the ",[30,20999,21000],{},"content/components"," directory so I can use the ",[30,21003,21004],{},"\u003CSubscribe>"," component here in the Markdown file where I am writing this article:",[1205,21007],{},[21009,21010],"subscribe",{},[1205,21012],{},[11,21014,21015],{},"I also use this component in the footer of the site. Feel free to sign up to get updates about what I'm doing on this site!",[736,21017,21019],{"id":21018},"formsubmit","FormSubmit",[11,21021,21022,21023,21027],{},"FormSubmit is a free service that lets people send me a message through a form on my site's ",[20,21024,21026],{"href":21025},"/contact","Contact"," page.",[736,21029,21031],{"id":21030},"disqus","Disqus",[11,21033,21034],{},"Disqus is a comments plugin that I use on my blog articles. I don't get a lot of comments, but comments are always welcome!",[56,21036,21038],{"id":21037},"todo","TODO",[736,21040,21042],{"id":21041},"feedxml","feed.xml",[11,21044,21045,21046,21048,21049,21054],{},"I need to find a way to automate ",[30,21047,21042],{}," generation. ",[20,21050,21053],{"href":21051,"rel":21052},"https://nuxt.com/modules/feed",[24],"The feed module"," is not yet compatible with Nuxt 3. I do use the RSS feed with DEV.to which allows me to set up canonical links back to my GitHub Pages site.",[11,21056,21057,21058,21060],{},"For now I am going to copy the ",[30,21059,21042],{}," to a file in the public directory and update it manually. Here's the entry I'll make for this article:",[459,21062,21064],{"className":19811,"code":21063,"language":19813,"meta":464,"style":464},"        \u003Citem>\n            \u003Ctitle>\n                \u003C![CDATA[ Upgrading my GitHub Pages blog to Nuxt 3 ]]>\n            \u003C/title>\n            \u003Clink>\n                https://briancaffey.github.io/2024/08/11/upgrading-my-github-pages-blog-to-nuxt-3\n            \u003C/link>\n            \u003Cguid>\n                https://briancaffey.github.io/2024/08/11/upgrading-my-github-pages-blog-to-nuxt-3\n            \u003C/guid>\n            \u003Cdescription>\n                \u003C![CDATA[ An overview of my newly upgraded GitHub Pages blog powered by Nuxt 3 ]]>\n            \u003C/description>\n        \u003C/item>\n",[30,21065,21066,21075,21084,21095,21104,21112,21117,21125,21134,21138,21146,21154,21163,21171],{"__ignoreMap":464},[151,21067,21068,21071,21073],{"class":469,"line":470},[151,21069,21070],{"class":503},"        \u003C",[151,21072,11601],{"class":6607},[151,21074,3742],{"class":503},[151,21076,21077,21080,21082],{"class":469,"line":488},[151,21078,21079],{"class":503},"            \u003C",[151,21081,19633],{"class":14368},[151,21083,3742],{"class":503},[151,21085,21086,21089,21092],{"class":469,"line":500},[151,21087,21088],{"class":503},"                \u003C![CDATA[",[151,21090,21091],{"class":481}," Upgrading my GitHub Pages blog to Nuxt 3 ",[151,21093,21094],{"class":503},"]]>\n",[151,21096,21097,21100,21102],{"class":469,"line":509},[151,21098,21099],{"class":503},"            \u003C/",[151,21101,19633],{"class":14368},[151,21103,3742],{"class":503},[151,21105,21106,21108,21110],{"class":469,"line":517},[151,21107,21079],{"class":503},[151,21109,19750],{"class":14368},[151,21111,3742],{"class":503},[151,21113,21114],{"class":469,"line":534},[151,21115,21116],{"class":503},"                https://briancaffey.github.io/2024/08/11/upgrading-my-github-pages-blog-to-nuxt-3\n",[151,21118,21119,21121,21123],{"class":469,"line":1413},[151,21120,21099],{"class":503},[151,21122,19750],{"class":6607},[151,21124,3742],{"class":503},[151,21126,21127,21129,21132],{"class":469,"line":1418},[151,21128,21079],{"class":503},[151,21130,21131],{"class":6607},"guid",[151,21133,3742],{"class":503},[151,21135,21136],{"class":469,"line":2462},[151,21137,21116],{"class":503},[151,21139,21140,21142,21144],{"class":469,"line":2471},[151,21141,21099],{"class":503},[151,21143,21131],{"class":6607},[151,21145,3742],{"class":503},[151,21147,21148,21150,21152],{"class":469,"line":2480},[151,21149,21079],{"class":503},[151,21151,19656],{"class":6607},[151,21153,3742],{"class":503},[151,21155,21156,21158,21161],{"class":469,"line":2489},[151,21157,21088],{"class":503},[151,21159,21160],{"class":481}," An overview of my newly upgraded GitHub Pages blog powered by Nuxt 3 ",[151,21162,21094],{"class":503},[151,21164,21165,21167,21169],{"class":469,"line":2497},[151,21166,21099],{"class":503},[151,21168,19656],{"class":6607},[151,21170,3742],{"class":503},[151,21172,21173,21176,21178],{"class":469,"line":3140},[151,21174,21175],{"class":503},"        \u003C/",[151,21177,11601],{"class":6607},[151,21179,3742],{"class":503},[736,21181,21183],{"id":21182},"console-errors","Console errors",[11,21185,21186],{},"I have tried to clean up as many of the errors as I could, but there are still some that I see in the dev console. Here is one of the issues that puzzles me:",[11,21188,21189,21192,21193,21195],{},[30,21190,21191],{},"Hydration completed but contains mismatches.",": I only see this error on the production build; I don't see it when running ",[30,21194,12175],{},". As I understand, this error message means that the HTML that was built during prerendering is not the same as the HTML on the site once the Javascript has all been loaded.",[736,21197,21199],{"id":21198},"build-errors","Build errors",[11,21201,21202,21203,21206,21207,21210],{},"I recently got some errors in my CI/CD pipeline related to the ",[30,21204,21205],{},"string-width"," package, and I was able to add the following to ",[30,21208,21209],{},"package.json"," to fix the build pipeline:",[459,21212,21214],{"className":19459,"code":21213,"language":19461,"meta":464,"style":464},"  \"resolutions\": {\n      \"string-width\": \"4.2.3\"\n  }\n",[30,21215,21216,21224,21234],{"__ignoreMap":464},[151,21217,21218,21221],{"class":469,"line":470},[151,21219,21220],{"class":481},"  \"resolutions\"",[151,21222,21223],{"class":503},": {\n",[151,21225,21226,21229,21231],{"class":469,"line":488},[151,21227,21228],{"class":481},"      \"string-width\"",[151,21230,6208],{"class":503},[151,21232,21233],{"class":481},"\"4.2.3\"\n",[151,21235,21236],{"class":469,"line":500},[151,21237,19957],{"class":503},[11,21239,21240],{},"I'm still not exactly sure what this is about.",[736,21242,21244],{"id":21243},"refactoring","Refactoring",[11,21246,21247,21248,21251],{},"I like using TailwindCSS, and I was able to use it to build a responsive design for my site. After upgrading to Nuxt 3, I feel like most of the technical debt is now in the design. I also don't change the design that often, but I think I could do a lot to refactor the use of Tailwind, such as using ",[30,21249,21250],{},"@apply"," in CSS to make classes more DRY across the different components I use to build this site.",[11,21253,21254],{},"Please let me know if you have any questions, suggestions or tips for using Nuxt and Vue to build static prerendered sites. Thanks for reading!",[589,21256,21257],{},"html pre.shiki code .sC2Qs, html code.shiki .sC2Qs{--shiki-default:#D73A49;--shiki-dark:#F97583;--shiki-sepia:#F92672}html pre.shiki code .srTi1, html code.shiki .srTi1{--shiki-default:#6F42C1;--shiki-dark:#B392F0;--shiki-sepia:#A6E22E}html pre.shiki code .sMOD_, html code.shiki .sMOD_{--shiki-default:#24292E;--shiki-dark:#E1E4E8;--shiki-sepia:#F8F8F2}html pre.shiki code .sstjo, html code.shiki .sstjo{--shiki-default:#032F62;--shiki-dark:#9ECBFF;--shiki-sepia:#E6DB74}html pre.shiki code .s8-w5, html code.shiki .s8-w5{--shiki-default:#6A737D;--shiki-dark:#6A737D;--shiki-sepia:#88846F}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html .sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html.sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html pre.shiki code .s_OQ2, html code.shiki .s_OQ2{--shiki-default:#6F42C1;--shiki-dark:#B392F0;--shiki-sepia:#F8F8F2}html pre.shiki code .s5clZ, html code.shiki .s5clZ{--shiki-default:#22863A;--shiki-dark:#85E89D;--shiki-sepia:#F92672}html pre.shiki code .s7F3e, html code.shiki .s7F3e{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#AE81FF}html pre.shiki code .sTHNf, html code.shiki .sTHNf{--shiki-default:#E36209;--shiki-default-font-style:inherit;--shiki-dark:#FFAB70;--shiki-dark-font-style:inherit;--shiki-sepia:#FD971F;--shiki-sepia-font-style:italic}html pre.shiki code .sq6CD, html code.shiki .sq6CD{--shiki-default:#D73A49;--shiki-default-font-style:inherit;--shiki-dark:#F97583;--shiki-dark-font-style:inherit;--shiki-sepia:#66D9EF;--shiki-sepia-font-style:italic}html pre.shiki code .sXSQT, html code.shiki .sXSQT{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#F8F8F2}html pre.shiki code .s1EfO, html code.shiki .s1EfO{--shiki-default:#032F62;--shiki-dark:#9ECBFF;--shiki-sepia:#F92672}html pre.shiki code .sinWB, html code.shiki .sinWB{--shiki-default:#032F62;--shiki-dark:#9ECBFF;--shiki-sepia:#F8F8F2}html pre.shiki code .sOrwc, html code.shiki .sOrwc{--shiki-default:#E36209;--shiki-dark:#FFAB70;--shiki-sepia:#F8F8F2}html pre.shiki code .st05x, html code.shiki .st05x{--shiki-default:#B31D28;--shiki-default-font-style:italic;--shiki-dark:#FDAEB7;--shiki-dark-font-style:italic;--shiki-sepia:#F44747;--shiki-sepia-font-style:inherit}",{"title":464,"searchDepth":488,"depth":488,"links":21259},[21260,21261,21269,21272,21276,21277,21283],{"id":19414,"depth":488,"text":19415},{"id":19445,"depth":488,"text":19446,"children":21262},[21263,21264,21265,21266,21267,21268],{"id":19449,"depth":500,"text":19450},{"id":19613,"depth":500,"text":19614},{"id":20337,"depth":500,"text":20338},{"id":20372,"depth":500,"text":20373},{"id":20393,"depth":500,"text":20394},{"id":12502,"depth":500,"text":12503},{"id":20771,"depth":488,"text":20772,"children":21270},[21271],{"id":20775,"depth":500,"text":20776},{"id":11627,"depth":488,"text":20853,"children":21273},[21274,21275],{"id":20856,"depth":500,"text":20857},{"id":20915,"depth":500,"text":20916},{"id":20939,"depth":488,"text":20940},{"id":20979,"depth":488,"text":20980,"children":21278},[21279,21280,21281,21282],{"id":20986,"depth":500,"text":20987},{"id":20993,"depth":500,"text":20994},{"id":21018,"depth":500,"text":21019},{"id":21030,"depth":500,"text":21031},{"id":21037,"depth":488,"text":21038,"children":21284},[21285,21286,21287,21288],{"id":21041,"depth":500,"text":21042},{"id":21182,"depth":500,"text":21183},{"id":21198,"depth":500,"text":21199},{"id":21243,"depth":500,"text":21244},"2024-08-11","An overview of my newly upgraded GitHub Pages blog powered by Nuxt 3","/static/nuxt/new-site.png",{},"/2024/08/11/upgrading-my-github-pages-blog-to-nuxt-3",{"title":19409,"description":21290},"2024/08/11/upgrading-my-github-pages-blog-to-nuxt-3",[12646,11803,21297,21298],"github","pinia","Rf1NvF_b9n-40t6oDSqtGN8PnJZUsfeLTPbjP6ECmx8",{"id":21301,"title":21302,"body":21303,"comments":609,"date":21561,"description":21562,"draft":602,"extension":605,"external":21563,"image":21566,"meta":21567,"navigation":609,"path":21568,"seo":21569,"stem":21570,"tags":21571,"__hash__":21580},"blog/2024/06/24/agents-of-inference-speed-of-light-nvidia-langchain-generative-ai-agents-developer-contest-update.md","Agents of Inference: Speed of Light -- Accelerating my Generative AI Agents project with NVIDIA NIMs, TensorRT and TensorRT-LLM",{"type":8,"value":21304,"toc":21554},[21305,21307,21314,21317,21320,21328,21332,21335,21341,21347,21354,21360,21366,21369,21375,21378,21384,21387,21393,21425,21428,21434,21437,21443,21452,21456,21459,21465,21472,21478,21481,21487,21490,21494,21503,21506,21512,21515,21521,21524,21530,21533,21536,21540,21543,21548,21551],[56,21306,16116],{"id":16115},[11,21308,21309,21310,643],{},"\"Agents of Inference: Speed of Light\" is an update to my original entry for the Generative AI Agents Developer Contest by NVIDIA and LangChain. This update focuses on how I accelerated local text, image and video generation using TensorRT, TensorRT-LLM and NVIDIA NIMs. You can read the original article about \"Agents of Inference\" ",[20,21311,13074],{"href":21312,"rel":21313},"https://briancaffey.github.io/2024/06/17/agents-of-inference-nvidia-and-langchain-generative-ai-agent-developer-contest",[24],[11,21315,21316],{},"Here's my original project submission post on 𝕏 that introduces the idea of generating short 007-style films using agents, LLMs and stable diffusion:",[21318,21319],"agents-of-inference-tweet",{},[11,21321,21322,21323,643],{},"Here's a link to the ",[20,21324,21327],{"href":21325,"rel":21326},"https://github.com/briancaffey/agents-of-inference",[24],"Agents of Inference code repository on GitHub",[56,21329,21331],{"id":21330},"nvidia-nim-inference-microservices","NVIDIA NIM inference microservices",[11,21333,21334],{},"I thought NVIDIA NIMs was one of the most exciting announcements from GTC 2024. I'm a big fan of using docker containers everywhere, and the idea of standardizing NVIDIA tools and dependencies seemed to make a lot of sense. I had previously struggled to get TensorRT-LLM installed on Windows using example repos provided by NVIDIA.",[11,21336,21337,21338,21340],{},"A few weeks ago NVIDIA announced that NVIDIA NIMs can be downloaded and run anywhere. I was able to download this NIM for the ",[30,21339,8349],{}," model:",[11,21342,21343],{},[2718,21344],{"alt":21345,"src":21346},"llama3 nim","/static/aoi/meta-llama3-nim.png",[11,21348,21349,21350,21353],{},"Here are the logs for my NVIDIA NIM ",[30,21351,21352],{},"Meta/Llama-3-8B-Instruct"," running in docker container on Windows Subsystem for Linux on my NVIDIA GeForce RTX 4090 GPU-powered PC. Notice that it generates over 50 tokens per second!",[11,21355,21356],{},[2718,21357],{"alt":21358,"src":21359},"trt llama3 local","/static/aoi/trt-llama3.png",[11,21361,21362],{},[2718,21363],{"alt":21364,"src":21365},"token factory","/static/aoi/token-factory.png",[11,21367,21368],{},"The one main hurdle I faced when running the NIM local was an error about no runnable profiles being available:",[459,21370,21373],{"className":21371,"code":21372,"language":997},[995],"ERROR 06-23 15:41:21.19 utils.py:21] Could not find a profile that is currently runnable with the detected hardware. Please check the system information below and make sure you have enough free GPUs.\nSYSTEM INFO\n- Free GPUs: \u003CNone>\n- Non-free GPUs:\n  -  [2684:10de] (0) NVIDIA GeForce RTX 4090 [current utilization: 7%]\n",[30,21374,21372],{"__ignoreMap":464},[11,21376,21377],{},"This seemed odd, and I found another user with the same issue on the NVIDIA Developer Forum. I was able to get around this by going into the EUFI/BIOS of my PC and switch to integrated graphics:",[11,21379,21380],{},[2718,21381],{"alt":21382,"src":21383},"bios","/static/aoi/bios.jpg",[11,21385,21386],{},"It was great to be able to run \"Agents of Inference\" using NVIDIA NIM because it is just as simple as running a docker container:",[459,21388,21391],{"className":21389,"code":21390,"language":997},[995],"export CONTAINER_NAME=llama3-8b-instruct\nexport IMG_NAME=\"nvcr.io/nim/meta/${CONTAINER_NAME}:1.0.0\"\nexport LOCAL_NIM_CACHE=~/.cache/nim\nmkdir -p \"$LOCAL_NIM_CACHE\"\ndocker run -it --rm --name=$CONTAINER_NAME \\\n  --runtime=nvidia \\\n  --gpus all \\\n  --shm-size=16GB \\\n  -e NGC_API_KEY \\\n  -v \"$LOCAL_NIM_CACHE:/opt/nim/.cache\" \\\n  -u $(id -u) \\\n  -p 8000:8000 \\\n  $IMG_NAME\n",[30,21392,21390],{"__ignoreMap":464},[11,21394,21395,21396,21399,21400,21405,21406,21409,21410,21417,21418,21420,21421,21424],{},"Before getting this to work, I was able to get a ",[30,21397,21398],{},"/chat/completions"," endpoint working with the Llama3 model on my fork of the ",[20,21401,21404],{"href":21402,"rel":21403},"https://github.com/briancaffey/trt-llm-as-openai-windows/commit/edaa15fd026fe95e645e3d4ae9718dc3ecc3bb65",[24],"trt-llm-as-openai-windows",". I borrowed code for the ",[30,21407,21408],{},"TrtLlmAPI"," from the ",[20,21411,21414],{"href":21412,"rel":21413},"https://github.com/NVIDIA/ChatRTX",[24],[30,21415,21416],{},"NVIDIA/ChatRTX"," repo and a function from ",[30,21419,19401],{}," called ",[30,21422,21423],{},"messages_to_prompt_v3_instruct"," which encodes messages with special tokens for chat. This was an interesting exercise and it taught me a lot about how LLMs do chat. I would like to continue working on this fork and see how to implement streaming endpoints for the Llama 3 model.",[11,21426,21427],{},"Here is how Llama 3 does the instruct prompting:",[459,21429,21432],{"className":21430,"code":21431,"language":997},[995],"\u003C|begin_of_text|>\u003C|start_header_id|>system\u003C|end_header_id|>\n\nYou are a helpful AI assistant for travel tips and recommendations\u003C|eot_id|>\u003C|start_header_id|>user\u003C|end_header_id|>\n\nWhat can you help me with?\u003C|eot_id|>\u003C|start_header_id|>assistant\u003C|end_header_id|>\n",[30,21433,21431],{"__ignoreMap":464},[11,21435,21436],{},"Compare this with how it was done with Llama2 chat:",[459,21438,21441],{"className":21439,"code":21440,"language":997},[995],"\u003Cs>[INST] \u003C\u003CSYS>>\n{{ system_prompt }}\n\u003C\u003C/SYS>>\n\n{{ user_message_1 }} [/INST] {{ model_answer_1 }} \u003C/s>\n\u003Cs>[INST] {{ user_message_2 }} [/INST]\n",[30,21442,21440],{"__ignoreMap":464},[11,21444,21445,21446,21451],{},"You can read more about the difference between Llama 2 and 3 on the ",[20,21447,21450],{"href":21448,"rel":21449},"https://llama.meta.com/docs/model-cards-and-prompt-formats",[24],"Model Card & Prompt formats"," page on Meta's Llama website.",[56,21453,21455],{"id":21454},"langsmith","LangSmith",[11,21457,21458],{},"I recently started using LangSmith. It is an awesome product and it ties in really well to doing prototype work like in my project \"Agents of Inference\". I wish I had started using it earlier in my development cycle! All you need to do is add an API key to your environment and your application automatically starts tracing LLM calls. It also works well with LangGraph and allows you to trace the execution path of your graph. Also it is good to be aware that there are other products similar to LangSmith like LangFuse. I also saw a really neat demo from Datadog at GTC showing an alpha version of their LLM tracing and observability product.",[11,21460,21461],{},[2718,21462],{"alt":21463,"src":21464},"langsmith screenshot","/static/aoi/langsmith.png",[11,21466,21467,21468,21471],{},"LangSmith can also be helpful when the wrong JSON shape is parsed. I had a lot of difficulty with this in my project. When I used the Q4_K_M gguf quantized ",[30,21469,21470],{},"Meta-Llama-3 8B-Instruct"," model I had no issues with output parsing. Switching to the TensorRT-LLM model provided by the NIM resulted in some parsing errors. The application would report that JSON could not be parsed because the result contained text like: \"Here is the JSON that you requested\". I was able to get around this by changing the prompt template from:",[459,21473,21476],{"className":21474,"code":21475,"language":997},[995],"Answer the user query.\n",[30,21477,21475],{"__ignoreMap":464},[11,21479,21480],{},"to",[459,21482,21485],{"className":21483,"code":21484,"language":997},[995],"Don't include ANYTHING except for valid JSON in your response. Answer the user query.\n",[30,21486,21484],{"__ignoreMap":464},[11,21488,21489],{},"This was the most frustrating part of development, and I'm still getting occasional errors that I just skip over. I'm also probably have not exhausted all of the tools that LangChain provides to avoid these types of errors. Don't assume that output parsing that works with one model will work with another! This is another good reason to use something like LangSmith when developing LLM-based applications.",[56,21491,21493],{"id":21492},"comfyui-tensorrt","ComfyUI TensorRT",[11,21495,21496,21497,21502],{},"My goal with \"Agents of Inference\" was to be able to test out how small upstream prompt changes can impact the quality and consistency of a series of generated images and videos. Iteration speed is very important! I was able to significantly speed up image and video generation by using the ",[20,21498,21501],{"href":21499,"rel":21500},"https://github.com/comfyanonymous/ComfyUI_TensorRT",[24],"ComfyUI TensorRT custom nodes",". These nodes allow you to build engines with specifications for parameters that can be either static or dynamic. I had better luck with building dynamic engines. I was able to build and use engines for Stable Diffusion SDXL and Stable Video Diffusion XT.",[11,21504,21505],{},"Building a TensorRT engine for ComfyUI can be done using the following workflow:",[11,21507,21508],{},[2718,21509],{"alt":21510,"src":21511},"trt comfyUI build process","/static/aoi/comfyui-trt-svd-xt.png",[11,21513,21514],{},"The engines can then be used in custom workflows like the following:",[11,21516,21517],{},[2718,21518],{"alt":21519,"src":21520},"trt comfyui workflow","/static/aoi/svd-workflow-trt.png",[11,21522,21523],{},"Once these workflows are configured and are working as expected, you can export them in API format (JSON) and use them to make API calls to the ComfyUI backend. The agents for stable diffusion and stable video diffusion made API calls in this way and it worked pretty well.",[11,21525,21526],{},[2718,21527],{"alt":21528,"src":21529},"comfy its","/static/aoi/comfy-its.png",[11,21531,21532],{},"Using 50 iterations, I was able to generate 1024x576 images in 3 seconds or about 19 iterations per second (it/s). Videos",[11,21534,21535],{},"ComfyUI is still early in development and it refers to itself as \"alpha software\" even though it has a large adoption by a very active community already. I'm excited to see what is next from the developers of ComfyUI.",[56,21537,21539],{"id":21538},"speed-of-light","Speed of Light",[11,21541,21542],{},"\"Speed of Light\" is a term that I learned from a stable diffusion talk at GTC.",[210,21544,21545],{},[11,21546,21547],{},"SOL analysis reveals how your code performs, and device utilization compared to relevant maximums.",[11,21549,21550],{},"Adding TensorRT and TensorRT-LLM to inference services on my RTX PC helped increase the throughput of text, image and video generation for my \"Agents of Inference\" project. I'm looking forward to learning more about profiling and optimization techniques for both LLMs and Stable Diffusion workloads.",[11,21552,21553],{},"Thanks again to NVIDIA and LangChain for organizing this contest! It was a lot of fun to learn about builing agents with LangChain and LangGraph and the latest developments from NVIDIA in Generative AI.",{"title":464,"searchDepth":488,"depth":488,"links":21555},[21556,21557,21558,21559,21560],{"id":16115,"depth":488,"text":16116},{"id":21330,"depth":488,"text":21331},{"id":21454,"depth":488,"text":21455},{"id":21492,"depth":488,"text":21493},{"id":21538,"depth":488,"text":21539},"2024-06-24","This article is a brief discusion on recent updates to my project for the Generative AI Agents Developer Contest by NVIDIA and LangChain",[21564],{"link":21565,"site":11126},"https://x.com/briancaffey/status/1802754703207583886","/static/aoi/aoi_title.png",{},"/2024/06/24/agents-of-inference-speed-of-light-nvidia-langchain-generative-ai-agents-developer-contest-update",{"title":21302,"description":21562},"2024/06/24/agents-of-inference-speed-of-light-nvidia-langchain-generative-ai-agents-developer-contest-update",[11133,21572,1822,14055,21573,21574,19403,614,615,21575,21576,21577,21578,21579],"langchain","gpu","tensorrt","llama","007","stable-diffusion","stable-video-diffusion","comfyui","IsVd9Qb_3DZvMylagXMt7xM1d0zC9SVRxuvMqVS_PS8",{"id":21582,"title":21583,"body":21584,"comments":609,"date":22454,"description":22455,"draft":602,"extension":605,"external":22456,"image":21566,"meta":22458,"navigation":609,"path":22459,"seo":22460,"stem":22461,"tags":22462,"__hash__":22463},"blog/2024/06/17/agents-of-inference-nvidia-and-langchain-generative-ai-agent-developer-contest.md","Agents of Inference: My submission for NVIDIA's Generative AI Agents Developer Contest by NVIDIA and LangChain",{"type":8,"value":21585,"toc":22433},[21586,21590,21595,21601,21603,21606,21609,21611,21616,21620,21623,21627,21630,21634,21637,21663,21669,21672,21676,21679,21687,21695,21702,21895,21898,21923,21926,22021,22036,22040,22043,22107,22121,22126,22129,22133,22136,22184,22188,22191,22252,22258,22261,22264,22295,22306,22310,22321,22343,22346,22349,22352,22359,22365,22369,22372,22375,22378,22382,22385,22389,22392,22396,22399,22405,22408,22412,22418,22421,22424,22427,22430],[56,21587,21589],{"id":21588},"update","Update",[11,21591,21592,21594],{},[15,21593,11825],{}," This article was published in June 2024. The models and services referenced (Meta-Llama-3-8B-Instruct, build.nvidia.com services, Stable Diffusion) may have updated versions since publication. Check official documentation for the latest releases and breaking changes.",[11,21596,21597,21598],{},"I recently posted another article about optimizing this project with TensorRT and TensorRT-LLM running on local NVIDIA NIM inference microservices, please have a look here: ",[20,21599,20909],{"href":20909,"rel":21600},[24],[56,21602,16116],{"id":16115},[11,21604,21605],{},"“Agents of Inference” is my entry for the Generative AI Agents Developer Contest by NVIDIA and LangChain. This project aims to integrate techniques for generating text, images and video to create an application capable of producing short thematic films. In this article, I will detail how I developed the project leveraging LangGraph—a library for building stateful, multi-actor applications with LLMs--and hybrid AI workflows using NVIDIA AI-powered tools and technologies running on RTX PCs and in the cloud.",[11,21607,21608],{},"Here's my project submission post on 𝕏:",[21318,21610],{},[11,21612,21322,21613,643],{},[20,21614,21327],{"href":21325,"rel":21615},[24],[56,21617,21619],{"id":21618},"nvidias-generative-ai-agents-developer-contest","NVIDIA's Generative AI Agents Developer Contest",[11,21621,21622],{},"AI agents are having a moment. They are the building blocks for building \"applications that reason\", and LangChain is a company that provides a comprehensive set of tools for developing, deploying and monitoring AI agents. I have struggled to understand how I can build or use agents in my own projects, and with the contest I have been able to just scratch the surface of what is possible with AI agents--but I think it is a promising paradigm for developing AI-driven applications.",[56,21624,21626],{"id":21625},"coming-up-with-an-idea","Coming up with an idea",[11,21628,21629],{},"I love stable diffusion. I closely follow the development of the three leading applications for generating images with stable diffusion models: Stable Diffusion WebUI, InvokeAI and ComfyUI. Write a prompt, instantly see the result, tweak the prompt and generate again. This is the basic process by which I have previously used stable diffusion. It is a satisfying mental exercise that feeds the creative and imaginative part of my brain. My idea for this project came from wanting to automate this process: use large language models to build cohesive scenes and detailed prompts and then feed them into my stable diffusion programs via API. Using LangChain and LangGraph allowed me to rapidly prototype the idea and start generating short feature films in the style of my favorite British Secret Agent: 007.",[56,21631,21633],{"id":21632},"putting-together-the-puzzle-pieces","Putting together the puzzle pieces",[11,21635,21636],{},"Here's how I set up an MVP for my project to get started. I set up a simple graph (a linked list, really) that included the following nodes. *Important: in this context, a node is an agent, and that agent is a simple Python function. It takes one parameter which is the state, a Python dictionary, that holds the output of LLM calls that the agents make. Not all nodes make LLM calls, some just run basic functions like initializing directories or calling external stable diffusion APIs.",[76,21638,21639,21642,21645,21648,21651,21654,21657,21660],{},[79,21640,21641],{},"Casting Agent → come up with some characters",[79,21643,21644],{},"Location Agent → come up with some locations",[79,21646,21647],{},"Synopsis Agent → write a synopsis based on the characters and locations",[79,21649,21650],{},"Scene Agent → write some number of scenes based on the synopsis based on the synopsis",[79,21652,21653],{},"Shot agent → describe some number of camera shots for each scene based on the scene",[79,21655,21656],{},"Photography agent → take each shot description and generate and image",[79,21658,21659],{},"Videography agent → take each image generated by the photography agent and convert it to a 4 second clip using stable video diffusion",[79,21661,21662],{},"Editor agent → compile the movie clips together",[11,21664,21665],{},[2718,21666],{"alt":21667,"src":21668},"simple graph of agents of inference","/static/aoi/graph.png",[11,21670,21671],{},"It may look simple, but there is a lot going on in this graph.",[736,21673,21675],{"id":21674},"casting-and-location","Casting and Location",[11,21677,21678],{},"The first two agents in my graph are tasked with generating characters and locations that would appear in a British secret agent film. The prompts used for these agents are as follows:",[210,21680,21681],{},[11,21682,21683,21686],{},[15,21684,21685],{},"casting",": \"Come up with four to five characters who will appear in an upcoming British spy movie. The list should include the main character who is male, the villain, an attractive female actress who eventually falls in love with the main character, and some other characters as well.\"",[210,21688,21689],{},[11,21690,21691,21694],{},[15,21692,21693],{},"locations",": \"Provide three main locations that can be used in an international British Spy movie. The locations should include a variety of cities, remote environments, iconic landmarks, etc. The locations should make for good background scenes for an action movie with lots of stunts, chases, explosions, fights, etc. and other things you would find in an action movie. Be sure to include the country and a description of the environment where these places are.\"",[11,21696,21697,21698,21701],{},"These agents leverage the LangChain Expression Language (LCEL) to generate ",[15,21699,21700],{},"structured output"," based on Pydantic models. For",[459,21703,21705],{"className":13136,"code":21704,"language":12886,"meta":464,"style":464},"class Character(BaseModel):\n    \"\"\"\n    The type for character that the casting agent casts for a role in the movie\n    \"\"\"\n    full_name: str = Field(description=\"The character's name\")\n    short_name: str = Field(description=\"The character's short name\")\n    background: str = Field(description=\"The character's background\")\n    physical_traits: str = Field(description=\"The physical traits of the character\")\n    ethnicity: str = Field(description=\"The character's ethnicity\")\n    gender: str = Field(description=\"The character's gender, either male of female\")\n    nationality: str = Field(description=\"The character's nationality\")\n    main_character: bool = Field(description=\"If the character is or is not the main character\")\n\n",[30,21706,21707,21721,21725,21730,21734,21755,21775,21795,21815,21835,21855,21875],{"__ignoreMap":464},[151,21708,21709,21711,21714,21716,21719],{"class":469,"line":470},[151,21710,16519],{"class":12347},[151,21712,21713],{"class":15254}," Character",[151,21715,12386],{"class":503},[151,21717,21718],{"class":15260},"BaseModel",[151,21720,15264],{"class":503},[151,21722,21723],{"class":469,"line":488},[151,21724,17384],{"class":481},[151,21726,21727],{"class":469,"line":500},[151,21728,21729],{"class":481},"    The type for character that the casting agent casts for a role in the movie\n",[151,21731,21732],{"class":469,"line":509},[151,21733,17384],{"class":481},[151,21735,21736,21739,21741,21743,21746,21748,21750,21753],{"class":469,"line":517},[151,21737,21738],{"class":503},"    full_name: ",[151,21740,15343],{"class":6205},[151,21742,19865],{"class":1869},[151,21744,21745],{"class":503}," Field(",[151,21747,19656],{"class":15210},[151,21749,1876],{"class":1869},[151,21751,21752],{"class":481},"\"The character's name\"",[151,21754,3640],{"class":503},[151,21756,21757,21760,21762,21764,21766,21768,21770,21773],{"class":469,"line":534},[151,21758,21759],{"class":503},"    short_name: ",[151,21761,15343],{"class":6205},[151,21763,19865],{"class":1869},[151,21765,21745],{"class":503},[151,21767,19656],{"class":15210},[151,21769,1876],{"class":1869},[151,21771,21772],{"class":481},"\"The character's short name\"",[151,21774,3640],{"class":503},[151,21776,21777,21780,21782,21784,21786,21788,21790,21793],{"class":469,"line":1413},[151,21778,21779],{"class":503},"    background: ",[151,21781,15343],{"class":6205},[151,21783,19865],{"class":1869},[151,21785,21745],{"class":503},[151,21787,19656],{"class":15210},[151,21789,1876],{"class":1869},[151,21791,21792],{"class":481},"\"The character's background\"",[151,21794,3640],{"class":503},[151,21796,21797,21800,21802,21804,21806,21808,21810,21813],{"class":469,"line":1418},[151,21798,21799],{"class":503},"    physical_traits: ",[151,21801,15343],{"class":6205},[151,21803,19865],{"class":1869},[151,21805,21745],{"class":503},[151,21807,19656],{"class":15210},[151,21809,1876],{"class":1869},[151,21811,21812],{"class":481},"\"The physical traits of the character\"",[151,21814,3640],{"class":503},[151,21816,21817,21820,21822,21824,21826,21828,21830,21833],{"class":469,"line":2462},[151,21818,21819],{"class":503},"    ethnicity: ",[151,21821,15343],{"class":6205},[151,21823,19865],{"class":1869},[151,21825,21745],{"class":503},[151,21827,19656],{"class":15210},[151,21829,1876],{"class":1869},[151,21831,21832],{"class":481},"\"The character's ethnicity\"",[151,21834,3640],{"class":503},[151,21836,21837,21840,21842,21844,21846,21848,21850,21853],{"class":469,"line":2471},[151,21838,21839],{"class":503},"    gender: ",[151,21841,15343],{"class":6205},[151,21843,19865],{"class":1869},[151,21845,21745],{"class":503},[151,21847,19656],{"class":15210},[151,21849,1876],{"class":1869},[151,21851,21852],{"class":481},"\"The character's gender, either male of female\"",[151,21854,3640],{"class":503},[151,21856,21857,21860,21862,21864,21866,21868,21870,21873],{"class":469,"line":2480},[151,21858,21859],{"class":503},"    nationality: ",[151,21861,15343],{"class":6205},[151,21863,19865],{"class":1869},[151,21865,21745],{"class":503},[151,21867,19656],{"class":15210},[151,21869,1876],{"class":1869},[151,21871,21872],{"class":481},"\"The character's nationality\"",[151,21874,3640],{"class":503},[151,21876,21877,21880,21882,21884,21886,21888,21890,21893],{"class":469,"line":2489},[151,21878,21879],{"class":503},"    main_character: ",[151,21881,17377],{"class":6205},[151,21883,19865],{"class":1869},[151,21885,21745],{"class":503},[151,21887,19656],{"class":15210},[151,21889,1876],{"class":1869},[151,21891,21892],{"class":481},"\"If the character is or is not the main character\"",[151,21894,3640],{"class":503},[11,21896,21897],{},"LCEL offers wonderful syntactic sugar, I can use this model in a parse and pip that into the output from the mode:",[459,21899,21901],{"className":13136,"code":21900,"language":12886,"meta":464,"style":464},"chain = prompt | model | parser\n",[30,21902,21903],{"__ignoreMap":464},[151,21904,21905,21908,21910,21913,21915,21918,21920],{"class":469,"line":470},[151,21906,21907],{"class":503},"chain ",[151,21909,1876],{"class":1869},[151,21911,21912],{"class":503}," prompt ",[151,21914,3947],{"class":1869},[151,21916,21917],{"class":503}," model ",[151,21919,3947],{"class":1869},[151,21921,21922],{"class":503}," parser\n",[11,21924,21925],{},"This results in our structured data:",[459,21927,21931],{"className":21928,"code":21929,"language":21930,"meta":464,"style":464},"language-yml shiki shiki-themes github-light github-dark monokai","cast:\n- background: Former MI6 agent\n  ethnicity: British\n  full_name: James Alexander\n  gender: Male\n  main_character: true\n  nationality: British\n  physical_traits: Tall, dark hair, blue eyes\n  short_name: Jamie\n","yml",[30,21932,21933,21940,21953,21963,21973,21983,21992,22001,22011],{"__ignoreMap":464},[151,21934,21935,21938],{"class":469,"line":470},[151,21936,21937],{"class":14368},"cast",[151,21939,14372],{"class":503},[151,21941,21942,21945,21948,21950],{"class":469,"line":488},[151,21943,21944],{"class":503},"- ",[151,21946,21947],{"class":14368},"background",[151,21949,6208],{"class":503},[151,21951,21952],{"class":481},"Former MI6 agent\n",[151,21954,21955,21958,21960],{"class":469,"line":500},[151,21956,21957],{"class":14368},"  ethnicity",[151,21959,6208],{"class":503},[151,21961,21962],{"class":481},"British\n",[151,21964,21965,21968,21970],{"class":469,"line":509},[151,21966,21967],{"class":14368},"  full_name",[151,21969,6208],{"class":503},[151,21971,21972],{"class":481},"James Alexander\n",[151,21974,21975,21978,21980],{"class":469,"line":517},[151,21976,21977],{"class":14368},"  gender",[151,21979,6208],{"class":503},[151,21981,21982],{"class":481},"Male\n",[151,21984,21985,21988,21990],{"class":469,"line":534},[151,21986,21987],{"class":14368},"  main_character",[151,21989,6208],{"class":503},[151,21991,14382],{"class":477},[151,21993,21994,21997,21999],{"class":469,"line":1413},[151,21995,21996],{"class":14368},"  nationality",[151,21998,6208],{"class":503},[151,22000,21962],{"class":481},[151,22002,22003,22006,22008],{"class":469,"line":1418},[151,22004,22005],{"class":14368},"  physical_traits",[151,22007,6208],{"class":503},[151,22009,22010],{"class":481},"Tall, dark hair, blue eyes\n",[151,22012,22013,22016,22018],{"class":469,"line":2462},[151,22014,22015],{"class":14368},"  short_name",[151,22017,6208],{"class":503},[151,22019,22020],{"class":481},"Jamie\n",[11,22022,22023,22024,22027,22028,22035],{},"I saved the state for all \"Agents of Inference\" invocations in the ",[30,22025,22026],{},"output"," directory of my ",[20,22029,22032],{"href":22030,"rel":22031},"https://github.com/briancaffey/agents-of-inference/tree/main/output",[24],[30,22033,22034],{},"agents-of-inference"," GitHub repo. I didn't commit the images and videos, but you can follow @AgentInference on X to see more of the results from my development process and future improvements, as well!",[736,22037,22039],{"id":22038},"synopsis-agent","Synopsis Agent",[11,22041,22042],{},"With a cast of characters and locations selected, we need a synopsis to determine what happens. Here's the prompt:",[459,22044,22046],{"className":14359,"code":22045,"language":14361,"meta":464,"style":464},"synopsis: |\n  Generate a synopsis for a British spy agent movie in the style of the James Bond series. The synopsis should include the following elements:\n  Protagonist: A charismatic and skilled British secret agent with a code name (e.g., \"Agent X\") who works for a top-secret government agency (e.g., MI6).\n  Antagonist: A formidable villain with a grand, sinister plan that threatens global security. The antagonist should have a unique, memorable persona and a well-defined motivation.\n  Mission: Outline the high-stakes mission that the protagonist must undertake to thwart the antagonist’s plan.\n  Gadgets and Vehicles: Mention the cutting-edge gadgets and vehicles that the protagonist uses throughout the mission. These should be inventive and integral to the plot.\n  Action Sequences: Include a brief description of some thrilling action sequences, such as car, boat, plane chases, hand-to-hand combat, and daring escapes, and dangerous situations.\n  Big Reveal: There is a big reveal toward the end of the storyline that is surprising and the reveal helps to move the story along.\n  Climactic Showdown: Describe the final confrontation between the protagonist and the antagonist. This should be intense and action-packed, leading to a satisfying resolution. Should include details about the main character is victorious.\n  Setting: Ensure that the settings are diverse and visually striking, adding to the overall excitement and suspense of the story. This should involve multiple locations in exotic environments, the wilderness, in dangerous situations, on board planes, trains, boats and fancy cars, etc.\n  Tone and Style: Maintain the sophisticated, suave, and adventurous tone that is characteristic of the James Bond series. Include elements of intrigue, romance, and humor.\n",[30,22047,22048,22057,22062,22067,22072,22077,22082,22087,22092,22097,22102],{"__ignoreMap":464},[151,22049,22050,22053,22055],{"class":469,"line":470},[151,22051,22052],{"class":14368},"synopsis",[151,22054,6208],{"class":503},[151,22056,20607],{"class":1869},[151,22058,22059],{"class":469,"line":488},[151,22060,22061],{"class":481},"  Generate a synopsis for a British spy agent movie in the style of the James Bond series. The synopsis should include the following elements:\n",[151,22063,22064],{"class":469,"line":500},[151,22065,22066],{"class":481},"  Protagonist: A charismatic and skilled British secret agent with a code name (e.g., \"Agent X\") who works for a top-secret government agency (e.g., MI6).\n",[151,22068,22069],{"class":469,"line":509},[151,22070,22071],{"class":481},"  Antagonist: A formidable villain with a grand, sinister plan that threatens global security. The antagonist should have a unique, memorable persona and a well-defined motivation.\n",[151,22073,22074],{"class":469,"line":517},[151,22075,22076],{"class":481},"  Mission: Outline the high-stakes mission that the protagonist must undertake to thwart the antagonist’s plan.\n",[151,22078,22079],{"class":469,"line":534},[151,22080,22081],{"class":481},"  Gadgets and Vehicles: Mention the cutting-edge gadgets and vehicles that the protagonist uses throughout the mission. These should be inventive and integral to the plot.\n",[151,22083,22084],{"class":469,"line":1413},[151,22085,22086],{"class":481},"  Action Sequences: Include a brief description of some thrilling action sequences, such as car, boat, plane chases, hand-to-hand combat, and daring escapes, and dangerous situations.\n",[151,22088,22089],{"class":469,"line":1418},[151,22090,22091],{"class":481},"  Big Reveal: There is a big reveal toward the end of the storyline that is surprising and the reveal helps to move the story along.\n",[151,22093,22094],{"class":469,"line":2462},[151,22095,22096],{"class":481},"  Climactic Showdown: Describe the final confrontation between the protagonist and the antagonist. This should be intense and action-packed, leading to a satisfying resolution. Should include details about the main character is victorious.\n",[151,22098,22099],{"class":469,"line":2471},[151,22100,22101],{"class":481},"  Setting: Ensure that the settings are diverse and visually striking, adding to the overall excitement and suspense of the story. This should involve multiple locations in exotic environments, the wilderness, in dangerous situations, on board planes, trains, boats and fancy cars, etc.\n",[151,22103,22104],{"class":469,"line":2480},[151,22105,22106],{"class":481},"  Tone and Style: Maintain the sophisticated, suave, and adventurous tone that is characteristic of the James Bond series. Include elements of intrigue, romance, and humor.\n",[11,22108,22109,22110,22113,22114,22117,22118,22120],{},"The synopsis to any good film is key, so I decided to use a feature of LangGraph that would allow a ",[30,22111,22112],{},"synopsis_review_agent"," to provide multiple rounds of feedback to the ",[30,22115,22116],{},"synopsis_agent"," to make it even better. Here's what the new graph look like after implementing the ",[30,22119,22112],{}," using conditional graph edges:",[11,22122,22123],{},[2718,22124],{"alt":22112,"src":22125},"/static/aoi/graph_with_cycle.png",[11,22127,22128],{},"Conditional edges are a very powerful feature and I just used it in one part of my graph. Other parts of the graph could benefit from this as well, and they can allow for \"human-in-the-loop\" interactions which are becoming very popular in AI-powered applications.",[736,22130,22132],{"id":22131},"scene-and-shot-agents","Scene and shot agents",[11,22134,22135],{},"With our perfected synopsis, we are ready to put more agents to work. The scene agent builds out the basic structure of the storyline. It provides a structured list of the main sections of the movie. The shot agent then loops over the scenes and creates a number of different shots for the given scene. This was an effective way to have consistent thematic content for shots within a scene. Here are the prompts I used for the scene and shot agents:",[459,22137,22139],{"className":14359,"code":22138,"language":14361,"meta":464,"style":464},"scenes: |\n  Create a list of detailed scenes for an exciting and entertaining British spy film. The scenes should be comprehensive and include all scenes necessary for a complete film. Each scene should include the following elements:\n  Location: Describe the location and setting of the scene, including any notable landmarks, time of day, and general atmosphere.\n  Characters Involved: List the main characters present in the scene, with a brief description of their roles and appearances.\n  Description of What Happens: Provide a detailed account of the action, and key events that take place in the scene.\nshot: |\n  You are a film director working on a new British spy film and your writers have provided you with a scene. Your task is to come up with four to five shots that will be filmed during the scene. The shot descriptions needs to be specific and should include a variety of closeup shots on characters, environment shots that consider the scene location and shots of specific items or other things that are featured in the scene. Each shot should also have a title. The description should be a brief densely worded block of text that captures the important elements of the scene. Consider the style of camera angle, lighting, character expressions, clothing, and other important visual elements for each shot. Be very descriptive. The description will be used to generate an image of the shot. Also, there should be at most one actor for each shot that contains people. Don't use the name of the character, instead use a physical description of the character based on their physical traits described below if needed. Also consider what the actor is wearing in the description.\n",[30,22140,22141,22150,22155,22160,22165,22170,22179],{"__ignoreMap":464},[151,22142,22143,22146,22148],{"class":469,"line":470},[151,22144,22145],{"class":14368},"scenes",[151,22147,6208],{"class":503},[151,22149,20607],{"class":1869},[151,22151,22152],{"class":469,"line":488},[151,22153,22154],{"class":481},"  Create a list of detailed scenes for an exciting and entertaining British spy film. The scenes should be comprehensive and include all scenes necessary for a complete film. Each scene should include the following elements:\n",[151,22156,22157],{"class":469,"line":500},[151,22158,22159],{"class":481},"  Location: Describe the location and setting of the scene, including any notable landmarks, time of day, and general atmosphere.\n",[151,22161,22162],{"class":469,"line":509},[151,22163,22164],{"class":481},"  Characters Involved: List the main characters present in the scene, with a brief description of their roles and appearances.\n",[151,22166,22167],{"class":469,"line":517},[151,22168,22169],{"class":481},"  Description of What Happens: Provide a detailed account of the action, and key events that take place in the scene.\n",[151,22171,22172,22175,22177],{"class":469,"line":534},[151,22173,22174],{"class":14368},"shot",[151,22176,6208],{"class":503},[151,22178,20607],{"class":1869},[151,22180,22181],{"class":469,"line":1413},[151,22182,22183],{"class":481},"  You are a film director working on a new British spy film and your writers have provided you with a scene. Your task is to come up with four to five shots that will be filmed during the scene. The shot descriptions needs to be specific and should include a variety of closeup shots on characters, environment shots that consider the scene location and shots of specific items or other things that are featured in the scene. Each shot should also have a title. The description should be a brief densely worded block of text that captures the important elements of the scene. Consider the style of camera angle, lighting, character expressions, clothing, and other important visual elements for each shot. Be very descriptive. The description will be used to generate an image of the shot. Also, there should be at most one actor for each shot that contains people. Don't use the name of the character, instead use a physical description of the character based on their physical traits described below if needed. Also consider what the actor is wearing in the description.\n",[736,22185,22187],{"id":22186},"stable-diffusion-and-stable-video-diffusion-agents","Stable Diffusion and Stable Video Diffusion agents",[11,22189,22190],{},"The stable diffusion agent makes an API call to a local instance of the Stable Diffusion WebUI API, saves the generated image and saves a reference to that image in the state:",[459,22192,22194],{"className":14359,"code":22193,"language":14361,"meta":464,"style":464},"- description: A medium close-up shot of Ethan Jameson's face, with a concerned expression,\n    as he reads the message from Natalie Jackson. The lighting is dim, with only a\n    single lamp on his desk casting a warm glow. His eyes are narrowed, and his brow\n    is furrowed in concentration. He is wearing a dark blue suit and a white shirt.\n  image: 000.png\n  title: Ethan's Concerned Expression\n  video: 000.mp4\n",[30,22195,22196,22207,22212,22217,22222,22232,22242],{"__ignoreMap":464},[151,22197,22198,22200,22202,22204],{"class":469,"line":470},[151,22199,21944],{"class":503},[151,22201,19656],{"class":14368},[151,22203,6208],{"class":503},[151,22205,22206],{"class":481},"A medium close-up shot of Ethan Jameson's face, with a concerned expression,\n",[151,22208,22209],{"class":469,"line":488},[151,22210,22211],{"class":481},"    as he reads the message from Natalie Jackson. The lighting is dim, with only a\n",[151,22213,22214],{"class":469,"line":500},[151,22215,22216],{"class":481},"    single lamp on his desk casting a warm glow. His eyes are narrowed, and his brow\n",[151,22218,22219],{"class":469,"line":509},[151,22220,22221],{"class":481},"    is furrowed in concentration. He is wearing a dark blue suit and a white shirt.\n",[151,22223,22224,22227,22229],{"class":469,"line":517},[151,22225,22226],{"class":14368},"  image",[151,22228,6208],{"class":503},[151,22230,22231],{"class":481},"000.png\n",[151,22233,22234,22237,22239],{"class":469,"line":534},[151,22235,22236],{"class":14368},"  title",[151,22238,6208],{"class":503},[151,22240,22241],{"class":481},"Ethan's Concerned Expression\n",[151,22243,22244,22247,22249],{"class":469,"line":1413},[151,22245,22246],{"class":14368},"  video",[151,22248,6208],{"class":503},[151,22250,22251],{"class":481},"000.mp4\n",[11,22253,22254],{},[2718,22255],{"alt":22256,"src":22257},"A medium close-up shot of Ethan Jameson's face","/static/aoi/ethan.png",[11,22259,22260],{},"With the perfectly prompted image in hand, we can use Stable Video Diffusion to bring it to life. I prompted phind to come up with a FastAPI service that would accept an image in a post request and return a short video created with stable video diffusion using the diffusers library.",[11,22262,22263],{},"Stable video diffusion can generate about 4 seconds of text at 7 frames per second. This isn't great, but I was able to use ffmpeg to do frame interpolation bringing the frame rate to a much smoother 14 fps using motion compensated interpolation (MCI):",[459,22265,22267],{"className":461,"code":22266,"language":463,"meta":464,"style":464},"ffmpeg -i output/1718453390/final.mp4 -crf 10 -vf \"minterpolate=fps=14:mi_mode=mci:mc_mode=aobmc:me_mode=bidir:vsbmc=1\" output/1718453390/final.14fps.mp4\n",[30,22268,22269],{"__ignoreMap":464},[151,22270,22271,22274,22277,22280,22283,22286,22289,22292],{"class":469,"line":470},[151,22272,22273],{"class":473},"ffmpeg",[151,22275,22276],{"class":477}," -i",[151,22278,22279],{"class":481}," output/1718453390/final.mp4",[151,22281,22282],{"class":477}," -crf",[151,22284,22285],{"class":477}," 10",[151,22287,22288],{"class":477}," -vf",[151,22290,22291],{"class":481}," \"minterpolate=fps=14:mi_mode=mci:mc_mode=aobmc:me_mode=bidir:vsbmc=1\"",[151,22293,22294],{"class":481}," output/1718453390/final.14fps.mp4\n",[11,22296,22297,22298,22301,22302,22305],{},"Finally, the ",[30,22299,22300],{},"editor_agent"," uses ",[30,22303,22304],{},"moviepy"," to join the clips together into a single video.",[56,22307,22309],{"id":22308},"development-environment","Development environment",[11,22311,22312,22313,22316,22317,22320],{},"I struggled to optimize the ",[30,22314,22315],{},"meta-llama/Meta-Llama-3-8B-Instruct"," with TensorRT-LLM, so I ran LLM inference on a combination of older Llama2 TensorRT-LLM models, and ",[30,22318,22319],{},"Meta-Llama-3-8B-Instruct"," on LM Studio (which I found to be painfully slow compared to TensorRT-LLM).",[11,22322,22323,22324,22326,22327,22329,22330,22333,22334,22339,22340,22342],{},"If you provide an ",[30,22325,13411],{}," in the ",[30,22328,11004],{}," file, LLM calls will be made using the ",[30,22331,22332],{},"meta/llam3-70b-instruct"," model on ",[20,22335,22338],{"href":22336,"rel":22337},"https://build.nvidia.com/meta/llama3-70b",[24],"build.nvidia.com/meta/llama3-70b",". In fact, ",[30,22341,16210],{}," also provides stable diffusion and stable video diffusion inference via API. This would be very convenient in the event that my RTX PCs become compromised.",[11,22344,22345],{},"My RTX 4090 GPU with 24 GB of memory was able to run lots of different inference servers concurrently (LLM, Stable Diffusion WebUI, ComfyUI, InvokeAI, Stable Video Diffusion FastAPI service), but I generally stuck to doing one type of inference at a time, otherwise things would grind to a hault or crash. I also experimented with ChatTTS, a new text-to-speech model.",[11,22347,22348],{},"I developed this project on a MacBook Pro, and I used my RTX PC as if it were a remote service providing inference for text, images and video. This is a helpful mindset when working with hybrid AI workflows that leverage inference services both on local machines and in the cloud.",[56,22350,22351],{"id":13562},"How it works",[11,22353,22354,22355,22358],{},"To run the program, you need to install python dependencies and then run an OpenAI compatible LLM and Stable Diffusion WebUI server with the ",[30,22356,22357],{},"--api"," flag. You also need to run the Stable Video Diffusion service. Apologies for any hardcoded local IP address in the source code. Deadlines, you know! With everything configured, you can run the following command:",[459,22360,22363],{"className":22361,"code":22362,"language":997},[995],"~/git/github/agents-of-inference$ poetry run python agents_of_inference/main.py\n## 📀 Using local models 📀 ##\n## 🎭 Generating Cast 🎭 ##\n## 🗺️ Generating Locations 🗺️ ##\n## ✍️ Generating Synopsis ✍️ ##\n## going to synopsis_review_agent ##\n## 📑 Reviewing Synopsis 📑 ##\n## ✍️ Generating Synopsis ✍️ ##\n## going to synopsis_review_agent ##\n## 📑 Reviewing Synopsis 📑 ##\n## ✍️ Generating Synopsis ✍️ ##\n## going to scene_agent ##\n## 📒 Generating Scenes 📒 ##\n## 🎬 Generating Shots 🎬 ##\n## Generated 5 shots for scene 1/5 ##\n## Generated 5 shots for scene 2/5 ##\n## Generated 5 shots for scene 3/5 ##\n## Generated 5 shots for scene 4/5 ##\n## Generated 5 shots for scene 5/5 ##\n\n000/0025\nA medium shot of a bustling Tokyo street, with neon lights reflecting off wet pavement. Jim Thompson, dressed in a black leather jacket and dark jeans, walks purposefully through the crowd, his piercing blue eyes scanning the area. The sound design features the hum of traffic and chatter of pedestrians.\nGenerated image output/1718426686/images/000.png\n\n001/0025\nA tight close-up shot of Emily Chen's face, her piercing brown eyes intense as she briefs Jim on the situation. Her short black hair is styled neatly, and she wears a crisp white blouse with a silver necklace. The camera lingers on her lips as she speaks, emphasizing the importance of the information.\nGenerated image output/1718426686/images/001.png\n\nGenerated video output/1718426686/videos/000.mp4\n== stable video diffusion generation complete ==\nGenerated video output/1718426686/videos/001.mp4\n== stable video diffusion generation complete ==\n",[30,22364,22362],{"__ignoreMap":464},[56,22366,22368],{"id":22367},"demo-video-for-contest-submission","Demo Video for Contest Submission",[22370,22371],"agents-of-inference-video",{},[11,22373,22374],{},"Making this video was a lot of fun. The \"Agents of Inference\" highlight reel includes some of the most interesting, exciting and fun clips that I found in the dozens of short films it created. It is important to note that a lot of the content is not very good. Misunderstood prompts, color confusion (prompt includes green eyes, but other things in the scene are also conspicuously green), unrealistic or noisy motion from Stable Video Diffusion--these are some of the issues you will find in the films. Generating AI images sometimes feels like panning for gold: you go through a lot of sediment to get a few good flakes.",[11,22376,22377],{},"Also, I added a few short animations that I made with Blender. The final scene shows the NVIDIA Omniverse orange humanoid from the barrel of a pistol. I think we are rapidly approaching a future where agents can generate full-scale theatrical movies by generating OpenUSD code, directly or indirectly. Maybe for the next NVIDIA Developer contest!",[56,22379,22381],{"id":22380},"shortcomings-of-my-project","Shortcomings of my project",[11,22383,22384],{},"My goodness, how embarrassing. There are quite a few shortcomings that can be easily identified looking over the output and the source code. Here are a few:",[736,22386,22388],{"id":22387},"character-variety","Character variety",[11,22390,22391],{},"When generating characters I would frequently see one named Dr. Sophia Patel who is apparently a brilliant cryptologist. Other characters would often have different names or backgrounds, but I saw Dr. Sophia Patel more often than not.",[736,22393,22395],{"id":22394},"character-consistency","Character consistency",[11,22397,22398],{},"The characters are not consistent. This is a notoriously difficult problem to solve, but I made a lot of progress on it during this contest. I experimented with calling the ComfyUI API to run a custom workflow built with the ComfyUI graph-based workflow tool for face transfer:",[11,22400,22401],{},[2718,22402],{"alt":22403,"src":22404},"Dr. Sophia Patel","/static/aoi/sophia.png",[11,22406,22407],{},"Using ComfyUI would be nice, but it wouldn't be as easy to tap into cloud APIs if my workflow heavily relied on ComfyUI server with custom models.",[736,22409,22411],{"id":22410},"understanding-langchain","Understanding LangChain",[11,22413,22414,22415,22417],{},"I started out with the idea I would store all LLM calls to a local JSON to serve as a cache, allowing me to avoid regenerating responses from early in the workflow. This worked well, until I tried to serialize an Annotated list (required for cycles such as the one used with ",[30,22416,22112],{},"). I ended up wasting a lot of time trying to figure this out, and I came across some built-in LangChain features for storing state in memory and in Sqlite. I'm sure there are other areas where I used the wrong pattern, but I turned over a lot of stones and look forward to continuing development with LangChain.",[56,22419,22420],{"id":16069},"What's next?",[11,22422,22423],{},"Thank you to NVIDIA and LangChain for organizing this contest. It was a great way to explore a powerful toolset for automated content generation using AI agents.",[11,22425,22426],{},"Video models like Dream Machine and Sora have made some big splashes on the internet and the results are remarkable. However, I'm still almost more interested in finding the limitations of quality content using open-source models on consumer hardware like RTX GPUs.",[11,22428,22429],{},"I would also have loved to generate my own music for these films. I am a Suno poweruser and love the songs I have generated on that site. Will the gap between video and music generation on private, payed services and local machines? Or does it just need time to catch up? Hopefully a future installment of \"Agents of Inference\" will integrate music and voice, and can't wait to hear it!",[589,22431,22432],{},"html pre.shiki code .sq6CD, html code.shiki .sq6CD{--shiki-default:#D73A49;--shiki-default-font-style:inherit;--shiki-dark:#F97583;--shiki-dark-font-style:inherit;--shiki-sepia:#66D9EF;--shiki-sepia-font-style:italic}html pre.shiki code .sz2Vg, html code.shiki .sz2Vg{--shiki-default:#6F42C1;--shiki-default-text-decoration:inherit;--shiki-dark:#B392F0;--shiki-dark-text-decoration:inherit;--shiki-sepia:#A6E22E;--shiki-sepia-text-decoration:underline}html pre.shiki code .sMOD_, html code.shiki .sMOD_{--shiki-default:#24292E;--shiki-dark:#E1E4E8;--shiki-sepia:#F8F8F2}html pre.shiki code .s30JN, html code.shiki .s30JN{--shiki-default:#6F42C1;--shiki-default-font-style:inherit;--shiki-default-text-decoration:inherit;--shiki-dark:#B392F0;--shiki-dark-font-style:inherit;--shiki-dark-text-decoration:inherit;--shiki-sepia:#A6E22E;--shiki-sepia-font-style:italic;--shiki-sepia-text-decoration:underline}html pre.shiki code .sstjo, html code.shiki .sstjo{--shiki-default:#032F62;--shiki-dark:#9ECBFF;--shiki-sepia:#E6DB74}html pre.shiki code .s-m8C, html code.shiki .s-m8C{--shiki-default:#005CC5;--shiki-default-font-style:inherit;--shiki-dark:#79B8FF;--shiki-dark-font-style:inherit;--shiki-sepia:#66D9EF;--shiki-sepia-font-style:italic}html pre.shiki code .sC2Qs, html code.shiki .sC2Qs{--shiki-default:#D73A49;--shiki-dark:#F97583;--shiki-sepia:#F92672}html pre.shiki code .sTHNf, html code.shiki .sTHNf{--shiki-default:#E36209;--shiki-default-font-style:inherit;--shiki-dark:#FFAB70;--shiki-dark-font-style:inherit;--shiki-sepia:#FD971F;--shiki-sepia-font-style:italic}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html .sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html.sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html pre.shiki code .s5clZ, html code.shiki .s5clZ{--shiki-default:#22863A;--shiki-dark:#85E89D;--shiki-sepia:#F92672}html pre.shiki code .s7F3e, html code.shiki .s7F3e{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#AE81FF}html pre.shiki code .srTi1, html code.shiki .srTi1{--shiki-default:#6F42C1;--shiki-dark:#B392F0;--shiki-sepia:#A6E22E}",{"title":464,"searchDepth":488,"depth":488,"links":22434},[22435,22436,22437,22438,22439,22445,22446,22447,22448,22453],{"id":21588,"depth":488,"text":21589},{"id":16115,"depth":488,"text":16116},{"id":21618,"depth":488,"text":21619},{"id":21625,"depth":488,"text":21626},{"id":21632,"depth":488,"text":21633,"children":22440},[22441,22442,22443,22444],{"id":21674,"depth":500,"text":21675},{"id":22038,"depth":500,"text":22039},{"id":22131,"depth":500,"text":22132},{"id":22186,"depth":500,"text":22187},{"id":22308,"depth":488,"text":22309},{"id":13562,"depth":488,"text":22351},{"id":22367,"depth":488,"text":22368},{"id":22380,"depth":488,"text":22381,"children":22449},[22450,22451,22452],{"id":22387,"depth":500,"text":22388},{"id":22394,"depth":500,"text":22395},{"id":22410,"depth":500,"text":22411},{"id":16069,"depth":488,"text":22420},"2024-06-17","This article discusses my entry for NVIDIA's Generative AI Agents Developer Contest entry: Agents of Inference",[22457],{"link":21565,"site":11126},{},"/2024/06/17/agents-of-inference-nvidia-and-langchain-generative-ai-agent-developer-contest",{"title":21583,"description":22455},"2024/06/17/agents-of-inference-nvidia-and-langchain-generative-ai-agent-developer-contest",[11133,21572,1822,14055,21573,21574,19403,614,615,21575,21576,21577,21578,21579],"VJuAFkVmecjYN7UIZiuqj73EfbFR0cBEPLkb34Z_2ps",{"id":22465,"title":22466,"body":22467,"comments":609,"date":23762,"description":23763,"draft":602,"extension":605,"external":23764,"image":23772,"meta":23773,"navigation":609,"path":23774,"seo":23775,"stem":23776,"tags":23777,"__hash__":23780},"blog/2024/02/17/rocket-league-botchat-nvidia-generative-ai-on-rtx-pcs-developer-contest.md","Rocket League BotChat powered by TensorRT-LLM: My submission for NVIDIA's Generative AI on RTX PCs Developer Contest",{"type":8,"value":22468,"toc":23739},[22469,22471,22476,22479,22481,22484,22491,22495,22498,22503,22506,22509,22512,22515,22521,22524,22527,22532,22537,22540,22546,22549,22552,22555,22558,22564,22566,22569,22638,22649,22656,22675,22678,22686,22701,22704,22710,22713,22782,22794,22799,22809,22837,22840,22860,22862,22865,22871,22905,22909,22915,23078,23082,23085,23158,23165,23168,23174,23180,23183,23186,23192,23194,23200,23204,23211,23302,23306,23313,23423,23427,23430,23464,23470,23490,23497,23501,23504,23510,23514,23517,23611,23622,23626,23629,23632,23635,23637,23640,23643,23649,23681,23687,23692,23698,23704,23706,23712,23715,23718,23725,23727,23730,23733,23736],[56,22470,16116],{"id":16115},[11,22472,22473,22475],{},[15,22474,11825],{}," This article was published in February 2024 (~13 months old). The TensorRT-LLM versions referenced (v0.6.1/v0.7.1) and Llama-2 models mentioned may have updated versions since publication (Llama-3 is now available). Check official documentation for the latest releases and breaking changes.",[11,22477,22478],{},"This article is about my submission to NVIDIA's Generative AI on RTX PCs Developer Contest: Rocket League BotChat. Rocket League BotChat is a BakkesMod plugin for Rocket League that allows bots to send chat messages based on in-game events. It is designed to be used with a local LLM service optimized and accelerated with NVIDIA's TensorRT-LLM library.",[11,22480,21608],{},[22482,22483],"rocket-league-bot-chat-tweet",{},[11,22485,21322,22486,643],{},[20,22487,22490],{"href":22488,"rel":22489},"https://github.com/briancaffey/RocketLeagueBotChat",[24],"Rocket League BotChat GitHub repository",[56,22492,22494],{"id":22493},"nvidias-gen-ai-developer-contest","NVIDIA's Gen AI Developer Contest",[11,22496,22497],{},"The following email caught my attention last month:",[210,22499,22500],{},[11,22501,22502],{},"Generative AI on RTX PCs Developer Contest: Build your next innovative Gen AI project using NVIDIA TensorRT or TensorRT-LLM on Windows PC with NVIDIA RTX systems",[11,22504,22505],{},"The part about “on Windows PC” made me think: why would a developer contest focus on a particular operating system? I use all three of the major operating systems: macOS, Ubuntu and Windows 11, but most of the development work I do is on macOS and Ubuntu. I discovered WSL (Windows Subsystem for Linux) a few years ago and really enjoy using that for development as well, but I had never considered doing development work on Windows outside of WSL. I had also never used any of the Windows-specific development frameworks like .NET or Visual Studio.",[11,22507,22508],{},"My experience with Windows goes back to 2016 when I built my first PC with an NVIDIA GeForce GTX 1080 graphics card. When I built another personal computer last year in 2023, getting the NVIDIA GeForce RTX 4090 graphics card was a big step up. I bought two NVMe drives in order to dual boot into both Windows and Ubuntu operating systems. Switching between the operating systems requires turning off the computer, going into the BIOS settings and changing the boot order and restarting the computer.",[11,22510,22511],{},"Last year I started learning more about AI image generation using Stable Diffusion with programs like Automatic1111, InvokeAI and ComfyUI. I set up everything on my PC's Ubuntu operating system, and frequently had to switch between using Ubuntu for working with stable diffusion and Windows for gaming and other Windows-specific software. The friction of having to constantly switch operating systems pushed me to move my stable diffusion software workflows to Windows. All of my models and images are stored to external drives, so moving things over to Windows was pretty easy.",[11,22513,22514],{},"I learned PowerShell and got more familiar with how Windows works as a development machine. Environment variables and system variables are one example of how Windows does things differently compared to Linux-based operating systems. And just like that, I became a Windows developer! This experience got me interested in coming up with an idea for the NVIDIA Generative AI on NVIDIA RTX PCs Developer Contest.",[11,22516,22517],{},[2718,22518],{"alt":22519,"src":22520},"Windows winfetch screenshot","/static/rlbc/winfetch.png",[56,22522,22523],{"id":21625},"Coming up with an Idea",[11,22525,22526],{},"The contest description and some related NVIDIA articles about the contest helped me with brainstorming:",[210,22528,22529],{},[11,22530,22531],{},"Whether it’s a RAG-based chatbot, a plug-in for an existing application, or a code generation tool, the possibilities are endless.",[210,22533,22534],{},[11,22535,22536],{},"Many use cases would benefit from running LLMs locally on Windows PCs, including gaming, creativity, productivity, and developer experiences.",[11,22538,22539],{},"This contest is focused on NVIDIA's consumer hardware line: GeForce RTX. It has a diverse set of use cases including gaming, crypto mining, VR, simulation software, creative tools and new AI techniques including image generation and LLM (Large Language Model) inference.",[11,22541,22542],{},[2718,22543],{"alt":22544,"src":22545},"A stacked bar chart showing the composition of Nvidia's revenue each quarter going back to fiscal 2019.","https://g.foolcdn.com/image/?url=https%3A%2F%2Fg.foolcdn.com%2Feditorial%2Fimages%2F764886%2Fnvda_revenue_bar.png&op=resize&w=700",[11,22547,22548],{},"Gaming seemed like an interesting avenue for me to explore. PC gaming is still an industry that is developed primarily for Windows operating systems, and the gaming industry has been the largest revenue driver of NVIDIA in recent years, only recently surpassed by the data center segment. GPUs are needed to render graphics of enormous open-world environments. Some story-driven games include huge amounts of dialogue that can be considered as huge literary works in their own right. Red Dead Redemption and Genshin Impact are two massively popular games of this type. There might be an interesting project idea that could use LLMs and RAG (retrieval augmented generation), but I don't play these types of games and it didn't seem practical for a project that would be built in just over a month. I thought about trying to build something for a simpler game that I already know.",[11,22550,22551],{},"Rocket League is a vehicular soccer game that is played on both game consoles and on PCs. It is an eSport with a very high skill ceiling and a massive player base (85 million active players in the last 30 days). I started playing it during the pandemic with some of my friends and all got hooked. We also came to learn that Rocket League's in-game chat varies from entertaining, annoying, toxic and in some cases, sportsmanlike.",[11,22553,22554],{},"One other thing I learned about Rocket League is that it has an active modding community. Developers create plugins for the game for all different purposes, such as coaching, practice drills, capturing replays, tracking player statistics, etc. Most Rocket League Mods are written in a popular framework called Bakkesmod (developed by Andreas \"bakkes\" Bakke, a Norwegian software engineer). Rocket League's in-game chat inspired the idea for my submission to NVIDIA's Generative AI Developer Contest: Rocket League BotChat. The idea for my project is to build a plugin with Bakkesmod that allows Rocket League bots to send chat messages based on game events using an LLM accelerated and optimized by TensorRT-LLM (more on TensorRT-LLM soon!)",[11,22556,22557],{},"Bots are built into the Rocket League game and you can play with or against them in offline matches. However, the built-in bots are not very good. Another 3rd-party project called RLBot allows players to play against community-developed AI bots that are developed with machine learning frameworks like TensorFlow and PyTorch. These bots are very good, but they are not infallible. My contest project idea was now clear: develop a plugin for Rocket League capable of sending messages from bot players. This idea seemed to check the boxes for the large language model category of NVIDIA's developer contest: develop a project in a Windows environment for a Windows-specific program, and use an LLM powered by TensorRT-LLM.",[11,22559,22560],{},[2718,22561],{"alt":22562,"src":22563},"RLBot Ascii Art","/static/rlbc/bot.png",[56,22565,21633],{"id":21632},[11,22567,22568],{},"With this idea in mind, I looked into the project's feasibility. I really had no idea if this would work. I looked through the Bakkesmod documentation and found some helpful resources that gave me confidence that I could pull something together for at least a proof-of-concept.",[76,22570,22571,22600,22612,22619],{},[79,22572,22573,22574,22578],{},"The Bakkesmod Plugin Wiki ",[20,22575,22576],{"href":22576,"rel":22577},"https://wiki.bakkesplugins.com/",[24],[76,22579,22580,22590],{},[79,22581,22582,22589],{},[20,22583,22586],{"href":22584,"rel":22585},"https://wiki.bakkesplugins.com/code_snippets/using_http_wrapper/",[24],[30,22587,22588],{},"HttpWrapper"," for sending HTTP requests from Bakkesmod",[79,22591,22592,22599],{},[20,22593,22596],{"href":22594,"rel":22595},"https://wiki.bakkesplugins.com/functions/stat_events/",[24],[30,22597,22598],{},"StatEvents"," that allow for running custom code when specific event functions are triggered in the game (such as scoring a goal, or making a save).",[79,22601,22602,22603,22607],{},"The Bakkesmod plugin template: ",[20,22604,22605],{"href":22605,"rel":22606},"https://github.com/Martinii89/BakkesmodPluginTemplate",[24],[76,22608,22609],{},[79,22610,22611],{},"This provides a great starting-off point for developing Bakkesmod plugins. Plugins for Bakkesmod are written in C++ and this repo provides an organized file structure that allows you to get started quickly",[79,22613,22614,22615],{},"Plugin Tutorial: ",[20,22616,22617],{"href":22617,"rel":22618},"https://wiki.bakkesplugins.com/plugin_tutorial/getting_started/",[24],[79,22620,22621,22622],{},"Open-source chat-related Bakkesmod plugins on GitHub\n",[76,22623,22624,22631],{},[79,22625,22626,22627],{},"BetterChat: ",[20,22628,22629],{"href":22629,"rel":22630},"https://github.com/JulienML/BetterChat",[24],[79,22632,22633,22634],{},"Translate: ",[20,22635,22636],{"href":22636,"rel":22637},"https://github.com/0xleft/trnslt",[24],[11,22639,22640,22641,22644,22645,22648],{},"Starting with the Plugin Template, I wrote a simple console command that when triggered sends an HTTP request to ",[30,22642,22643],{},"localhost:8000/hello",". I set up a Hello World Flask app running on ",[30,22646,22647],{},"localhost:8000"," and I was able to get a response from my Hello World server! There didn't seem to be any network or permission errors that would prevent my game code from communicating with other applications on my PC.",[11,22650,22651,22652,22655],{},"Next I started looking into how to build and run optimized LLMs with NVIDIA's TensorRT-LLM library, the software that this contest is promoting. The contest announcement included an interesting building block that I thought could be very useful: an example repo showing how to run ",[30,22653,22654],{},"CodeLlama-13b-instruct-hf"," optimized by TensorRT-LLM to provide inference for a VSCode extension called Continue (Continue.dev).",[76,22657,22658,22663,22669,22672],{},[79,22659,22660,22662],{},[30,22661,22654],{}," is an open source model from Meta that is trained on code and can help with code generation tasks",[79,22664,22665,22666,22668],{},"TensorRT-LLM is a Python library that accelerates and optimizes inference performance of large language models. It takes a Large Language Model like ",[30,22667,22654],{}," and generates an engine that can be used for doing inference",[79,22670,22671],{},"VSCode is an open source code editor developed by Microsoft with an large number of community plugins",[79,22673,22674],{},"Continue.dev is a startup backed by Y Combinator that is developing an open-source autopilot (code assistant) for VSCode and JetBrains that works with local LLMs or paid services like ChatGPT",[11,22676,22677],{},"To get the coding assistant project working I needed to build the TensorRT-LLM engine. Building TensorRT-LLM engines on Windows can be done in one of two ways:",[76,22679,22680,22683],{},[79,22681,22682],{},"using a \"bare-metal\" virtual environment on Windows (with PowerShell)",[79,22684,22685],{},"using WSL",[11,22687,22688,22689,22692,22693,22696,22697,22700],{},"At the time of writing, building a TensorRT-LLM engine on Windows can only be done with version ",[30,22690,22691],{},"v0.6.1"," of the TensorRT-LLM repo and version ",[30,22694,22695],{},"v0.7.1"," of the ",[30,22698,22699],{},"tensorrt_llm"," Python package.",[11,22702,22703],{},"With WSL you can use the up-to-date versions of the TensorRT-LLM repo (main branch). The engines produced by Windows and WSL (Ubuntu) are not interchangeable and you will get errors if you try to use an engine created with one operating system on another operating system.",[11,22705,22706,22707,22709],{},"Once the engines are built you can use them to run the example from the ",[30,22708,21404],{}," repo.",[11,22711,22712],{},"The example repo exposes an OpenAI-compatible API locally that can do chat completions. You then need to configure the Continue.dev extension to use the local LLM service:",[459,22714,22716],{"className":6194,"code":22715,"language":6196,"meta":464,"style":464},"{\n  \"title\": \"CodeLlama-13b-instruct-hf\",\n  \"apiBase\": \"http://192.168.5.96:5000/\",\n  \"provider\": \"openai\",\n  \"apiKey\": \"None\",\n  \"model\": \"gpt-4\"\n}\n",[30,22717,22718,22722,22734,22746,22758,22769,22778],{"__ignoreMap":464},[151,22719,22720],{"class":469,"line":470},[151,22721,12966],{"class":503},[151,22723,22724,22727,22729,22732],{"class":469,"line":488},[151,22725,22726],{"class":6205},"  \"title\"",[151,22728,6208],{"class":503},[151,22730,22731],{"class":6211},"\"CodeLlama-13b-instruct-hf\"",[151,22733,9417],{"class":503},[151,22735,22736,22739,22741,22744],{"class":469,"line":500},[151,22737,22738],{"class":6205},"  \"apiBase\"",[151,22740,6208],{"class":503},[151,22742,22743],{"class":6211},"\"http://192.168.5.96:5000/\"",[151,22745,9417],{"class":503},[151,22747,22748,22751,22753,22756],{"class":469,"line":509},[151,22749,22750],{"class":6205},"  \"provider\"",[151,22752,6208],{"class":503},[151,22754,22755],{"class":6211},"\"openai\"",[151,22757,9417],{"class":503},[151,22759,22760,22763,22765,22767],{"class":469,"line":517},[151,22761,22762],{"class":6205},"  \"apiKey\"",[151,22764,6208],{"class":503},[151,22766,6262],{"class":6211},[151,22768,9417],{"class":503},[151,22770,22771,22773,22775],{"class":469,"line":534},[151,22772,9471],{"class":6205},[151,22774,6208],{"class":503},[151,22776,22777],{"class":6211},"\"gpt-4\"\n",[151,22779,22780],{"class":469,"line":1413},[151,22781,6274],{"class":503},[11,22783,22784,22785,22787,22788,22793],{},"The Continue.dev extension using ",[30,22786,22654],{}," accelerated and optimized by TensorRT-LLM is very fast. According to ",[20,22789,22792],{"href":22790,"rel":22791},"https://blog.continue.dev/programming-languages/",[24],"this post on Continue.dev's blog",", C++ is a \"first tier\" language:",[210,22795,22796],{},[11,22797,22798],{},"C++ has one of the largest presences on GitHub and Stack Overflow. This shows up in its representation in public LLM datasets, where it is one of the languages with the most data. Its performance is near the top of the MultiPL-E, BabelCode / TP3, MBXP / Multilingual HumanEval, and HumanEval-X benchmarks. However, given that C++ is often used when code performance and exact algorithm implementation is very important, many developers don’t believe that LLMs are as helpful for C++ as some of the other languages in this tier.",[11,22800,22801,22802,22805,22806,22808],{},"Most of the time I'm working with either Python and TypeScript. I've read about C++ but haven't used it for anything before doing this project. I primarily used Microsoft Visual Studio to build the plugin, but VSCode with the Continue.dev autopilot extension was helpful for tackling smaller problems in a REPL-like environment. For example, I used Continue.dev in VSCode to figure out how to handle JSON. Coming from Python and JavaScript languages, I found the ",[30,22803,22804],{},"nlohmann/json"," JSON library syntax to be somewhat different. For example, here is how to add a message to ",[30,22807,8360],{}," in the body of an OpenAI API request:",[459,22810,22814],{"className":22811,"code":22812,"language":22813,"meta":464,"style":464},"language-cpp shiki shiki-themes github-light github-dark monokai","messages.push_back({ {\"role\", role}, {\"content\", content } });\n","cpp",[30,22815,22816],{"__ignoreMap":464},[151,22817,22818,22821,22824,22827,22829,22832,22834],{"class":469,"line":470},[151,22819,22820],{"class":503},"messages.",[151,22822,22823],{"class":473},"push_back",[151,22825,22826],{"class":503},"({ {",[151,22828,9798],{"class":481},[151,22830,22831],{"class":503},", role}, {",[151,22833,9808],{"class":481},[151,22835,22836],{"class":503},", content } });\n",[11,22838,22839],{},"In Python the code for appending a message to a list of messages would be written differently:",[459,22841,22843],{"className":13136,"code":22842,"language":12886,"meta":464,"style":464},"messages.append({\"role\": role, \"content\": content})\n",[30,22844,22845],{"__ignoreMap":464},[151,22846,22847,22850,22852,22855,22857],{"class":469,"line":470},[151,22848,22849],{"class":503},"messages.append({",[151,22851,9798],{"class":481},[151,22853,22854],{"class":503},": role, ",[151,22856,9808],{"class":481},[151,22858,22859],{"class":503},": content})\n",[56,22861,22309],{"id":22308},[11,22863,22864],{},"While working on different projects using web technologies and frameworks in the Python and JavaScript ecosystems, I developed an appreciation for well-structured development environments that are easy to use. Development environment refers to the tools and processes by which a developer can make a change to source code and see these changes reflected in some version of the application running on a local environment. The local environment (the developer's computer) should be a close proxy for the production environment where the code will ultimately deployed to for end users. For this project the local development environment is our PC itself, which simplifies things. A development environment should support hot-reloading so incremental changes can be run to test functionality, offering a tight feedback loop. I really like the development environment for this project. Here's a screenshot that shows the different parts of the development environment I used for working on Rocket League BotChat:",[11,22866,22867],{},[2718,22868],{"alt":22869,"src":22870},"Screenshot of Rocket League BotChat development environment","/static/rlbc/devenv2.png",[76,22872,22873,22880,22892,22899],{},[79,22874,22875,22876,22879],{},"Rocket League (running with the ",[30,22877,22878],{},"-dev"," flag turned on). The console is helpful for viewing log messages and the plugin settings panel can be used to view and change plugin configuration values. The BakkesMod plugin also needs to be running in order to inject plugin code into the game engine",[79,22881,22882,22883,22885,22886,22885,22889,22891],{},"Visual Studio for working on the plugin code. ",[30,22884,2939],{},"+",[30,22887,22888],{},"Shift",[30,22890,109],{}," rebuilds the code and automatically reloads the plugin in the game",[79,22893,22894,22895,22898],{},"OpenAI-compatible LLM server powered by TensorRT-LLM (using ",[30,22896,22897],{},"Llama-2-13b-chat-hf"," with AWQ INT4 quantization) running in a docker container on Ubuntu in WSL",[79,22900,22901,22902,22904],{},"VSCode for debugging C++ code with Continue.dev extension powered by TensorRT-LLM (using ",[30,22903,22654],{}," with AWQ INT4 quantization) running in a virtual environment on Windows",[736,22906,22908],{"id":22907},"building-the-tensorrt-llm-engines","Building the TensorRT-LLM engines",[11,22910,22911,22912,22914],{},"I was able to build and run the TensorRT LLM engines for my game plugin's inference and the Continue.dev extension's inference both in Python virtual environments on Windows and on Ubuntu in WSL. For building the ",[30,22913,22897],{}," model with INT4 AWQ quantization on Windows 11 I used this command:",[459,22916,22920],{"className":22917,"code":22918,"language":22919,"meta":464,"style":464},"language-powershell shiki shiki-themes github-light github-dark monokai","(.venv) PS C:\\Users\\My PC\\GitHub\\TensorRT-LLM\\examples\\llama> python build.py --model_dir D:\\llama\\Llama-2-13b-chat-hf\\ --quant_ckpt_path D:\\llama\\Llama-2-13b-chat-hf\\llama_tp1_rank0.npz --dtype float16 --use_gpt_attention_plugin float16 --use_gemm_plugin float16 --use_weight_only --weight_only_precision int4_awq --per_group --enable_context_fmha --max_batch_size 1 --max_input_len 3500 --max_output_len 1024 --output_dir D:\\llama\\Llama-2-13b-chat-hf\\single-gpu\\ --vocab_size 32064\n","powershell",[30,22921,22922],{"__ignoreMap":464},[151,22923,22924,22927,22929,22932,22934,22937,22940,22943,22945,22947,22949,22952,22954,22957,22959,22962,22964,22967,22969,22971,22973,22975,22977,22979,22981,22984,22986,22989,22991,22994,22996,22999,23001,23004,23006,23009,23011,23014,23016,23019,23021,23024,23026,23029,23032,23035,23037,23040,23043,23045,23048,23050,23052,23054,23056,23058,23060,23062,23065,23067,23070,23072,23075],{"class":469,"line":470},[151,22925,22926],{"class":503},"(.venv) PS C:\\Users\\My PC\\GitHub\\TensorRT",[151,22928,12445],{"class":1869},[151,22930,22931],{"class":503},"LLM\\examples\\llama",[151,22933,3663],{"class":1869},[151,22935,22936],{"class":503}," python build.py ",[151,22938,22939],{"class":1869},"--",[151,22941,22942],{"class":503},"model_dir D:\\llama\\Llama",[151,22944,12445],{"class":1869},[151,22946,6619],{"class":477},[151,22948,12445],{"class":1869},[151,22950,22951],{"class":503},"13b",[151,22953,12445],{"class":1869},[151,22955,22956],{"class":503},"chat",[151,22958,12445],{"class":1869},[151,22960,22961],{"class":503},"hf\\ ",[151,22963,22939],{"class":1869},[151,22965,22966],{"class":503},"quant_ckpt_path D:\\llama\\Llama",[151,22968,12445],{"class":1869},[151,22970,6619],{"class":477},[151,22972,12445],{"class":1869},[151,22974,22951],{"class":503},[151,22976,12445],{"class":1869},[151,22978,22956],{"class":503},[151,22980,12445],{"class":1869},[151,22982,22983],{"class":503},"hf\\llama_tp1_rank0.npz ",[151,22985,22939],{"class":1869},[151,22987,22988],{"class":503},"dtype float16 ",[151,22990,22939],{"class":1869},[151,22992,22993],{"class":503},"use_gpt_attention_plugin float16 ",[151,22995,22939],{"class":1869},[151,22997,22998],{"class":503},"use_gemm_plugin float16 ",[151,23000,22939],{"class":1869},[151,23002,23003],{"class":503},"use_weight_only ",[151,23005,22939],{"class":1869},[151,23007,23008],{"class":503},"weight_only_precision int4_awq ",[151,23010,22939],{"class":1869},[151,23012,23013],{"class":503},"per_group ",[151,23015,22939],{"class":1869},[151,23017,23018],{"class":503},"enable_context_fmha ",[151,23020,22939],{"class":1869},[151,23022,23023],{"class":503},"max_batch_size ",[151,23025,6760],{"class":477},[151,23027,23028],{"class":1869}," --",[151,23030,23031],{"class":503},"max_input_len ",[151,23033,23034],{"class":477},"3500",[151,23036,23028],{"class":1869},[151,23038,23039],{"class":503},"max_output_len ",[151,23041,23042],{"class":477},"1024",[151,23044,23028],{"class":1869},[151,23046,23047],{"class":503},"output_dir D:\\llama\\Llama",[151,23049,12445],{"class":1869},[151,23051,6619],{"class":477},[151,23053,12445],{"class":1869},[151,23055,22951],{"class":503},[151,23057,12445],{"class":1869},[151,23059,22956],{"class":503},[151,23061,12445],{"class":1869},[151,23063,23064],{"class":503},"hf\\single",[151,23066,12445],{"class":1869},[151,23068,23069],{"class":503},"gpu\\ ",[151,23071,22939],{"class":1869},[151,23073,23074],{"class":503},"vocab_size ",[151,23076,23077],{"class":477},"32064\n",[736,23079,23081],{"id":23080},"running-the-tensorrt-llm-engines","Running the TensorRT-LLM engines",[11,23083,23084],{},"Using Windows PowerShell to start the CodeLlama server for Continue.dev:",[459,23086,23088],{"className":22917,"code":23087,"language":22919,"meta":464,"style":464},"(.venv) PS C:\\Users\\My PC\\GitHub\\trt-llm-as-openai-windows> python .\\app.py --trt_engine_path \"D:\\llama\\CodeLlama-13b-Instruct-hf\\trt_engines\\1-gpu\\\" --trt_engine_name llama_float16_tp1_rank0.engine --tokenizer_dir_path \"D:\\llama\\CodeLlama-13b-Instruct-hf\\\" --port 5000 --host 0.0.0.0\n",[30,23089,23090],{"__ignoreMap":464},[151,23091,23092,23095,23097,23099,23101,23103,23105,23107,23109,23112,23114,23117,23119,23122,23125,23127,23130,23132,23135,23138,23140,23143,23146,23148,23151,23154,23156],{"class":469,"line":470},[151,23093,23094],{"class":503},"(.venv) PS C:\\Users\\My PC\\GitHub\\trt",[151,23096,12445],{"class":1869},[151,23098,615],{"class":503},[151,23100,12445],{"class":1869},[151,23102,16998],{"class":503},[151,23104,12445],{"class":1869},[151,23106,11805],{"class":503},[151,23108,12445],{"class":1869},[151,23110,23111],{"class":503},"windows",[151,23113,3663],{"class":1869},[151,23115,23116],{"class":503}," python .\\app.py ",[151,23118,22939],{"class":1869},[151,23120,23121],{"class":503},"trt_engine_path ",[151,23123,23124],{"class":481},"\"D:\\llama\\CodeLlama-13b-Instruct-hf\\trt_engines\\1-gpu\\\"",[151,23126,23028],{"class":1869},[151,23128,23129],{"class":503},"trt_engine_name llama_float16_tp1_rank0.engine ",[151,23131,22939],{"class":1869},[151,23133,23134],{"class":503},"tokenizer_dir_path ",[151,23136,23137],{"class":481},"\"D:\\llama\\CodeLlama-13b-Instruct-hf\\\"",[151,23139,23028],{"class":1869},[151,23141,23142],{"class":503},"port ",[151,23144,23145],{"class":477},"5000",[151,23147,23028],{"class":1869},[151,23149,23150],{"class":503},"host ",[151,23152,23153],{"class":477},"0.0",[151,23155,643],{"class":503},[151,23157,9549],{"class":477},[11,23159,23160,23161,23164],{},"Tip: Adding ",[30,23162,23163],{},"--host 0.0.0.0"," isn't required here, but it allows me to use the CodeLlama/TensorRT-LLM server with VSCode any computer on my local network using my PC's local IP address in the Continue.dev configuration.",[11,23166,23167],{},"Using docker in WSL to start the Llama-2-13b-chat-hf LLM server:",[459,23169,23172],{"className":23170,"code":23171,"language":997},[995],"root@0a5b5b75f079:/code/git/TensorRT-LLM/examples/server/flask# python3 app.py --trt_engine_path /llama/Llama-2-13b-chat-hf/trt_engines/1-gpu/ --trt_engine_name  llama_float16_t_rank0.engine --tokenizer_dir_path /llama/Llama-2-13b-chat-hf/ --port 5001 --host 0.0.0.0\n",[30,23173,23171],{"__ignoreMap":464},[11,23175,23176,23177,23179],{},"Note: Here I also add ",[30,23178,23163],{},", but this is required in order for the service in the docker container to be reached from WSL by the game running on Windows.",[11,23181,23182],{},"BakkesMod includes a console window that came in handy for debugging errors during development.",[11,23184,23185],{},"At the beginning of this developer contest on January 9, NVIDIA announced Chat with RTX. This is a demo program for Windows that automates a lots of the processes needed to set up a TensorRT-LLM-powered LLM running on your PC. Keep an eye on this project as it may become the best way to install and manage large language models on Windows PCs.",[11,23187,23188],{},[2718,23189],{"alt":23190,"src":23191},"Chat with RTX image","/static/rlbc/chat_with_rtx.jpeg",[56,23193,22351],{"id":13562},[11,23195,23196,23197,13576],{},"Here's a quick look at key parts of the plugin source code (",[20,23198,22488],{"href":22488,"rel":23199},[24],[736,23201,23203],{"id":23202},"hooking-events","Hooking events",[11,23205,23206,23207,23210],{},"Hooking events is the core of how this plugin works. ",[30,23208,23209],{},"StatTickerMessage"," events cover most of the events that are triggered in Rocket League, such as scoring a goal, making a save or demolishing a car.",[459,23212,23214],{"className":22811,"code":23213,"language":22813,"meta":464,"style":464},"    // Hooks different types of events that are handled in onStatTickerMessage\n    // See https://wiki.bakkesplugins.com/functions/stat_events/\n    gameWrapper->HookEventWithCallerPost\u003CServerWrapper>(\"Function TAGame.GFxHUD_TA.HandleStatTickerMessage\",\n        [this](ServerWrapper caller, void* params, std::string eventname) {\n            onStatTickerMessage(params);\n        });\n",[30,23215,23216,23221,23226,23245,23289,23297],{"__ignoreMap":464},[151,23217,23218],{"class":469,"line":470},[151,23219,23220],{"class":1527},"    // Hooks different types of events that are handled in onStatTickerMessage\n",[151,23222,23223],{"class":469,"line":488},[151,23224,23225],{"class":1527},"    // See https://wiki.bakkesplugins.com/functions/stat_events/\n",[151,23227,23228,23231,23233,23236,23238,23240,23243],{"class":469,"line":500},[151,23229,23230],{"class":503},"    gameWrapper->HookEventWithCallerPost",[151,23232,3613],{"class":1869},[151,23234,23235],{"class":503},"ServerWrapper",[151,23237,3663],{"class":1869},[151,23239,12386],{"class":503},[151,23241,23242],{"class":481},"\"Function TAGame.GFxHUD_TA.HandleStatTickerMessage\"",[151,23244,9417],{"class":503},[151,23246,23247,23250,23253,23256,23258,23261,23263,23266,23269,23272,23274,23277,23280,23283,23286],{"class":469,"line":509},[151,23248,23249],{"class":503},"        [",[151,23251,23252],{"class":15289},"this",[151,23254,23255],{"class":503},"](",[151,23257,23235],{"class":15254},[151,23259,23260],{"class":15210}," caller",[151,23262,106],{"class":503},[151,23264,23265],{"class":12347},"void",[151,23267,23268],{"class":1869},"*",[151,23270,23271],{"class":15210}," params",[151,23273,106],{"class":503},[151,23275,23276],{"class":15254},"std",[151,23278,23279],{"class":503},"::",[151,23281,23282],{"class":15254},"string",[151,23284,23285],{"class":15210}," eventname",[151,23287,23288],{"class":503},") {\n",[151,23290,23291,23294],{"class":469,"line":517},[151,23292,23293],{"class":473},"            onStatTickerMessage",[151,23295,23296],{"class":503},"(params);\n",[151,23298,23299],{"class":469,"line":534},[151,23300,23301],{"class":503},"        });\n",[736,23303,23305],{"id":23304},"handling-events-and-building-the-prompt","Handling events and building the prompt",[11,23307,23308,23309,23312],{},"We can unpack values from the event to determine the player to which the event should be attributed. The code then translates the game event and related data into an English sentence. This is appended to a vector of message objects with the ",[30,23310,23311],{},"appendToPrompt"," method.",[459,23314,23316],{"className":22811,"code":23315,"language":22813,"meta":464,"style":464},"    // handle different events like scoring a goal or making a save\n    if (statEvent.GetEventName() == \"Goal\") {\n\n        // was the goal scored by the human player or the bot?\n        if (playerPRI.memory_address == receiver.memory_address) {\n            appendToPrompt(\"Your human opponent just scored a goal against you! \" + score_sentence, \"user\");\n        }\n        else {\n            appendToPrompt(\"You just scored a goal against the human player! \" + score_sentence, \"user\");\n        }\n    }\n",[30,23317,23318,23323,23344,23348,23353,23366,23386,23391,23398,23415,23419],{"__ignoreMap":464},[151,23319,23320],{"class":469,"line":470},[151,23321,23322],{"class":1527},"    // handle different events like scoring a goal or making a save\n",[151,23324,23325,23328,23331,23334,23337,23339,23342],{"class":469,"line":488},[151,23326,23327],{"class":1869},"    if",[151,23329,23330],{"class":503}," (statEvent.",[151,23332,23333],{"class":473},"GetEventName",[151,23335,23336],{"class":503},"() ",[151,23338,17223],{"class":1869},[151,23340,23341],{"class":481}," \"Goal\"",[151,23343,23288],{"class":503},[151,23345,23346],{"class":469,"line":500},[151,23347,1090],{"emptyLinePlaceholder":609},[151,23349,23350],{"class":469,"line":509},[151,23351,23352],{"class":1527},"        // was the goal scored by the human player or the bot?\n",[151,23354,23355,23358,23361,23363],{"class":469,"line":517},[151,23356,23357],{"class":1869},"        if",[151,23359,23360],{"class":503}," (playerPRI.memory_address ",[151,23362,17223],{"class":1869},[151,23364,23365],{"class":503}," receiver.memory_address) {\n",[151,23367,23368,23371,23373,23376,23379,23382,23384],{"class":469,"line":534},[151,23369,23370],{"class":473},"            appendToPrompt",[151,23372,12386],{"class":503},[151,23374,23375],{"class":481},"\"Your human opponent just scored a goal against you! \"",[151,23377,23378],{"class":1869}," +",[151,23380,23381],{"class":503}," score_sentence, ",[151,23383,16772],{"class":481},[151,23385,20129],{"class":503},[151,23387,23388],{"class":469,"line":1413},[151,23389,23390],{"class":503},"        }\n",[151,23392,23393,23396],{"class":469,"line":1418},[151,23394,23395],{"class":1869},"        else",[151,23397,19833],{"class":503},[151,23399,23400,23402,23404,23407,23409,23411,23413],{"class":469,"line":2462},[151,23401,23370],{"class":473},[151,23403,12386],{"class":503},[151,23405,23406],{"class":481},"\"You just scored a goal against the human player! \"",[151,23408,23378],{"class":1869},[151,23410,23381],{"class":503},[151,23412,16772],{"class":481},[151,23414,20129],{"class":503},[151,23416,23417],{"class":469,"line":2471},[151,23418,23390],{"class":503},[151,23420,23421],{"class":469,"line":2480},[151,23422,9461],{"class":503},[736,23424,23426],{"id":23425},"making-requests-and-handling-responses","Making requests and handling responses",[11,23428,23429],{},"The last main part of the code is making a request to the LLM server with the prompt that we have formed above based on game messages. This code should look familiar to anyone who has worked with OpenAI's API.",[459,23431,23433],{"className":22811,"code":23432,"language":22813,"meta":464,"style":464},"std::string message = response_json[\"choices\"][0][\"message\"][\"content\"];\n",[30,23434,23435],{"__ignoreMap":464},[151,23436,23437,23439,23442,23444,23447,23449,23451,23453,23455,23457,23459,23461],{"class":469,"line":470},[151,23438,23276],{"class":15254},[151,23440,23441],{"class":503},"::string message ",[151,23443,1876],{"class":1869},[151,23445,23446],{"class":503}," response_json[",[151,23448,9778],{"class":481},[151,23450,6704],{"class":503},[151,23452,9181],{"class":477},[151,23454,6704],{"class":503},[151,23456,6247],{"class":481},[151,23458,6704],{"class":503},[151,23460,9808],{"class":481},[151,23462,23463],{"class":503},"];\n",[11,23465,19225,23466,23469],{},[30,23467,23468],{},"LogToChatbox"," method is used to send a message to the in-game chat box with the name of the bot that is sending the message. Since messages could possibly be longer than the limit of 120 characters, I send messages to the chatbox in chunks of 120 characters at a time.",[459,23471,23473],{"className":22811,"code":23472,"language":22813,"meta":464,"style":464},"gameWrapper->LogToChatbox(messages[i], this->bot_name);\n",[30,23474,23475],{"__ignoreMap":464},[151,23476,23477,23480,23482,23485,23487],{"class":469,"line":470},[151,23478,23479],{"class":503},"gameWrapper->",[151,23481,23468],{"class":473},[151,23483,23484],{"class":503},"(messages[i], ",[151,23486,23252],{"class":15289},[151,23488,23489],{"class":503},"->bot_name);\n",[11,23491,23492,23493,23496],{},"That's it! The code isn't that complicated. I had to sanitize the message so that it would not include emoji or the stop character that the LLM server would include in messages (",[30,23494,23495],{},"\u003C/s>","). Oddly, I had a hard time getting the LLM to not use emoji even when I instructed it to not use emoji in the system prompt.",[56,23498,23500],{"id":23499},"rocket-league-botchat-ui","Rocket League BotChat UI",[11,23502,23503],{},"Most BakkesMod plugins for RocketLeague UIs that allow for controlling settings. Here's what the UI for Rocket League BotChat looks like:",[11,23505,23506],{},[2718,23507],{"alt":23508,"src":23509},"Rocket League BotChat Plugin UI","/static/rlbc/rlbcui.png",[736,23511,23513],{"id":23512},"system-prompt","System prompt",[11,23515,23516],{},"The system prompt instructs the bot on how it shoud reply. This is an important part of the prompt engineering for this project, and I used Postman to experiment with lots of different types of instructions. Here's the default prompt that I used:",[459,23518,23520],{"className":22811,"code":23519,"language":22813,"meta":464,"style":464},"    std::string ai_player = \"You are an elite AI player in the car soccer game Rocket League. \";\n    std::string one_v_one = \"You are playing a 1v1 match against a human player. \";\n    std::string instructions = \"You will send short chat messages to your human opponent in response to what happens in the game. \";\n    std::string details = \"Respond to the human player with brief messages no more than 12 words long.\";\n    // initial system prompt\n    std::string initial_system_prompt = ai_player + one_v_one + instructions + details;\n",[30,23521,23522,23537,23551,23565,23579,23584],{"__ignoreMap":464},[151,23523,23524,23527,23530,23532,23535],{"class":469,"line":470},[151,23525,23526],{"class":15254},"    std",[151,23528,23529],{"class":503},"::string ai_player ",[151,23531,1876],{"class":1869},[151,23533,23534],{"class":481}," \"You are an elite AI player in the car soccer game Rocket League. \"",[151,23536,20086],{"class":503},[151,23538,23539,23541,23544,23546,23549],{"class":469,"line":488},[151,23540,23526],{"class":15254},[151,23542,23543],{"class":503},"::string one_v_one ",[151,23545,1876],{"class":1869},[151,23547,23548],{"class":481}," \"You are playing a 1v1 match against a human player. \"",[151,23550,20086],{"class":503},[151,23552,23553,23555,23558,23560,23563],{"class":469,"line":500},[151,23554,23526],{"class":15254},[151,23556,23557],{"class":503},"::string instructions ",[151,23559,1876],{"class":1869},[151,23561,23562],{"class":481}," \"You will send short chat messages to your human opponent in response to what happens in the game. \"",[151,23564,20086],{"class":503},[151,23566,23567,23569,23572,23574,23577],{"class":469,"line":509},[151,23568,23526],{"class":15254},[151,23570,23571],{"class":503},"::string details ",[151,23573,1876],{"class":1869},[151,23575,23576],{"class":481}," \"Respond to the human player with brief messages no more than 12 words long.\"",[151,23578,20086],{"class":503},[151,23580,23581],{"class":469,"line":517},[151,23582,23583],{"class":1527},"    // initial system prompt\n",[151,23585,23586,23588,23591,23593,23596,23598,23601,23603,23606,23608],{"class":469,"line":534},[151,23587,23526],{"class":15254},[151,23589,23590],{"class":503},"::string initial_system_prompt ",[151,23592,1876],{"class":1869},[151,23594,23595],{"class":503}," ai_player ",[151,23597,22885],{"class":1869},[151,23599,23600],{"class":503}," one_v_one ",[151,23602,22885],{"class":1869},[151,23604,23605],{"class":503}," instructions ",[151,23607,22885],{"class":1869},[151,23609,23610],{"class":503}," details;\n",[11,23612,23613,23614,23617,23618,23621],{},"The last part about ",[30,23615,23616],{},"no more than 12 words long"," was the most effective way of controlling the length responses from the LLM. I tried changing the ",[30,23619,23620],{},"max_output_len"," when building the TensorRT engine, but this degraded the quality of the responses. The system prompt can be changed by the user. Changing the system prompt was a lot of fun to expirment with!",[736,23623,23625],{"id":23624},"temperature-and-seed","Temperature and Seed",[11,23627,23628],{},"These values are included in the body of the request to the LLM, but I didn't have much luck with these. Early on I had issues with getting sufficient variation in the responses from the LLM, so I tried using random values for seed and temperature, but this didn't really work.",[736,23630,23631],{"id":8360},"Messages",[11,23633,23634],{},"This section of the UI displays the messages that are used in requests to the LLM. In order keep the prompt within the context window limit, I only used the most recent six messages sent from the \"user\" (which are messages about game events) and the \"assistant\" (which are LLM responses from the bot). Whenever the user changes the system prompt, the messages vector is reset to only include the new system prompt.",[56,23636,22368],{"id":22367},[23638,23639],"rocket-league-bot-chat-video",{},[11,23641,23642],{},"I used Blender's sequence editor to create a demo video for my contest submission. I don't edit a lot of videos, but it is a fun process and I learned a lot about Blender and non-linear video editing in the process. Here's how I approached creating the demo video for my project.",[11,23644,23645],{},[2718,23646],{"alt":23647,"src":23648},"Blender video sequence editor UI used to create my project video","/static/rlbc/blender.png",[76,23650,23651,23654,23662,23675,23678],{},[79,23652,23653],{},"Structure the video in three main parts: introduction to my project and the contest, description of how it works, demo of my project in action",[79,23655,23656,23657],{},"Find an upbeat song from playlists included in Rocket League with no vocals to use as background music. I used ",[20,23658,23661],{"href":23659,"rel":23660},"https://open.spotify.com/track/68ahXxPJrxcEvQFjRmC2ja?si=2147d6d652064d51",[24],"\"Dads in Space\" by Steven Walking",[79,23663,23664,23665,23668,23669,23674],{},"Get stock Rocket League footage from YouTube with ",[30,23666,23667],{},"youtube-dl"," (this is an amazing tool!). I mostly used footage from the ",[20,23670,23673],{"href":23671,"rel":23672},"https://www.youtube.com/watch?v=e1tqWldCYOI&pp=ygUQcmxjcyB3aW50ZXIgMjAyMw%3D%3D",[24],"RLCS 2023 Winter Major Trailer",". This video was uploaded at 24 fps, and my Blender Video project frame rate was set to 29.97, so I used ffmpeg to convert this video from 24 fps to 29.97 fps.",[79,23676,23677],{},"Record myself playing Rocket League with my plugin enabled using NVIDIA Share. Miraculously, I was able to score against the Nexto bot!",[79,23679,23680],{},"Use ComfyUI to animate some of the images used in the contest description and use these in my video",[11,23682,23683],{},[2718,23684],{"alt":23685,"src":23686},"ComfyUI workflow for animating images using img2vid model","/static/rlbc/comfyui.png",[76,23688,23689],{},[79,23690,23691],{},"Use ElevenLabs to narrate a simple voice over script that describes the video content. This turned out a lot better than I expected. I paid $1 for the ElevenLabs creator plan and got lots of tokens to experiment with different settings for voice generation using a clone of my voice.",[11,23693,23694],{},[2718,23695],{"alt":23696,"src":23697},"Eleven Labs Voice Generation Web UI","/static/rlbc/elevenlabs.png",[11,23699,23700],{},[20,23701,23703],{"href":23702},"#","Embed twitter video here",[56,23705,22381],{"id":22380},[11,23707,23708,23709,23711],{},"This plugin is a proof of concept and it has some shortcomings. One issue is that some events that my plugin listens to can happen in rapid succession. This results in \"user\" and \"assistant\" prompts getting out of order which breaks assertions on the ",[30,23710,21404],{}," repo. It would make more sense to have the bot send messages not immediately after the events are triggered, but on a different type of schedule that allows for multiple events to happen before sending the prompt to the LLM.",[11,23713,23714],{},"There are lots of events that are triggered that would be interesting things for the bot to react to, but I decided not to prompt on every event since the above situation would be triggered frequently. For example, suppose I listen for events like taking a shot on goal and scoring a goal. If the goal is scored immediately after the shot is taken, then the second prompt is sent before the response for the first prompt comes back. For this reason I decided to simply not listen to events like \"shot on goal\" to avoid prompt messages getting out of order. This could also be addressed with more code logic.",[11,23716,23717],{},"Prompt engineering is something that can always be improved. It is hard to measure and testing it is subjective. I am pleased with the results I was able to capture for the demo video, but the quality of the LLM responses can vary depending on what happens during gameplay. One idea I had to address this would be to provide multiple English translations for any given event, and then select one at random. This might help improve the variety of responses, for example.",[11,23719,23720,23721,23724],{},"I faced some limitations that are built in to the game itself. For example, it is not possible for a player to send messages to the in-game chat in offline matches, which makes sense! I built a backdoor for doing this through the BakkesMod developer console, so you can send messages to the bot by typing something like ",[30,23722,23723],{},"SendMessage Good shot, bot!",", for example.",[56,23726,22420],{"id":16069},[11,23728,23729],{},"Participating in this contest was a great opportunity to learn more about LLMs and how to use them to extend programs in a Windows environment. It was also a lot of fun to build something by putting together new tools like TensorRT-LLM. Seeing the bot send me chat messages was very satisfying when I first got it to work! Overall it is a pretty simple implementation, but this idea could be extended to produce useful application. I could imagine a \"Rocket League Coach\" plugin that expands on this idea to give helpful feedback based on higher-level data, statistical trends, training goals, etc.",[11,23731,23732],{},"I think the gaming industry's adoption of LLMs for new games will be BIG, and it will present a huge opportunity for LLM optimization and acceleration software like TensorRT-LLM that I was able to use in my Rocket League BotChat. This is not to discredit the work of writers which play an important role in game development. I'm excited to see what other developers have built for this contest, especially submissions that are building mods for games using TensorRT-LLM.",[11,23734,23735],{},"Thanks NVIDIA and the TensorRT and TensorRT-LLM teams for organizing this contest! Keep on building!!",[589,23737,23738],{},"html pre.shiki code .sMOD_, html code.shiki .sMOD_{--shiki-default:#24292E;--shiki-dark:#E1E4E8;--shiki-sepia:#F8F8F2}html pre.shiki code .s-m8C, html code.shiki .s-m8C{--shiki-default:#005CC5;--shiki-default-font-style:inherit;--shiki-dark:#79B8FF;--shiki-dark-font-style:inherit;--shiki-sepia:#66D9EF;--shiki-sepia-font-style:italic}html pre.shiki code .sCZoN, html code.shiki .sCZoN{--shiki-default:#032F62;--shiki-dark:#9ECBFF;--shiki-sepia:#CFCFC2}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html .sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html.sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html pre.shiki code .sstjo, html code.shiki .sstjo{--shiki-default:#032F62;--shiki-dark:#9ECBFF;--shiki-sepia:#E6DB74}html pre.shiki code .srTi1, html code.shiki .srTi1{--shiki-default:#6F42C1;--shiki-dark:#B392F0;--shiki-sepia:#A6E22E}html pre.shiki code .sC2Qs, html code.shiki .sC2Qs{--shiki-default:#D73A49;--shiki-dark:#F97583;--shiki-sepia:#F92672}html pre.shiki code .s7F3e, html code.shiki .s7F3e{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#AE81FF}html pre.shiki code .s8-w5, html code.shiki .s8-w5{--shiki-default:#6A737D;--shiki-dark:#6A737D;--shiki-sepia:#88846F}html pre.shiki code .sP7S_, html code.shiki .sP7S_{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#FD971F}html pre.shiki code .sz2Vg, html code.shiki .sz2Vg{--shiki-default:#6F42C1;--shiki-default-text-decoration:inherit;--shiki-dark:#B392F0;--shiki-dark-text-decoration:inherit;--shiki-sepia:#A6E22E;--shiki-sepia-text-decoration:underline}html pre.shiki code .sTHNf, html code.shiki .sTHNf{--shiki-default:#E36209;--shiki-default-font-style:inherit;--shiki-dark:#FFAB70;--shiki-dark-font-style:inherit;--shiki-sepia:#FD971F;--shiki-sepia-font-style:italic}html pre.shiki code .sq6CD, html code.shiki .sq6CD{--shiki-default:#D73A49;--shiki-default-font-style:inherit;--shiki-dark:#F97583;--shiki-dark-font-style:inherit;--shiki-sepia:#66D9EF;--shiki-sepia-font-style:italic}",{"title":464,"searchDepth":488,"depth":488,"links":23740},[23741,23742,23743,23744,23745,23749,23754,23759,23760,23761],{"id":16115,"depth":488,"text":16116},{"id":22493,"depth":488,"text":22494},{"id":21625,"depth":488,"text":22523},{"id":21632,"depth":488,"text":21633},{"id":22308,"depth":488,"text":22309,"children":23746},[23747,23748],{"id":22907,"depth":500,"text":22908},{"id":23080,"depth":500,"text":23081},{"id":13562,"depth":488,"text":22351,"children":23750},[23751,23752,23753],{"id":23202,"depth":500,"text":23203},{"id":23304,"depth":500,"text":23305},{"id":23425,"depth":500,"text":23426},{"id":23499,"depth":488,"text":23500,"children":23755},[23756,23757,23758],{"id":23512,"depth":500,"text":23513},{"id":23624,"depth":500,"text":23625},{"id":8360,"depth":500,"text":23631},{"id":22367,"depth":488,"text":22368},{"id":22380,"depth":488,"text":22381},{"id":16069,"depth":488,"text":22420},"2024-02-17","This article discusses my entry for NVIDIA's Generative AI on RTX PCs Developer Contest: Rocket League BotChat",[23765,23767,23770],{"link":23766,"site":11126},"https://twitter.com/briancaffey/status/1760529251072118901",{"link":23768,"site":23769},"https://www.reddit.com/r/RocketLeague/comments/1au0po3/rocket_league_botchat_an_llmpowered_bakkesmod/","reddit",{"link":23771,"site":10715},"https://dev.to/briancaffey/rocket-league-botchat-powered-by-tensorrt-llm-my-submission-for-nvidias-generative-ai-on-rtx-pcs-developer-contest-2oao","/static/rlbc/cover.png",{},"/2024/02/17/rocket-league-botchat-nvidia-generative-ai-on-rtx-pcs-developer-contest",{"title":22466,"description":23763},"2024/02/17/rocket-league-botchat-nvidia-generative-ai-on-rtx-pcs-developer-contest",[11133,14055,21573,19403,614,615,21575,23778,23779,23111],"rocket-league","gaming","D4JaAhpSMnYictCtnqNM0fD3J1nndB9bItKf9o1e1yQ",{"id":23782,"title":23783,"body":23784,"comments":609,"date":25720,"description":25721,"draft":602,"extension":605,"external":25722,"image":25730,"meta":25731,"navigation":609,"path":25732,"seo":25733,"stem":25734,"tags":25735,"__hash__":25739},"blog/2023/08/27/python-vue-chinese-llama-2-and-the-three-body-problem.md","Python, Vue, Chinese-LLaMA-2 and The Three-Body Problem",{"type":8,"value":23785,"toc":25703},[23786,23788,23793,23796,23822,23825,23828,23832,23835,23841,23850,23853,23859,23862,23868,23871,23877,23880,23888,23891,23895,23898,23913,23916,23936,23940,23949,23960,23964,23973,23976,23993,23999,24003,24006,24009,24012,24015,24021,24024,24028,24031,24045,24049,24052,24182,24188,24196,24202,24207,24314,24321,24326,24329,24352,24356,24359,24364,24371,24376,24382,24385,24389,24392,24396,24399,24991,24994,25055,25058,25279,25291,25294,25391,25397,25404,25495,25498,25530,25536,25539,25573,25579,25582,25586,25589,25595,25599,25602,25606,25610,25613,25619,25622,25628,25631,25634,25637,25641,25649,25652,25685,25694,25700],[14063,23787,16116],{"id":16115},[11,23789,23790,23792],{},[15,23791,11825],{}," This article references models and tools from 2023 (Chinese-LLaMA-2, Qwen-7B-Chat, Baichuan-13B, Stable Diffusion, InvokeAI, CuPy/CUDA). These projects evolve quickly; check official documentation for current model names, versions, and best practices.",[11,23794,23795],{},"This article brings together several of my interests, both old and new:",[76,23797,23798,23801,23804,23807,23810,23813,23816,23819],{},[79,23799,23800],{},"The Sci-Fi book series 'Three-Body Problem' by Liu Cixun",[79,23802,23803],{},"Chinese language",[79,23805,23806],{},"NLP techniques",[79,23808,23809],{},"Large Language Models (LLMs)",[79,23811,23812],{},"Stable Diffusion",[79,23814,23815],{},"Data visualization and 3D graphics",[79,23817,23818],{},"Mathematics",[79,23820,23821],{},"NVIDIA / CUDA",[11,23823,23824],{},"This is a linguistic, artistic and computational experiment with the two big AI algorithms of 2023: large language models (LLMs) and Stable Diffusion. I used the leading open-source LLMs from China’s tech sector to translate and summarize the text of Chinese author Liu Cixin’s award-winning science fiction novel: The Three-Body Problem. The book's storyline is based on a simple yet elusive problem from classical physics: predicting the movement of three gravitationally-attracted objects in space. I generated code for simulations and visualizations of this physics problem to present my own solutions to the three-body problem based on parallel computation. I also used Stable Diffusion to portray the imaginitive solutions to the three-body physics problem from one of the book’s main settings: an immersive virtual-reality game that spans centries of world history.",[11,23826,23827],{},"I also share some of my experiences in China as an exchange student and research manager in the renewable energy technology sector. I wrote this article in English and translated it into Chinese using the same large language models I used to translate the Chinese text of the sci-fi novel into English. Warning: this article contains spoilers for the first book in the trilogy!",[56,23829,23831],{"id":23830},"back-story","Back story",[11,23833,23834],{},"A few months ago my company announced that another round of layoffs was to come the following week. I'm on an engineering team that had already been impacted by a few rounds of layoffs in the past year, and I was expecting to be let go. On an impulse I bought a book at the top of my reading list from Amazon: \"Three-Body Problem\". It is an award-winning Sci-Fi trilogy written by Liu Cixin, a Chinese computer engineer who started writing the book as a series of essays that were published in China's \"World of Sci-Fi\" magazine.",[11,23836,23837],{},[2718,23838],{"alt":23839,"src":23840},"Images of Three Body Problem Book Series","/static/three-body-problem/books.png",[11,23842,23843,23844,23849],{},"I started learning Chinese in college, adding a major in Chinese Language to the mathematics major I decided on in my freshman year after taking vector calculus and linear algebra. In my sophmore year I did a semester abroad at Fudan University's ",[20,23845,23848],{"href":23846,"rel":23847},"https://ices.fudan.edu.cn/6628/list.htm",[24],"International Cultural Exchange School",". In 2007, living and studying Chinese in Shanghai as a 19 year old American was a really fun time. I was placed in an advanced-level course with a diverse group of students where English was not the lowest common linguistic denominator. We had a demanding curriculum that emphasized reading, listening and speaking Chinese, but most of the language learning came through extracurricular activities: exploring Shanghai's food scene, bartering with vendors at the fabric markets, late night clubbing, walking around the Bund and the French Concession and chatting with my taxi cab drivers. It is hard to imagine how I did this without an iPhone, but I was able to get pretty far with an old Nokia 3310.",[11,23851,23852],{},"At the end of one night of particularly heavy drinking, some of my classmates and I dropped in on an wangba (internet cafe) before heading back to the international dorm. Chinese internet cafes in 2007 were an expansive underground dens of computers, monitors, MMORPGs, FPSs, cigarets, and on-demand instant noodles delivered directly to your seat through an app on the desktop. That night our game of choice was Counter-Strike. In one of the lowest points of my gaming career, my classmates and I were crushed by our Chinese counterterrorist opponent.",[11,23854,23855],{},[2718,23856],{"alt":23857,"src":23858},"Chinese internet cafe","/static/three-body-problem/wangba.webp",[11,23860,23861],{},"My favorite memory of that semester at Fudan University was travelling on an epic over-night sleeper train from Shanghai to Guangxi province with a school-sponsored class trip to see Guilin. Multiple games of sam-yuk-gu (3-6-9) ran in parallel across the matrix of 3-by-2 sleeper car bunk beds lining the train car like workloads distributed across multiple GPU cores. The rules of 3-6-9 are simple: a group of people go around in a circle counting up from 1. If your number contains a 3, 6 or 9, you clap once for each occurance of the number instead of saying your number. The first person to break the rules takes a drink. Then repeat indefinitely. The next morning we all boarded a boat cruise in a daze to see the Lijiang river's stunning limestone peaks featured on the 20 yuan note:",[11,23863,23864],{},[2718,23865],{"alt":23866,"src":23867},"20 yuan note with Guilin rock formations","/static/three-body-problem/twenty_small.gif",[11,23869,23870],{},"My second job after college took me back to China where I specialized in the technologies, policies and applications of large scale battery projects as a research manager for China's energy storage industry association. The job exposed me to the power industry and cutting-edge battery projects, and also sharpened my technical Chinese as I was frequently reading, translating in a bi-linguagl environment. It was fun  I didn't realize it at the time, but that job was great preperation for reading Chinese Sci-Fi novels.",[11,23872,23873],{},[2718,23874],{"alt":23875,"src":23876},"State Grid HQ in Xi Cheng","/static/three-body-problem/invokeai/castles.png",[11,23878,23879],{},"My first introduction to the 'Three-Body Problem' book came from one of my best friends from college. He lived at the inner-most leaf-node of one of Beijing's most labyrinthian hutongs next to a family that trained racing pigeons. My friend and I bonded over our study of Chinese language, classical guitar and our experiences in Beijing. I strongly considered his recommendation to check out 三体 (Three Body), the Chinese Sci-Fi novel about alien life in a solar system with three stars as he described it, but I never had the chance to read the book.",[23881,23882],"iframe",{"width":23883,"height":23884,"src":23885,"title":23886,"frameBorder":9181,"allow":23887,"allowFullScreen":609},"100%",315,"https://www.youtube.com/embed/5lj99Uz1d50?si=TwrypbY4vTfeWGRf","YouTube video player","accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share",[11,23889,23890],{},"Almost 10 years later I came across a preview for the Netflix production of \"3 Body Problem\" scheduled to come out in early 2024. With the possibility of loosing my job weighing heavily on me, I picked up the books on Amazon hoping to have something to if I was going to be layed off. Over the weekend I was able to read a few of the chapters on my Kindle. I had not completely forgotten how to speak Chinese, and I could easily look up words and translate entire paragraphs with Google Translate.",[56,23892,23894],{"id":23893},"chinese-in-numbers","Chinese in numbers",[11,23896,23897],{},"Here's a quick primer on the Chinese language from a mathematical perspective. This will be helpful before jumping into using NLP and LLMs with Chinese text later in this article.",[11,23899,23900,23901,23906,23907,23912],{},"First, how many Chinese characters are there? This question isn't specific enough to have a single answer. A common rule of thumb that I have heard before says that there are over 50,000 characters in total with roughly 10,000 characters in use and about 3,000 characters frequently used in Chinese media and newspapers (",[20,23902,23905],{"href":23903,"rel":23904},"https://en.wikipedia.org/wiki/Chinese_language#Vocabulary",[24],"source","). ",[20,23908,23911],{"href":23909,"rel":23910},"https://stackoverflow.com/a/1366113/6084948",[24],"This answer"," from StackOverflow's legendary #1 ranked user VonC gives a good answer based on the number of Unicode characters in the CJK Unified Ideographs block: 20,992.",[11,23914,23915],{},"Here are some numbers and statistics to be better understand the text of the Three-Body Problem Chinese text:",[76,23917,23918,23921,23924,23927,23930,23933],{},[79,23919,23920],{},"188,380 total charactes in the book",[79,23922,23923],{},"2,859 unique characters in the book",[79,23925,23926],{},"36 chapters in the book",[79,23928,23929],{},"average of 69.78 paragraphs per chapter",[79,23931,23932],{},"total of 2,512 paragraphs in the book",[79,23934,23935],{},"average of 74.99 characters per paragraph",[736,23937,23939],{"id":23938},"character-frequency","Character Frequency",[11,23941,23942,23943,23948],{},"Let's look at how frequently each character in the book is used. We can also combine this with some data on the overall frequency of Chinese characters. The best measurement I found for overall character frequency is from ",[20,23944,23947],{"href":23945,"rel":23946},"https://lingua.mtsu.edu/chinese-computing/statistics/char/list.php?Which=MO",[24],"Middle Tennessee State University",". Here's a visualization that shows of all of unique characters in the book. The height of a column represents how frequently a character occurs in the book, and the color represents the relatively frequency of the character in Chinese language overall.",[23950,23951,23954],"div",{"className":23952},[23953],"wrap",[23881,23955],{"className":23956,"src":23958,"width":23883,"height":23959},[23957],"p-4","https://briancaffey.github.io/three-body-problem/tjs/load.html",550,[56,23961,23963],{"id":23962},"meta-llms-and-grass-mud-horse","Meta, LLMs, and Grass Mud Horse",[11,23965,23966,23967,23972],{},"In recent months I have been following the development of big open source AI projects. Two projects in particular are InvokeAI, an image generation tool based on Stable Diffusion, and ",[20,23968,23971],{"href":23969,"rel":23970},"https://ai.meta.com/llama/",[24],"LLaMA 2",", the latest generation of Meta's open source LLM. The name LLaMA stands for 'Large Language Model Meta AI', which happens to be the same spelling as the word for the domesticated South American camelid: llama. Before going deeper into LLMs we need a quick Chinese lesson.",[11,23974,23975],{},"草泥马 is a non-technical word that referes to animals like Llama or Alpaca. It can be directly translated as \"Grass Mud Horse\" and it is phonetically similar to the most common Chinese profanity: 操你妈, which literally means \"f*** your mother\". The characters in these two words are nearly synonymous: the sounds of both words are \"cao ni ma\", but the tones are different, which in Chinese changes the meaning completely. The llama is basically a legendary Chinese internet meme subversive in the face of government censorship. 🦙 was approved as part of Unicode 11.0 in 2018. The extended version of this profanity is 草泥马戈壁 (Cǎonímǎ Gēbì: Grass Mud Horse Gobi), refering to the geographical origin of this mythical creature: the Gobi Dessert. This term is more explicit as it synonymous with \"f*** your mother's c***\". Coincidentally, the Gobi Desert is a region of Inner Mongolia which borders the mountainous region of Greater Khingan Range (大兴安岭), the location of Red Coast and Radar Peak in Three-Body Problem where Ye Wenjie makes first contact with the Trisolarians.",[11,23977,23978,23979,23986,23987,23992],{},"I requested access to Meta's LLaMa 2 models as soon as they came out and I was able to get it to run on my NVIDIA RTX 4090 GPU. I also joined a subreddit called ",[20,23980,23983],{"href":23981,"rel":23982},"https://www.reddit.com/r/LocalLLaMA/",[24],[30,23984,23985],{},"r/LocalLLaMa"," with over seventy thousand members discussing how to run large language models on consumer hardware. Another annoucement that caught my attention in July was the release of ",[20,23988,23991],{"href":23989,"rel":23990},"https://github.com/ymcui/Chinese-LLaMA-Alpaca-2",[24],"Chinese LLaMa 2",", an open-source large language model trained on Chinese and English which does very well against Chinese Language LLM Benchmarks such as the CMMCU: Chinese Massive Multitask Language Understanding.",[11,23994,23995],{},[2718,23996],{"alt":23997,"src":23998},"image of CMMLU","/static/three-body-problem/cmmlu.jpeg",[56,24000,24002],{"id":24001},"translation-in-and-of-the-three-body-problem","Translation in and of The Three-Body Problem",[11,24004,24005],{},"There are two important plot developments related to language translation in the Three-Body Problem novel, both of which involve book’s main female protagonist Ye Wenjie. First, copying the translation of Rachel Carson's 'Silent Spring' leads to her being relegated to the Red Coast project. At the Red Coast Ye Wenjie communicates with extraterrestrial life through a universal translation technology developed by the top-secret project.",[11,24007,24008],{},"Ken Liu’s translation of the Three-Body Problem book from Chinese to English places the events during the Cultural Revolution at the beginning of the book rather than in the middle of the book. According to Liu, this was done in order to avoid attention of government censors, and his original intention was to tell the story in this way, starting with the events of the late 1960's in China.",[11,24010,24011],{},"I tried translating the Chinese text of the Three-Body Problem book using LLMs. I started with the Chinese-LLaMA-2 model and then tried Qwen-7B-Chat, Baichuan-13B-Chat when these models came out. I found that the Qwen-7B-Chat model worked best for my translation tasks. Qwen is short for Qian Wen (千问, or \"one thousand questions\") and is developed by Alibaba Cloud.",[11,24013,24014],{},"How do you get an LLM to translate text? Ultimately the quality of the translation returned by the LLM depends on the prompt and other parameters used for inference. I experimented with both chat and completion approaches and tried lots of different kinds of prompts. The models I worked with have a 4K context window (the number of tokens the model can take as input when generating responses), so for translation tasks I had the LLM work on one paragraph at a time. Here's the prompt I used with the Qwen-7B-Chat model:",[459,24016,24019],{"className":24017,"code":24018,"language":997},[995],"\"你是一名翻译。请将每条消息从中文翻译成英文。\"\n(You are a translator. Please translate each message from Chinese to English.)\n",[30,24020,24018],{"__ignoreMap":464},[11,24022,24023],{},"I did some basic prompt engineering to get the LLM to translate the books in the Three-Body problem paragraph by paragraph. My computer was able to translate the first book overnight in under 500 minutes. Here are the results of my translation of Three-Body Problem with Qwen-7B-Chat model:",[23881,24025],{"className":24026,"src":24027,"width":23883,"height":23959},[23957],"https://briancaffey.github.io/three-body-problem/reader/?book=three_body&chapterNumber=1",[11,24029,24030],{},"It was interesting to see the failure modes of translation tasks for the different models. Most of the time the LLM was able to provide accurate translations. Some of the failure modes I observed were:",[76,24032,24033,24036,24039,24042],{},[79,24034,24035],{},"a few Chinese characters would show up in the English translations",[79,24037,24038],{},"a complete Chinese sentence would show up in an otherwise complete translation of a paragraph",[79,24040,24041],{},"The LLM refused to translate certain paragraphs that included violent imagery, such as the violent scenes from the Cultural Revolution chapters",[79,24043,24044],{},"If the sentence it was asked to translate was a question, the LLM would respond in Chinese to the question rather than providing a translation of the question itself",[736,24046,24048],{"id":24047},"tokenization","Tokenization",[11,24050,24051],{},"When you feed a prompt to an LLM, it first puts the prompt through a process called tokenization. Tokenization takes a string of text and breaks it down into tokens (defined by the Large Language Model you are using). The process of tokenization is similar to the tokenization done by spaCy mentioned earlier. These tokens produced by LLM tokenization are numbers. Here's an example of tokenization in action using the Chinese-Llama-2 model:",[459,24053,24055],{"className":13136,"code":24054,"language":12886,"meta":464,"style":464},"import json\nimport os\nfrom llama_cpp import Llama, LlamaTokenizer\n\nllm = Llama(\n    model_path=\"/path/to/models/ggml-model-q4_0.bin\",\n    n_ctx=4096,\n    n_gpu_layers=30\n)\n\ntokenizer = LlamaTokenizer(llama=llm)\n\nTEXT=\"在那个已被忘却的日子里，它的世界颠覆了。泥土飞走，出现了一条又深又宽的峡谷，然后泥土又轰隆隆地飞回来，峡谷消失了，在原来峡谷的尽头出现了一座黑色的孤峰。其实，在这片广阔的疆域上，这种事常常发生，泥土飞走又飞回，峡谷出现又消失，然后是孤峰降临，好像是给每次灾变打上一个醒目的标记。褐蚁和几百个同族带着幸存的蚁后向太阳落下的方向走了一段路，建立了新的帝国。\"\ntokens = tokenizer.encode(TEXT)\n",[30,24056,24057,24064,24071,24083,24087,24096,24108,24119,24129,24133,24137,24154,24158,24168],{"__ignoreMap":464},[151,24058,24059,24061],{"class":469,"line":470},[151,24060,16859],{"class":1869},[151,24062,24063],{"class":503}," json\n",[151,24065,24066,24068],{"class":469,"line":488},[151,24067,16859],{"class":1869},[151,24069,24070],{"class":503}," os\n",[151,24072,24073,24075,24078,24080],{"class":469,"line":500},[151,24074,16853],{"class":1869},[151,24076,24077],{"class":503}," llama_cpp ",[151,24079,16859],{"class":1869},[151,24081,24082],{"class":503}," Llama, LlamaTokenizer\n",[151,24084,24085],{"class":469,"line":509},[151,24086,1090],{"emptyLinePlaceholder":609},[151,24088,24089,24091,24093],{"class":469,"line":517},[151,24090,16325],{"class":503},[151,24092,1876],{"class":1869},[151,24094,24095],{"class":503}," Llama(\n",[151,24097,24098,24101,24103,24106],{"class":469,"line":534},[151,24099,24100],{"class":15210},"    model_path",[151,24102,1876],{"class":1869},[151,24104,24105],{"class":481},"\"/path/to/models/ggml-model-q4_0.bin\"",[151,24107,9417],{"class":503},[151,24109,24110,24113,24115,24117],{"class":469,"line":1413},[151,24111,24112],{"class":15210},"    n_ctx",[151,24114,1876],{"class":1869},[151,24116,316],{"class":477},[151,24118,9417],{"class":503},[151,24120,24121,24124,24126],{"class":469,"line":1418},[151,24122,24123],{"class":15210},"    n_gpu_layers",[151,24125,1876],{"class":1869},[151,24127,24128],{"class":477},"30\n",[151,24130,24131],{"class":469,"line":2462},[151,24132,3640],{"class":503},[151,24134,24135],{"class":469,"line":2471},[151,24136,1090],{"emptyLinePlaceholder":609},[151,24138,24139,24142,24144,24147,24149,24151],{"class":469,"line":2480},[151,24140,24141],{"class":503},"tokenizer ",[151,24143,1876],{"class":1869},[151,24145,24146],{"class":503}," LlamaTokenizer(",[151,24148,21575],{"class":15210},[151,24150,1876],{"class":1869},[151,24152,24153],{"class":503},"llm)\n",[151,24155,24156],{"class":469,"line":2489},[151,24157,1090],{"emptyLinePlaceholder":609},[151,24159,24160,24163,24165],{"class":469,"line":2497},[151,24161,24162],{"class":477},"TEXT",[151,24164,1876],{"class":1869},[151,24166,24167],{"class":481},"\"在那个已被忘却的日子里，它的世界颠覆了。泥土飞走，出现了一条又深又宽的峡谷，然后泥土又轰隆隆地飞回来，峡谷消失了，在原来峡谷的尽头出现了一座黑色的孤峰。其实，在这片广阔的疆域上，这种事常常发生，泥土飞走又飞回，峡谷出现又消失，然后是孤峰降临，好像是给每次灾变打上一个醒目的标记。褐蚁和几百个同族带着幸存的蚁后向太阳落下的方向走了一段路，建立了新的帝国。\"\n",[151,24169,24170,24173,24175,24178,24180],{"class":469,"line":3140},[151,24171,24172],{"class":503},"tokens ",[151,24174,1876],{"class":1869},[151,24176,24177],{"class":503}," tokenizer.encode(",[151,24179,24162],{"class":477},[151,24181,3640],{"class":503},[459,24183,24186],{"className":24184,"code":24185,"language":997},[995],"print(str(tokens[:4]) + \" ...\")\n",[30,24187,24185],{"__ignoreMap":464},[210,24189,24190],{},[11,24191,24192,24195],{},[151,24193,24194],{},"1, 30505, 32380, 36812"," ...",[459,24197,24200],{"className":24198,"code":24199,"language":997},[995],"for token in tokens:\n    text = tokenizer.decode([token])\n    print(text, end=\" \")\n",[30,24201,24199],{"__ignoreMap":464},[210,24203,24204],{},[11,24205,24206],{},"在 那个 已被 忘 却 的日子 里 ， 它的 世界 颠覆 了 。 泥 土 飞 走 ， 出现了 一条 又 深 又 宽 的 峡谷 ， 然后 泥 土 又 轰 隆 隆 地 飞 回来 ， 峡谷 消失 了 ， 在 原来 峡谷 的 尽头 出现了 一座 黑色 的 孤 峰 。 其实 ， 在这 片 广阔 的 疆 域 上 ， 这种事 常常 发生 ， 泥 土 飞 走 又 飞 回 ， 峡谷 出现 又 消失 ， 然后 是 孤 峰 降临 ， 好像是 给 每次 灾 变 打 上 一个 醒目 的 标记 。 褐 蚁 和 几百 个 同 族 带着 幸 存 的 蚁 后 向 太阳 落 下的 方向 走了 一段 路 ， 建立了 新的 帝国 。",[459,24208,24210],{"className":13136,"code":24209,"language":12886,"meta":464,"style":464},"english_text = \"This is an example of tokenization using a large language model.\"\nenglish_tokens = tokenizer.encode(english_text)\nprint(str(english_tokens[:4]) + \" ...\")\n\nfor token in english_tokens:\n    text = tokenizer.decode([token])\n    print(f\"'{text}'\", end=\" \")\n",[30,24211,24212,24222,24232,24255,24259,24271,24281],{"__ignoreMap":464},[151,24213,24214,24217,24219],{"class":469,"line":470},[151,24215,24216],{"class":503},"english_text ",[151,24218,1876],{"class":1869},[151,24220,24221],{"class":481}," \"This is an example of tokenization using a large language model.\"\n",[151,24223,24224,24227,24229],{"class":469,"line":488},[151,24225,24226],{"class":503},"english_tokens ",[151,24228,1876],{"class":1869},[151,24230,24231],{"class":503}," tokenizer.encode(english_text)\n",[151,24233,24234,24236,24238,24240,24243,24245,24248,24250,24253],{"class":469,"line":500},[151,24235,18513],{"class":2226},[151,24237,12386],{"class":503},[151,24239,15343],{"class":6205},[151,24241,24242],{"class":503},"(english_tokens[:",[151,24244,9187],{"class":477},[151,24246,24247],{"class":503},"]) ",[151,24249,22885],{"class":1869},[151,24251,24252],{"class":481}," \" ...\"",[151,24254,3640],{"class":503},[151,24256,24257],{"class":469,"line":509},[151,24258,1090],{"emptyLinePlaceholder":609},[151,24260,24261,24263,24266,24268],{"class":469,"line":517},[151,24262,16732],{"class":1869},[151,24264,24265],{"class":503}," token ",[151,24267,16417],{"class":1869},[151,24269,24270],{"class":503}," english_tokens:\n",[151,24272,24273,24276,24278],{"class":469,"line":534},[151,24274,24275],{"class":503},"    text ",[151,24277,1876],{"class":1869},[151,24279,24280],{"class":503}," tokenizer.decode([token])\n",[151,24282,24283,24286,24288,24290,24293,24295,24297,24299,24302,24304,24307,24309,24312],{"class":469,"line":1413},[151,24284,24285],{"class":2226},"    print",[151,24287,12386],{"class":503},[151,24289,13214],{"class":12347},[151,24291,24292],{"class":481},"\"'",[151,24294,5729],{"class":477},[151,24296,997],{"class":503},[151,24298,2001],{"class":477},[151,24300,24301],{"class":481},"'\"",[151,24303,106],{"class":503},[151,24305,24306],{"class":15210},"end",[151,24308,1876],{"class":1869},[151,24310,24311],{"class":481},"\" \"",[151,24313,3640],{"class":503},[210,24315,24316],{},[11,24317,24318,24195],{},[151,24319,24320],{},"1, 4013, 338, 385",[210,24322,24323],{},[11,24324,24325],{},"'' 'This' ' is' ' an' ' example' ' of' ' token' 'ization' ' using' ' a' ' large' ' language' ' model' '.'",[11,24327,24328],{},"Here are some key differences between English and Chinese that have implications for how the language is tokenized by large language models:",[76,24330,24331,24334,24337,24340,24343,24346,24349],{},[79,24332,24333],{},"Chinese does not use spaces between words like English does",[79,24335,24336],{},"Chinese words are typically formed from 2 or more characters",[79,24338,24339],{},"Chinese verbs are not conjugated and do not have different tenses",[79,24341,24342],{},"Chinese words don't have singular and plural variants",[79,24344,24345],{},"Chinese grammar is very simple and is similar to English",[79,24347,24348],{},"Chinese characters do not have capitization like ASCII characters",[79,24350,24351],{},"The token represented by the number 1 encodes a starting token",[56,24353,24355],{"id":24354},"imagining-scenes-from-three-body-problem-with-stable-diffusion","Imagining scenes from Three-Body Problem with Stable Diffusion",[11,24357,24358],{},"Here are some images I generated using Stable Diffusion with InvokeAI that depict scenes from the Three-Body Problem book. These scenes portray solutions to the Three-Body Problem that players in the Three-Body game devised. The first is a Confucian system of etiquette for predicting the movement of the three suns. The second is a human-powered computer that Qin Shi Huang used to try to predict the movement of the three suns.",[210,24360,24361],{},[11,24362,24363],{},"Prompt: Ceremonies and etiquette system related to the sun and multiple celestial++ bodies Confucius artistic style",[142,24365,24366],{},[24367,24368],"carousel",{":count":24369,"dir":24370},"8","confucius",[210,24372,24373],{},[11,24374,24375],{},"array of chinese++ warriors++ on a electronics+ circuit+ board qing+ dynasty style art logic puzzle",[142,24377,24378],{},[24367,24379],{":count":24380,"dir":24381},"5","computer",[11,24383,24384],{},"Congrats to the InvokeAI team on the 3.0 release. It has been awesome to use and the current docker compose setup is a huge improvement on the 2.x version.",[56,24386,24388],{"id":24387},"n-body-simulations-cuda-and-threejs","n-body simulations, CUDA and Three.js",[11,24390,24391],{},"The nbody problem has no closed-form analytical solution, but it is possible to do a basic simulation of the three-body problem on consumer hardware and open source software, like NVIDIA and CUDA.",[736,24393,24395],{"id":24394},"three-body-cuda-simulation","Three-Body CUDA simulation",[11,24397,24398],{},"I wrote a simple program with the help of ChatGPT for running nbody problem simulations. The program uses CuPy, a Python library that exposes APIs for doing matrix multiplication to predict the position of three bodies using Euclidian Integration. Here's the script:",[459,24400,24404],{"className":24401,"code":24402,"language":24403,"meta":464,"style":464},"language-py shiki shiki-themes github-light github-dark monokai","import numpy as np\nimport cupy as cp\nimport time\nimport json\n\n# Simulation parameters\nNUM_PARTICLES = 3\nDIMENSIONS = 3 # 3D space\nNUM_STEPS = 30\nDT = 0.1\n\n# Generate initial positions and velocities\nnp_positions = np.random.randn(NUM_PARTICLES, DIMENSIONS)\nnp_velocities = np.random.randn(NUM_PARTICLES, DIMENSIONS)\n\ncp_positions = cp.array(np_positions)\ncp_velocities = cp.array(np_velocities)\n\nnp_ticks = np.expand_dims(np_positions, axis=0)\ncp_ticks = cp.array(np_ticks)\n\n# nbody simulation loop\nstart_time = time.time()\nfor step in range(NUM_STEPS):\n\n    # this gets pairwise differences\n    diff = cp_positions[:, None, :] - cp_positions[None, :, :]\n    distances = cp.sqrt(cp.sum(diff**2, axis=2))\n\n    # avoid division by zero\n    epsilon = 1e-5\n    inv_distances = 1.0 / cp.maximum(distances, epsilon)\n\n    # calculate forces\n    cp_forces = cp.sum((diff.T * inv_distances**3).T, axis=1)\n\n    # update velocities and positions\n    cp_velocities += DT * cp_forces\n    cp_positions += DT * cp_velocities\n    cp_ticks = cp.append(cp_ticks, cp.expand_dims(cp_positions, 0), 0)\n\nsim_time = time.time() - start_time\nprint(\"Simulation time:\", sim_time)\n\n\nclass NumpyArrayEncoder(json.JSONEncoder):\n    def default(self, obj):\n        if isinstance(obj, np.ndarray):\n            return obj.tolist()\n        return json.JSONEncoder.default(self, obj)\n\n\nnp_ticks = cp_ticks.get()\n\n\n# this is data we can work with in python and write to a file\nwith open(\"ticks.json\", \"w\") as f:\n    f.write(json.dumps(np_ticks, cls=NumpyArrayEncoder))\n","py",[30,24405,24406,24418,24430,24437,24443,24447,24452,24462,24474,24484,24494,24498,24503,24521,24538,24542,24552,24562,24566,24585,24595,24599,24604,24614,24631,24635,24640,24665,24690,24694,24699,24709,24725,24729,24734,24764,24768,24773,24789,24803,24822,24826,24841,24853,24857,24861,24879,24896,24906,24913,24925,24929,24933,24942,24946,24950,24955,24978],{"__ignoreMap":464},[151,24407,24408,24410,24413,24415],{"class":469,"line":470},[151,24409,16859],{"class":1869},[151,24411,24412],{"class":503}," numpy ",[151,24414,16998],{"class":1869},[151,24416,24417],{"class":503}," np\n",[151,24419,24420,24422,24425,24427],{"class":469,"line":488},[151,24421,16859],{"class":1869},[151,24423,24424],{"class":503}," cupy ",[151,24426,16998],{"class":1869},[151,24428,24429],{"class":503}," cp\n",[151,24431,24432,24434],{"class":469,"line":500},[151,24433,16859],{"class":1869},[151,24435,24436],{"class":503}," time\n",[151,24438,24439,24441],{"class":469,"line":509},[151,24440,16859],{"class":1869},[151,24442,24063],{"class":503},[151,24444,24445],{"class":469,"line":517},[151,24446,1090],{"emptyLinePlaceholder":609},[151,24448,24449],{"class":469,"line":534},[151,24450,24451],{"class":1527},"# Simulation parameters\n",[151,24453,24454,24457,24459],{"class":469,"line":1413},[151,24455,24456],{"class":477},"NUM_PARTICLES",[151,24458,19865],{"class":1869},[151,24460,24461],{"class":477}," 3\n",[151,24463,24464,24467,24469,24471],{"class":469,"line":1418},[151,24465,24466],{"class":477},"DIMENSIONS",[151,24468,19865],{"class":1869},[151,24470,3650],{"class":477},[151,24472,24473],{"class":1527}," # 3D space\n",[151,24475,24476,24479,24481],{"class":469,"line":2462},[151,24477,24478],{"class":477},"NUM_STEPS",[151,24480,19865],{"class":1869},[151,24482,24483],{"class":477}," 30\n",[151,24485,24486,24489,24491],{"class":469,"line":2471},[151,24487,24488],{"class":477},"DT",[151,24490,19865],{"class":1869},[151,24492,24493],{"class":477}," 0.1\n",[151,24495,24496],{"class":469,"line":2480},[151,24497,1090],{"emptyLinePlaceholder":609},[151,24499,24500],{"class":469,"line":2489},[151,24501,24502],{"class":1527},"# Generate initial positions and velocities\n",[151,24504,24505,24508,24510,24513,24515,24517,24519],{"class":469,"line":2497},[151,24506,24507],{"class":503},"np_positions ",[151,24509,1876],{"class":1869},[151,24511,24512],{"class":503}," np.random.randn(",[151,24514,24456],{"class":477},[151,24516,106],{"class":503},[151,24518,24466],{"class":477},[151,24520,3640],{"class":503},[151,24522,24523,24526,24528,24530,24532,24534,24536],{"class":469,"line":3140},[151,24524,24525],{"class":503},"np_velocities ",[151,24527,1876],{"class":1869},[151,24529,24512],{"class":503},[151,24531,24456],{"class":477},[151,24533,106],{"class":503},[151,24535,24466],{"class":477},[151,24537,3640],{"class":503},[151,24539,24540],{"class":469,"line":3149},[151,24541,1090],{"emptyLinePlaceholder":609},[151,24543,24544,24547,24549],{"class":469,"line":3158},[151,24545,24546],{"class":503},"cp_positions ",[151,24548,1876],{"class":1869},[151,24550,24551],{"class":503}," cp.array(np_positions)\n",[151,24553,24554,24557,24559],{"class":469,"line":3167},[151,24555,24556],{"class":503},"cp_velocities ",[151,24558,1876],{"class":1869},[151,24560,24561],{"class":503}," cp.array(np_velocities)\n",[151,24563,24564],{"class":469,"line":3175},[151,24565,1090],{"emptyLinePlaceholder":609},[151,24567,24568,24571,24573,24576,24579,24581,24583],{"class":469,"line":3184},[151,24569,24570],{"class":503},"np_ticks ",[151,24572,1876],{"class":1869},[151,24574,24575],{"class":503}," np.expand_dims(np_positions, ",[151,24577,24578],{"class":15210},"axis",[151,24580,1876],{"class":1869},[151,24582,9181],{"class":477},[151,24584,3640],{"class":503},[151,24586,24587,24590,24592],{"class":469,"line":3193},[151,24588,24589],{"class":503},"cp_ticks ",[151,24591,1876],{"class":1869},[151,24593,24594],{"class":503}," cp.array(np_ticks)\n",[151,24596,24597],{"class":469,"line":3720},[151,24598,1090],{"emptyLinePlaceholder":609},[151,24600,24601],{"class":469,"line":3729},[151,24602,24603],{"class":1527},"# nbody simulation loop\n",[151,24605,24606,24609,24611],{"class":469,"line":3735},[151,24607,24608],{"class":503},"start_time ",[151,24610,1876],{"class":1869},[151,24612,24613],{"class":503}," time.time()\n",[151,24615,24616,24618,24621,24623,24625,24627,24629],{"class":469,"line":3745},[151,24617,16732],{"class":1869},[151,24619,24620],{"class":503}," step ",[151,24622,16417],{"class":1869},[151,24624,2793],{"class":2226},[151,24626,12386],{"class":503},[151,24628,24478],{"class":477},[151,24630,15264],{"class":503},[151,24632,24633],{"class":469,"line":3754},[151,24634,1090],{"emptyLinePlaceholder":609},[151,24636,24637],{"class":469,"line":3760},[151,24638,24639],{"class":1527},"    # this gets pairwise differences\n",[151,24641,24642,24645,24647,24650,24652,24655,24657,24660,24662],{"class":469,"line":3773},[151,24643,24644],{"class":503},"    diff ",[151,24646,1876],{"class":1869},[151,24648,24649],{"class":503}," cp_positions[:, ",[151,24651,15437],{"class":477},[151,24653,24654],{"class":503},", :] ",[151,24656,12445],{"class":1869},[151,24658,24659],{"class":503}," cp_positions[",[151,24661,15437],{"class":477},[151,24663,24664],{"class":503},", :, :]\n",[151,24666,24667,24670,24672,24675,24678,24680,24682,24684,24686,24688],{"class":469,"line":3782},[151,24668,24669],{"class":503},"    distances ",[151,24671,1876],{"class":1869},[151,24673,24674],{"class":503}," cp.sqrt(cp.sum(diff",[151,24676,24677],{"class":1869},"**",[151,24679,6619],{"class":477},[151,24681,106],{"class":503},[151,24683,24578],{"class":15210},[151,24685,1876],{"class":1869},[151,24687,6619],{"class":477},[151,24689,12451],{"class":503},[151,24691,24692],{"class":469,"line":3791},[151,24693,1090],{"emptyLinePlaceholder":609},[151,24695,24696],{"class":469,"line":3803},[151,24697,24698],{"class":1527},"    # avoid division by zero\n",[151,24700,24701,24704,24706],{"class":469,"line":3811},[151,24702,24703],{"class":503},"    epsilon ",[151,24705,1876],{"class":1869},[151,24707,24708],{"class":477}," 1e-5\n",[151,24710,24711,24714,24716,24719,24722],{"class":469,"line":3820},[151,24712,24713],{"class":503},"    inv_distances ",[151,24715,1876],{"class":1869},[151,24717,24718],{"class":477}," 1.0",[151,24720,24721],{"class":1869}," /",[151,24723,24724],{"class":503}," cp.maximum(distances, epsilon)\n",[151,24726,24727],{"class":469,"line":7084},[151,24728,1090],{"emptyLinePlaceholder":609},[151,24730,24731],{"class":469,"line":7148},[151,24732,24733],{"class":1527},"    # calculate forces\n",[151,24735,24736,24739,24741,24744,24746,24749,24751,24753,24756,24758,24760,24762],{"class":469,"line":7211},[151,24737,24738],{"class":503},"    cp_forces ",[151,24740,1876],{"class":1869},[151,24742,24743],{"class":503}," cp.sum((diff.T ",[151,24745,23268],{"class":1869},[151,24747,24748],{"class":503}," inv_distances",[151,24750,24677],{"class":1869},[151,24752,6557],{"class":477},[151,24754,24755],{"class":503},").T, ",[151,24757,24578],{"class":15210},[151,24759,1876],{"class":1869},[151,24761,6760],{"class":477},[151,24763,3640],{"class":503},[151,24765,24766],{"class":469,"line":7273},[151,24767,1090],{"emptyLinePlaceholder":609},[151,24769,24770],{"class":469,"line":7335},[151,24771,24772],{"class":1527},"    # update velocities and positions\n",[151,24774,24775,24778,24781,24784,24786],{"class":469,"line":7398},[151,24776,24777],{"class":503},"    cp_velocities ",[151,24779,24780],{"class":1869},"+=",[151,24782,24783],{"class":477}," DT",[151,24785,12439],{"class":1869},[151,24787,24788],{"class":503}," cp_forces\n",[151,24790,24791,24794,24796,24798,24800],{"class":469,"line":7462},[151,24792,24793],{"class":503},"    cp_positions ",[151,24795,24780],{"class":1869},[151,24797,24783],{"class":477},[151,24799,12439],{"class":1869},[151,24801,24802],{"class":503}," cp_velocities\n",[151,24804,24805,24808,24810,24813,24815,24818,24820],{"class":469,"line":7467},[151,24806,24807],{"class":503},"    cp_ticks ",[151,24809,1876],{"class":1869},[151,24811,24812],{"class":503}," cp.append(cp_ticks, cp.expand_dims(cp_positions, ",[151,24814,9181],{"class":477},[151,24816,24817],{"class":503},"), ",[151,24819,9181],{"class":477},[151,24821,3640],{"class":503},[151,24823,24824],{"class":469,"line":7532},[151,24825,1090],{"emptyLinePlaceholder":609},[151,24827,24828,24831,24833,24836,24838],{"class":469,"line":7537},[151,24829,24830],{"class":503},"sim_time ",[151,24832,1876],{"class":1869},[151,24834,24835],{"class":503}," time.time() ",[151,24837,12445],{"class":1869},[151,24839,24840],{"class":503}," start_time\n",[151,24842,24843,24845,24847,24850],{"class":469,"line":7603},[151,24844,18513],{"class":2226},[151,24846,12386],{"class":503},[151,24848,24849],{"class":481},"\"Simulation time:\"",[151,24851,24852],{"class":503},", sim_time)\n",[151,24854,24855],{"class":469,"line":7608},[151,24856,1090],{"emptyLinePlaceholder":609},[151,24858,24859],{"class":469,"line":7673},[151,24860,1090],{"emptyLinePlaceholder":609},[151,24862,24863,24865,24868,24870,24872,24874,24877],{"class":469,"line":7678},[151,24864,16519],{"class":12347},[151,24866,24867],{"class":15254}," NumpyArrayEncoder",[151,24869,12386],{"class":503},[151,24871,6196],{"class":15260},[151,24873,643],{"class":503},[151,24875,24876],{"class":15260},"JSONEncoder",[151,24878,15264],{"class":503},[151,24880,24881,24883,24885,24887,24889,24891,24894],{"class":469,"line":7708},[151,24882,16566],{"class":12347},[151,24884,19470],{"class":473},[151,24886,12386],{"class":503},[151,24888,15277],{"class":15232},[151,24890,106],{"class":503},[151,24892,24893],{"class":15232},"obj",[151,24895,15264],{"class":503},[151,24897,24898,24900,24903],{"class":469,"line":7713},[151,24899,23357],{"class":1869},[151,24901,24902],{"class":2226}," isinstance",[151,24904,24905],{"class":503},"(obj, np.ndarray):\n",[151,24907,24908,24910],{"class":469,"line":7746},[151,24909,15386],{"class":1869},[151,24911,24912],{"class":503}," obj.tolist()\n",[151,24914,24915,24917,24920,24922],{"class":469,"line":7751},[151,24916,16833],{"class":1869},[151,24918,24919],{"class":503}," json.JSONEncoder.default(",[151,24921,15277],{"class":15289},[151,24923,24924],{"class":503},", obj)\n",[151,24926,24927],{"class":469,"line":7816},[151,24928,1090],{"emptyLinePlaceholder":609},[151,24930,24931],{"class":469,"line":7821},[151,24932,1090],{"emptyLinePlaceholder":609},[151,24934,24935,24937,24939],{"class":469,"line":7847},[151,24936,24570],{"class":503},[151,24938,1876],{"class":1869},[151,24940,24941],{"class":503}," cp_ticks.get()\n",[151,24943,24944],{"class":469,"line":7852},[151,24945,1090],{"emptyLinePlaceholder":609},[151,24947,24948],{"class":469,"line":7887},[151,24949,1090],{"emptyLinePlaceholder":609},[151,24951,24952],{"class":469,"line":7892},[151,24953,24954],{"class":1527},"# this is data we can work with in python and write to a file\n",[151,24956,24957,24960,24962,24964,24967,24969,24972,24974,24976],{"class":469,"line":7924},[151,24958,24959],{"class":1869},"with",[151,24961,16970],{"class":2226},[151,24963,12386],{"class":503},[151,24965,24966],{"class":481},"\"ticks.json\"",[151,24968,106],{"class":503},[151,24970,24971],{"class":481},"\"w\"",[151,24973,16995],{"class":503},[151,24975,16998],{"class":1869},[151,24977,17001],{"class":503},[151,24979,24980,24983,24986,24988],{"class":469,"line":7929},[151,24981,24982],{"class":503},"    f.write(json.dumps(np_ticks, ",[151,24984,24985],{"class":15210},"cls",[151,24987,1876],{"class":1869},[151,24989,24990],{"class":503},"NumpyArrayEncoder))\n",[11,24992,24993],{},"To better understand the matrix math here I walked through a simple example of what each step does:",[459,24995,24997],{"className":24401,"code":24996,"language":24403,"meta":464,"style":464},"# particle coordinates (x,y,z) in 3D space\npositions = cp.array([[1,2.5,3], [4,5,6], [7,8,9]])\n",[30,24998,24999,25004],{"__ignoreMap":464},[151,25000,25001],{"class":469,"line":470},[151,25002,25003],{"class":1527},"# particle coordinates (x,y,z) in 3D space\n",[151,25005,25006,25009,25011,25014,25016,25018,25021,25023,25025,25028,25030,25032,25034,25036,25039,25041,25044,25046,25048,25050,25052],{"class":469,"line":488},[151,25007,25008],{"class":503},"positions ",[151,25010,1876],{"class":1869},[151,25012,25013],{"class":503}," cp.array([[",[151,25015,6760],{"class":477},[151,25017,3634],{"class":503},[151,25019,25020],{"class":477},"2.5",[151,25022,3634],{"class":503},[151,25024,6557],{"class":477},[151,25026,25027],{"class":503},"], [",[151,25029,9187],{"class":477},[151,25031,3634],{"class":503},[151,25033,24380],{"class":477},[151,25035,3634],{"class":503},[151,25037,25038],{"class":477},"6",[151,25040,25027],{"class":503},[151,25042,25043],{"class":477},"7",[151,25045,3634],{"class":503},[151,25047,24369],{"class":477},[151,25049,3634],{"class":503},[151,25051,7918],{"class":477},[151,25053,25054],{"class":503},"]])\n",[11,25056,25057],{},"The first operation creates an array for pairwise distances for each dimension:",[459,25059,25061],{"className":13136,"code":25060,"language":12886,"meta":464,"style":464},"diff = positions[None, :, :] - positions[:, None, :]\nprint(diff)\n\narray([[[ 0. ,  0. ,  0. ],\n        [ 3. ,  2.5,  3. ],\n        [ 6. ,  5.5,  6. ]],\n\n       [[-3. , -2.5, -3. ],\n        [ 0. ,  0. ,  0. ],\n        [ 3. ,  3. ,  3. ]],\n\n       [[-6. , -5.5, -6. ],\n        [-3. , -3. , -3. ],\n        [ 0. ,  0. ,  0. ]]])\n",[30,25062,25063,25088,25095,25099,25118,25136,25154,25158,25182,25198,25214,25218,25240,25262],{"__ignoreMap":464},[151,25064,25065,25068,25070,25073,25075,25078,25080,25083,25085],{"class":469,"line":470},[151,25066,25067],{"class":503},"diff ",[151,25069,1876],{"class":1869},[151,25071,25072],{"class":503}," positions[",[151,25074,15437],{"class":477},[151,25076,25077],{"class":503},", :, :] ",[151,25079,12445],{"class":1869},[151,25081,25082],{"class":503}," positions[:, ",[151,25084,15437],{"class":477},[151,25086,25087],{"class":503},", :]\n",[151,25089,25090,25092],{"class":469,"line":488},[151,25091,18513],{"class":2226},[151,25093,25094],{"class":503},"(diff)\n",[151,25096,25097],{"class":469,"line":500},[151,25098,1090],{"emptyLinePlaceholder":609},[151,25100,25101,25104,25106,25109,25111,25113,25115],{"class":469,"line":509},[151,25102,25103],{"class":503},"array([[[ ",[151,25105,9181],{"class":477},[151,25107,25108],{"class":503},". ,  ",[151,25110,9181],{"class":477},[151,25112,25108],{"class":503},[151,25114,9181],{"class":477},[151,25116,25117],{"class":503},". ],\n",[151,25119,25120,25123,25125,25127,25129,25132,25134],{"class":469,"line":517},[151,25121,25122],{"class":503},"        [ ",[151,25124,6557],{"class":477},[151,25126,25108],{"class":503},[151,25128,25020],{"class":477},[151,25130,25131],{"class":503},",  ",[151,25133,6557],{"class":477},[151,25135,25117],{"class":503},[151,25137,25138,25140,25142,25144,25147,25149,25151],{"class":469,"line":534},[151,25139,25122],{"class":503},[151,25141,25038],{"class":477},[151,25143,25108],{"class":503},[151,25145,25146],{"class":477},"5.5",[151,25148,25131],{"class":503},[151,25150,25038],{"class":477},[151,25152,25153],{"class":503},". ]],\n",[151,25155,25156],{"class":469,"line":1413},[151,25157,1090],{"emptyLinePlaceholder":609},[151,25159,25160,25163,25165,25167,25170,25172,25174,25176,25178,25180],{"class":469,"line":1418},[151,25161,25162],{"class":503},"       [[",[151,25164,12445],{"class":1869},[151,25166,6557],{"class":477},[151,25168,25169],{"class":503},". , ",[151,25171,12445],{"class":1869},[151,25173,25020],{"class":477},[151,25175,106],{"class":503},[151,25177,12445],{"class":1869},[151,25179,6557],{"class":477},[151,25181,25117],{"class":503},[151,25183,25184,25186,25188,25190,25192,25194,25196],{"class":469,"line":2462},[151,25185,25122],{"class":503},[151,25187,9181],{"class":477},[151,25189,25108],{"class":503},[151,25191,9181],{"class":477},[151,25193,25108],{"class":503},[151,25195,9181],{"class":477},[151,25197,25117],{"class":503},[151,25199,25200,25202,25204,25206,25208,25210,25212],{"class":469,"line":2471},[151,25201,25122],{"class":503},[151,25203,6557],{"class":477},[151,25205,25108],{"class":503},[151,25207,6557],{"class":477},[151,25209,25108],{"class":503},[151,25211,6557],{"class":477},[151,25213,25153],{"class":503},[151,25215,25216],{"class":469,"line":2480},[151,25217,1090],{"emptyLinePlaceholder":609},[151,25219,25220,25222,25224,25226,25228,25230,25232,25234,25236,25238],{"class":469,"line":2489},[151,25221,25162],{"class":503},[151,25223,12445],{"class":1869},[151,25225,25038],{"class":477},[151,25227,25169],{"class":503},[151,25229,12445],{"class":1869},[151,25231,25146],{"class":477},[151,25233,106],{"class":503},[151,25235,12445],{"class":1869},[151,25237,25038],{"class":477},[151,25239,25117],{"class":503},[151,25241,25242,25244,25246,25248,25250,25252,25254,25256,25258,25260],{"class":469,"line":2497},[151,25243,23249],{"class":503},[151,25245,12445],{"class":1869},[151,25247,6557],{"class":477},[151,25249,25169],{"class":503},[151,25251,12445],{"class":1869},[151,25253,6557],{"class":477},[151,25255,25169],{"class":503},[151,25257,12445],{"class":1869},[151,25259,6557],{"class":477},[151,25261,25117],{"class":503},[151,25263,25264,25266,25268,25270,25272,25274,25276],{"class":469,"line":3140},[151,25265,25122],{"class":503},[151,25267,9181],{"class":477},[151,25269,25108],{"class":503},[151,25271,9181],{"class":477},[151,25273,25108],{"class":503},[151,25275,9181],{"class":477},[151,25277,25278],{"class":503},". ]]])\n",[11,25280,25281,25282,106,25284,187,25287,25290],{},"The rows of zeros correspond to a particle's ",[30,25283,11126],{},[30,25285,25286],{},"y",[30,25288,25289],{},"z"," distances to itself, which are all zero by axioms of Euclidian vector spaces.",[11,25292,25293],{},"The next operation calculates the distance between each particle:",[459,25295,25297],{"className":13136,"code":25296,"language":12886,"meta":464,"style":464},"distances = cp.sqrt(cp.sum(diff**2, axis=2))\nprint(distances)\n\narray([[ 0.        ,  4.9244289 , 10.11187421],\n       [ 4.9244289 ,  0.        ,  5.19615242],\n       [10.11187421,  5.19615242,  0.        ]])\n",[30,25298,25299,25322,25329,25333,25354,25373],{"__ignoreMap":464},[151,25300,25301,25304,25306,25308,25310,25312,25314,25316,25318,25320],{"class":469,"line":470},[151,25302,25303],{"class":503},"distances ",[151,25305,1876],{"class":1869},[151,25307,24674],{"class":503},[151,25309,24677],{"class":1869},[151,25311,6619],{"class":477},[151,25313,106],{"class":503},[151,25315,24578],{"class":15210},[151,25317,1876],{"class":1869},[151,25319,6619],{"class":477},[151,25321,12451],{"class":503},[151,25323,25324,25326],{"class":469,"line":488},[151,25325,18513],{"class":2226},[151,25327,25328],{"class":503},"(distances)\n",[151,25330,25331],{"class":469,"line":500},[151,25332,1090],{"emptyLinePlaceholder":609},[151,25334,25335,25338,25340,25343,25346,25349,25352],{"class":469,"line":509},[151,25336,25337],{"class":503},"array([[ ",[151,25339,9181],{"class":477},[151,25341,25342],{"class":503},".        ,  ",[151,25344,25345],{"class":477},"4.9244289",[151,25347,25348],{"class":503}," , ",[151,25350,25351],{"class":477},"10.11187421",[151,25353,18746],{"class":503},[151,25355,25356,25359,25361,25364,25366,25368,25371],{"class":469,"line":517},[151,25357,25358],{"class":503},"       [ ",[151,25360,25345],{"class":477},[151,25362,25363],{"class":503}," ,  ",[151,25365,9181],{"class":477},[151,25367,25342],{"class":503},[151,25369,25370],{"class":477},"5.19615242",[151,25372,18746],{"class":503},[151,25374,25375,25378,25380,25382,25384,25386,25388],{"class":469,"line":534},[151,25376,25377],{"class":503},"       [",[151,25379,25351],{"class":477},[151,25381,25131],{"class":503},[151,25383,25370],{"class":477},[151,25385,25131],{"class":503},[151,25387,9181],{"class":477},[151,25389,25390],{"class":503},".        ]])\n",[11,25392,25393,25394,25396],{},"The diagonal or zeros represents that fact that a particle ",[30,25395,8521],{}," has a distance of zero to iself.",[11,25398,25399,25400,25403],{},"The next step calculates inverse distances and uses a small ",[30,25401,25402],{},"epsilon"," value to avoid division by 0:",[459,25405,25407],{"className":13136,"code":25406,"language":12886,"meta":464,"style":464},"epsilon = 1e-5\ninv_distances = 1.0 / cp.maximum(distances, epsilon)\nprint(inv_distances)\n\narray([[1.00000000e+05, 2.03069233e-01, 9.88936353e-02],\n       [2.03069233e-01, 1.00000000e+05, 1.92450090e-01],\n       [9.88936353e-02, 1.92450090e-01, 1.00000000e+05]])\n",[30,25408,25409,25418,25431,25438,25442,25462,25479],{"__ignoreMap":464},[151,25410,25411,25414,25416],{"class":469,"line":470},[151,25412,25413],{"class":503},"epsilon ",[151,25415,1876],{"class":1869},[151,25417,24708],{"class":477},[151,25419,25420,25423,25425,25427,25429],{"class":469,"line":488},[151,25421,25422],{"class":503},"inv_distances ",[151,25424,1876],{"class":1869},[151,25426,24718],{"class":477},[151,25428,24721],{"class":1869},[151,25430,24724],{"class":503},[151,25432,25433,25435],{"class":469,"line":500},[151,25434,18513],{"class":2226},[151,25436,25437],{"class":503},"(inv_distances)\n",[151,25439,25440],{"class":469,"line":509},[151,25441,1090],{"emptyLinePlaceholder":609},[151,25443,25444,25447,25450,25452,25455,25457,25460],{"class":469,"line":517},[151,25445,25446],{"class":503},"array([[",[151,25448,25449],{"class":477},"1.00000000e+05",[151,25451,106],{"class":503},[151,25453,25454],{"class":477},"2.03069233e-01",[151,25456,106],{"class":503},[151,25458,25459],{"class":477},"9.88936353e-02",[151,25461,18746],{"class":503},[151,25463,25464,25466,25468,25470,25472,25474,25477],{"class":469,"line":534},[151,25465,25377],{"class":503},[151,25467,25454],{"class":477},[151,25469,106],{"class":503},[151,25471,25449],{"class":477},[151,25473,106],{"class":503},[151,25475,25476],{"class":477},"1.92450090e-01",[151,25478,18746],{"class":503},[151,25480,25481,25483,25485,25487,25489,25491,25493],{"class":469,"line":1413},[151,25482,25377],{"class":503},[151,25484,25459],{"class":477},[151,25486,106],{"class":503},[151,25488,25476],{"class":477},[151,25490,106],{"class":503},[151,25492,25449],{"class":477},[151,25494,25054],{"class":503},[11,25496,25497],{},"The next step is the most elegant part of the simulation and really flexes the GPU's parallel compute capabilities:",[459,25499,25501],{"className":13136,"code":25500,"language":12886,"meta":464,"style":464},"cp_forces = cp.sum((diff.T * inv_distances**3).T, axis=1)\n",[30,25502,25503],{"__ignoreMap":464},[151,25504,25505,25508,25510,25512,25514,25516,25518,25520,25522,25524,25526,25528],{"class":469,"line":470},[151,25506,25507],{"class":503},"cp_forces ",[151,25509,1876],{"class":1869},[151,25511,24743],{"class":503},[151,25513,23268],{"class":1869},[151,25515,24748],{"class":503},[151,25517,24677],{"class":1869},[151,25519,6557],{"class":477},[151,25521,24755],{"class":503},[151,25523,24578],{"class":15210},[151,25525,1876],{"class":1869},[151,25527,6760],{"class":477},[151,25529,3640],{"class":503},[11,25531,19225,25532,25535],{},[30,25533,25534],{},".T"," operation transposes a matrix, multiplies by the cube of inverse distances, then transposes the matrix again before summing along the first axis. Transposing a matrix basically swaps rows and columns.",[11,25537,25538],{},"The next two steps are also pretty elegant:",[459,25540,25542],{"className":13136,"code":25541,"language":12886,"meta":464,"style":464},"# update velocities and positions\ncp_velocities += DT * cp_forces\ncp_positions += DT * cp_velocities\n",[30,25543,25544,25549,25561],{"__ignoreMap":464},[151,25545,25546],{"class":469,"line":470},[151,25547,25548],{"class":1527},"# update velocities and positions\n",[151,25550,25551,25553,25555,25557,25559],{"class":469,"line":488},[151,25552,24556],{"class":503},[151,25554,24780],{"class":1869},[151,25556,24783],{"class":477},[151,25558,12439],{"class":1869},[151,25560,24788],{"class":503},[151,25562,25563,25565,25567,25569,25571],{"class":469,"line":500},[151,25564,24546],{"class":503},[151,25566,24780],{"class":1869},[151,25568,24783],{"class":477},[151,25570,12439],{"class":1869},[151,25572,24802],{"class":503},[11,25574,25575,25576,25578],{},"In the last step I append the updated positions to an array that holds every \"tick\" (the positions of each particle between each time interval, ",[30,25577,24488],{}," - \"delta time\")",[11,25580,25581],{},"Here's the formula for the mathematical equation used to calculate the force on any given body in an n-body system:",[23881,25583],{"src":25584,"width":23883,"height":25585},"https://briancaffey.github.io/three-body-problem/iframe/formula.html",110,[11,25587,25588],{},"To test that the simulation was working correctly I used ChatGPT again to construct a 3D scene in Blender with a Python script:",[11,25590,25591],{},[2718,25592],{"alt":25593,"src":25594},"Blender Animation","/static/three-body-problem/blender.png",[736,25596,25598],{"id":25597},"threejs","Three.js",[11,25600,25601],{},"Imagine that we are working for a Chinese startup called the Qin Dynasty. The founder, Qin Shi Huang, is a brutal tyrant with an obsessive fear of assassination. Let's put on our product hat for a minute and think about how we can impress him with a clean solution to the three-body problem. A recent attempt involved building a 30,000-person analog computer that was destroyed in tri-solar syzygy. Failing to accurately predict the movement of the suns could mean execution by live-burial, a fiery death or worse. Using CUDA and Blender is a good MVP but doesn't make for the best technical demo since it involves so many different steps: running the simulation in CUDA, exporting data to JSON, loading data into a visualuzation and then finally rendering a video of the simulation. With a popular Javascript library called Three.js we can run an interactive three-body problem simulation in real-time right in the browser. Here's the three-body simulation I also co-authored with ChatGPT-4 using Three.js:",[23881,25603],{"src":25604,"width":23883,"height":25605},"https://briancaffey.github.io/three-body-problem/three/",350,[56,25607,25609],{"id":25608},"screen-adaptations-the-gaming-industry-and-the-ccp","Screen adaptations, the gaming industry and the CCP",[11,25611,25612],{},"Dream of the Red Chamber is one of China's Four Great Classical Novels and is often seen as the pinnacle of Chinese fiction. It was written in the mid 18th century and first published in 1791. It is a long saga that totals 960,000 characters in length, on similar scale to the length of the Three-Body Problem trilogy. Sun Wen, a Qing dynasty artist, spent 36 years of his life doing a series of 230 paintings depicting scenes from the Dream of the Red Chamber: dream sequences, demons, goddesses, nuns, nobles, beggars, raging fires, landscapes, interiors, wildlife, gardens, temples, funerals, battles, processions, banquets, trials, operas, marriages.",[11,25614,25615],{},[2718,25616],{"alt":25617,"src":25618},"Sun Wen paintings sample","/static/three-body-problem/dorc.png",[11,25620,25621],{},"Following in this tradition of celebrating great literature, Tencent Video and China Central Television produced a 30-episode adaptation of the Three-Body Problem that was released in February 2023. It is a surprisingly faithful reproduction of the book that is worth checking out. The portrayal of Shi Qiang (Da Shi) was easily my favorite part of the series. I was also impressed by how the Three-Body VR game scenes were done with computer graphics. It got me thinking about how China is represented in some of the worlds most popular video games.",[11,25623,25624],{},[2718,25625],{"alt":25626,"src":25627},"games","/static/three-body-problem/game.png",[11,25629,25630],{},"This is Rocket League, a competitive vehicular soccer game where players, like in the Three-Body game, must master the laws of gravity. The Chinese-themed Forbiden Temple arena shown here is one of many virtual international venues in the game. Epic Games (creator of Fortnite) bought Rocket League in 2019 for an estimated $250 to $300 million.",[11,25632,25633],{},"Like Rocket League, Overwatch is a highly-competitive eSport on a global scale. It features a large roster of 38 players from all over the world. Dr. Mei-Ling Zhou (周美灵) is a Chinese climatologist who uses ice both to attack opponents and to defend herself. Mei became controversial in China due to her adoption as a symbolic figure in the 2019 Hong Kong protests.",[11,25635,25636],{},"Three college friends combined their interest in anime, comics and games (ACG) and literature to publish one of the most successful games created by a Chinese company and arguably one of China's most important cultural exports: Genshin Impact. miHoYo, the Shanghai-based company that develops Genshin Impact, grossed $4 billion of revenue globally in the game's first year setting a new record in the gaming industry. Like other companies of its size, miHoYo has a party committee under the Chinese Communist Party that influences the company's operations.",[56,25638,25640],{"id":25639},"ai-and-layoffs-in-the-tech-industry","AI and layoffs in the tech industry",[11,25642,25643,25644,643],{},"I have a positive attitude toward AI and its ability to supercharge the creative work we do, but I also think that replacing humans and AI-related layoffs should should be a part of the conversation. I wasn't impacted by the recent round of layoffs at my company, and I'm grateful to have the opportunity to work with a talented team on interesting problems in the health tech industry. I do have some solid references for folks in DevOps, product and backend and frontend engineering, and I’m happy to ",[20,25645,25648],{"href":25646,"rel":25647},"https://www.linkedin.com/in/brian-caffey-06b22a18/",[24],"connect and share via LinkedIn",[11,25650,25651],{},"Layoffs in both China and the U.S. have been pummled the tech sector over the last two years. New graduates in China are also facing a difficult job market. There are some popular expressions that paint a picture of the job market in China:",[76,25653,25654,25657,25660,25668,25677],{},[79,25655,25656],{},"躺平 Lying flat: avoiding relentless work",[79,25658,25659],{},"九九六 996 Work culture: describes working from 9AM to 9PM, 6 days a week, a common work schedule for many Chinese employees",[79,25661,25662,25663],{},"全职儿女 Full-time Children: ",[20,25664,25667],{"href":25665,"rel":25666},"https://www.cnn.com/2023/07/26/economy/china-youth-unemployment-intl-hnk/index.html",[24],"Young Chinese are getting paid to be ‘full-time children’ as jobs become harder to find",[79,25669,25670,25671,25676],{},"35岁诅咒 The curse of 35: ",[20,25672,25675],{"href":25673,"rel":25674},"https://www.marketplace.org/shows/marketplace-tech/chinas-tech-workers-ageism-the-curse-of-35/",[24],"Ageism in China’s tech sector has workers fearing the “curse of 35”",". Shout out to my fellow Year of the Dragon 35 year olds! 🐲",[79,25678,25679,25680],{},"吃苦 Eat Bitterness: ",[20,25681,25684],{"href":25682,"rel":25683},"https://www.nytimes.com/2023/05/30/business/china-youth-unemployment.html",[24],"China’s Young People Can’t Find Jobs. Xi Jinping Says to ‘Eat Bitterness.’",[11,25686,25687,25688,25693],{},"It is an exciting time for AI. Elon Musk and Kaifu Lee have both recently released open-source large language models: Grok and Yi. Sam Altman was fired as CEO of OpenAI, then came back. Stable Diffusion just released a text to video model. AGI might already be here. In the U.S., we are going into our first presidential election cycle with AI fully turned on. Here's a ",[20,25689,25692],{"href":25690,"rel":25691},"https://www.amazon.com/Three-Body-Problem-Cixin-Liu/dp/0765382032",[24],"link to The Three-Body Problem book on Amazon",". Thanks for reading and Happy Thanksgiving!",[11,25695,25696],{},[2718,25697],{"alt":25698,"src":25699},"Happy Thanksgiving","/static/three-body-problem/thanksgiving.png",[589,25701,25702],{},"html pre.shiki code .sC2Qs, html code.shiki .sC2Qs{--shiki-default:#D73A49;--shiki-dark:#F97583;--shiki-sepia:#F92672}html pre.shiki code .sMOD_, html code.shiki .sMOD_{--shiki-default:#24292E;--shiki-dark:#E1E4E8;--shiki-sepia:#F8F8F2}html pre.shiki code .sTHNf, html code.shiki .sTHNf{--shiki-default:#E36209;--shiki-default-font-style:inherit;--shiki-dark:#FFAB70;--shiki-dark-font-style:inherit;--shiki-sepia:#FD971F;--shiki-sepia-font-style:italic}html pre.shiki code .sstjo, html code.shiki .sstjo{--shiki-default:#032F62;--shiki-dark:#9ECBFF;--shiki-sepia:#E6DB74}html pre.shiki code .s7F3e, html code.shiki .s7F3e{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#AE81FF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html .sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html.sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html pre.shiki code .sTrkL, html code.shiki .sTrkL{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#66D9EF}html pre.shiki code .s-m8C, html code.shiki .s-m8C{--shiki-default:#005CC5;--shiki-default-font-style:inherit;--shiki-dark:#79B8FF;--shiki-dark-font-style:inherit;--shiki-sepia:#66D9EF;--shiki-sepia-font-style:italic}html pre.shiki code .sq6CD, html code.shiki .sq6CD{--shiki-default:#D73A49;--shiki-default-font-style:inherit;--shiki-dark:#F97583;--shiki-dark-font-style:inherit;--shiki-sepia:#66D9EF;--shiki-sepia-font-style:italic}html pre.shiki code .s8-w5, html code.shiki .s8-w5{--shiki-default:#6A737D;--shiki-dark:#6A737D;--shiki-sepia:#88846F}html pre.shiki code .sz2Vg, html code.shiki .sz2Vg{--shiki-default:#6F42C1;--shiki-default-text-decoration:inherit;--shiki-dark:#B392F0;--shiki-dark-text-decoration:inherit;--shiki-sepia:#A6E22E;--shiki-sepia-text-decoration:underline}html pre.shiki code .s30JN, html code.shiki .s30JN{--shiki-default:#6F42C1;--shiki-default-font-style:inherit;--shiki-default-text-decoration:inherit;--shiki-dark:#B392F0;--shiki-dark-font-style:inherit;--shiki-dark-text-decoration:inherit;--shiki-sepia:#A6E22E;--shiki-sepia-font-style:italic;--shiki-sepia-text-decoration:underline}html pre.shiki code .srTi1, html code.shiki .srTi1{--shiki-default:#6F42C1;--shiki-dark:#B392F0;--shiki-sepia:#A6E22E}html pre.shiki code .so59x, html code.shiki .so59x{--shiki-default:#24292E;--shiki-default-font-style:inherit;--shiki-dark:#E1E4E8;--shiki-dark-font-style:inherit;--shiki-sepia:#FD971F;--shiki-sepia-font-style:italic}html pre.shiki code .sP7S_, html code.shiki .sP7S_{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#FD971F}",{"title":464,"searchDepth":488,"depth":488,"links":25704},[25705,25706,25709,25710,25713,25714,25718,25719],{"id":23830,"depth":488,"text":23831},{"id":23893,"depth":488,"text":23894,"children":25707},[25708],{"id":23938,"depth":500,"text":23939},{"id":23962,"depth":488,"text":23963},{"id":24001,"depth":488,"text":24002,"children":25711},[25712],{"id":24047,"depth":500,"text":24048},{"id":24354,"depth":488,"text":24355},{"id":24387,"depth":488,"text":24388,"children":25715},[25716,25717],{"id":24394,"depth":500,"text":24395},{"id":25597,"depth":500,"text":25598},{"id":25608,"depth":488,"text":25609},{"id":25639,"depth":488,"text":25640},"2023-11-23","Translating The Three-Body Problem book to English with Chinese LLMs, making visualizations with stable diffusion and running n-body simulations with CUDA",[25723,25726,25728],{"link":25724,"site":25725},"https://news.ycombinator.com/item?id=38393757","hn",{"link":25727,"site":23769},"https://www.reddit.com/r/threebodyproblem/comments/1823l5j/python_vue_chinesellama2_and_the_threebody_problem/",{"link":25729,"site":11126},"https://twitter.com/briancaffey/status/1727710878349332614","/static/three-body-problem/cover.png",{},"/2023/08/27/python-vue-chinese-llama-2-and-the-three-body-problem",{"title":23783,"description":25721},"2023/08/27/python-vue-chinese-llama-2-and-the-three-body-problem",[25736,21575,615,614,19404,12886,11133,25737,21573,21577,21572,12646,25738],"three-body-problem","cuda","three.js","E2nj5gEJprRwLJsRZyvvyzuI1LqgPEy4id_faTp7zVI",{"id":25741,"title":25742,"body":25743,"comments":609,"date":30098,"description":30099,"draft":602,"extension":605,"external":30100,"image":30116,"meta":30117,"navigation":609,"path":30118,"seo":30119,"stem":30120,"tags":30121,"__hash__":30130},"blog/2023/01/07/i-deployed-the-same-containerized-serverless-django-app-with-aws-cdk-terraform-and-pulumi.md","My Infrastructure as Code Rosetta Stone - Deploying the same web application on AWS ECS Fargate with CDK, Terraform and Pulumi",{"type":8,"value":25744,"toc":30043},[25745,25747,25750,25800,25803,25806,25809,25820,25824,25829,25837,25842,25853,25858,25869,25874,25893,25902,25914,25919,25930,25935,25943,25948,25966,25970,25974,25977,25996,26000,26003,26010,26013,26024,26043,26053,26056,26084,26088,26091,26120,26124,26142,26151,26160,26164,26189,26211,26220,26226,26230,26233,26292,26298,26416,26422,26425,26429,26454,26458,26461,26477,26490,26495,26501,26506,26512,26517,26523,26528,26534,26538,26544,26550,26556,26562,26568,26574,26580,26584,26587,26610,26613,26616,26622,26625,26629,26636,26656,26659,26685,26688,26708,26712,26718,26731,26738,26742,26745,26769,26772,26775,26784,26816,26819,26827,26831,26834,26860,26864,26871,26877,26881,26884,26892,26895,26899,26905,26922,26933,26936,26939,27041,27045,27048,27062,27065,27068,27072,27078,27081,27084,27188,27195,27201,27208,27230,27233,27236,27252,27255,27277,27281,27284,27316,27329,27333,27336,27339,27345,27349,27364,27373,27377,27383,27424,27427,27468,27474,27480,27484,27487,27498,27502,27505,27508,27512,27532,27535,27551,27554,27558,27566,27575,27579,27596,27604,27610,27641,27644,27851,27854,27938,27941,27944,27947,28347,28350,28535,28538,29057,29061,29064,29097,29101,29113,29121,29124,29148,29152,29155,29163,29166,29193,29203,29208,29277,29280,29371,29374,29402,29405,29498,29509,29578,29581,29585,29591,29594,29600,29606,29612,29616,29621,29634,29643,29654,29662,29665,29676,29679,29685,29689,29703,29707,29757,29764,29769,29772,29778,29782,29793,29805,29808,29812,29844,29848,29851,29960,29964,29967,30025,30028,30031,30040],[56,25746,16116],{"id":16115},[11,25748,25749],{},"I wrote three infrastructure as code libraries for deploying containerized 3-tier web apps on AWS ECS Fargate using CDK, Terraform and Pulumi. This article will provide an overview of my experience working with these three IaC tools and will show how I use my libraries in automated infrastructure deployment pipelines with GitHub Actions.",[76,25751,25752,25762,25772,25782],{},[79,25753,25754,6208,25757],{},[15,25755,25756],{},"CDK Construct Library",[20,25758,25761],{"href":25759,"rel":25760},"https://github.com/briancaffey/cdk-django",[24],"github.com/briancaffey/cdk-django",[79,25763,25764,6208,25767],{},[15,25765,25766],{},"Terraform Modules",[20,25768,25771],{"href":25769,"rel":25770},"https://github.com/briancaffey/terraform-aws-django",[24],"github.com/briancaffey/terraform-aws-django",[79,25773,25774,6208,25777],{},[15,25775,25776],{},"Pulumi Component Library",[20,25778,25781],{"href":25779,"rel":25780},"https://github.com/briancaffey/pulumi-aws-django",[24],"github.com/briancaffey/pulumi-aws-django",[79,25783,25784,25785,25788,25789,25794,25795],{},"Mono repo with a sample Django micro blogging app (μblog) and frontend app (Vue SPA written with Quasar), GitHub Action workflows for infrastructure and (separate) application deployment pipelines, IaC code that ",[51,25786,25787],{},"consumes"," each of the libraries listed above, ",[20,25790,25793],{"href":25791,"rel":25792},"https://briancaffey.github.io/django-step-by-step/",[24],"VuePress documentation site"," and miscellaneous items (k6 load testing scripts, Cypress tests, docker-compose, etc.): ",[20,25796,25799],{"href":25797,"rel":25798},"https://github.com/briancaffey/django-step-by-step",[24],"github.com/briancaffey/django-step-by-step",[56,25801,25802],{"id":25802},"eli5",[11,25804,25805],{},"Pretend we are at the beach building sandcastles. We can build sandcastles using our hands, but this takes a lot of time, and we might bump into each other and accidentally knock over part of our sandcastle. I made some tools for building sandcastles. We have one tool for building a sand castle base that includes the wall around the outside, the moat, the door, and different sections inside the walls. And I made another tool for deploying smaller sand \\castle houses inside the walls of the sandcastle base. We fill the tool with sand and water and then turn it over inside of our base and we can build an entire city of sandcastles. Also, the tool lets us carefully remove parts of our sandcastle without knocking over any of the other parts. We can share the tool with all of our friends and they can make cool sandcastles too, and the tool is free for them to use.",[11,25807,25808],{},"Instead of sandcastles, I'm working with computer systems that can power internet applications, like YouTube for example. I'm building tools that can allow me or anyone else to build really awesome internet applications using computers.",[11,25810,25811,25812,25815,25816,25819],{},"The tools are not physical tools like the ones for building sandcastles, but instead, these tools are made with code. The code for websites like YouTube allows you to upload videos ",[51,25813,25814],{},"to YouTube",", but the code I'm writing allows you to upload any type of website (even site YouTube) ",[51,25817,25818],{},"to the internet",". When we run this code, it creates applications on the internet. Also, sand is very expensive and Jeff Bezos owns the beach.",[56,25821,25823],{"id":25822},"why-i-made-an-infrastructure-as-code-rosetta-stone-with-cdk-terraform-and-pulumi","Why I made an Infrastructure as Code Rosetta Stone with CDK, Terraform and Pulumi",[11,25825,25826],{},[15,25827,25828],{},"To push me to learn more about AWS, IaC, CI/CD, automation, and Platform Engineering",[76,25830,25831,25834],{},[79,25832,25833],{},"Learn the differences between major IaC tools and how to use them to do exactly the same thing (build a web app) on the same Cloud (AWS) in the same way (serverless container technology using ECS Fargate).",[79,25835,25836],{},"Get more experience publishing software packages (npm) and finding the right level of abstraction for IaC libraries that is both dynamic and straightforward",[11,25838,25839],{},[15,25840,25841],{},"To fail as many times as possible",[76,25843,25844,25847,25850],{},[79,25845,25846],{},"Every time I fail when I think I have things right, I learn something new",[79,25848,25849],{},"Failed IaC pipelines can sometimes be scary, and every failure I have on these projects can teach me about potential failure modes for live projects running in production",[79,25851,25852],{},"You can oftentimes be \"stuck\" where you have a set of resources that you can't update or delete. Learning to get unstuck from these scenarios is important",[11,25854,25855],{},[15,25856,25857],{},"To take an application-first approach to DevOps",[76,25859,25860,25863,25866],{},[79,25861,25862],{},"Application developers are increasingly being tasked with operational duties",[79,25864,25865],{},"While learning about IaC, I had a hard time finding in-depth materials covering application development, CI/CD pipelines, automation, and Infrastructure as Code and how these three knowledge domains work together. There are important considerations to make when  between a Hello World docker image",[79,25867,25868],{},"You could probably use another framework with these IaC libraries like Flask or Rails, but for now I'm building these projects with Django first-in-mind",[11,25870,25871],{},[15,25872,25873],{},"To develop a project I can reference when helping myself and others",[76,25875,25876,25879,25886],{},[79,25877,25878],{},"companies and projects that do IaC and CI/CD for the most part have things in private repos for obvious reasons, there isn't any good reason to share this type of code unless you are sharing it with an auditor",[79,25880,25881,25882,25885],{},"Hopefully, the sample application, IaC, and CI/CD pipelines ",[51,25883,25884],{},"aren't overly complex",". There are more complex examples of open-source companies out there, but their repos have steep learning curves and a lot going on",[79,25887,25888,25889,25892],{},"People often ask about how to split up IaC deployments and application deployments. I want to be able to use this project to ",[15,25890,25891],{},"show"," people how it can be done",[11,25894,25895],{},[15,25896,25897,25898,25901],{},"To encourage others (specifically Developer Advocates / Developer Relations / Solutions Architects in the CDK, Terraform, and Pulumi communities) to share complete and non-trivial examples of IaC software ",[15,25899,25900],{},"in use"," with an actual application.",[76,25903,25904,25911],{},[79,25905,25906,25907,25910],{},"There are many ways one could create an \"IaC Rosetta Stone\" (",[30,25908,25909],{},"public cloud providers x CI/CD providers x IaC tools"," is a big number)",[79,25912,25913],{},"This takes a lot of effort and time",[11,25915,25916],{},[15,25917,25918],{},"I have nothing to sell you",[76,25920,25921,25924,25927],{},[79,25922,25923],{},"So many articles about Cloud/DevOps are trying to sell you a tool. Outside of what I consider to be mainstream vendors like GitHub and AWS, there are no products that I'm promoting here",[79,25925,25926],{},"I'm also not trying to sell anyone on using my IaC packages",[79,25928,25929],{},"Hopefully, my IaC packages can serve as a helpful reference or starting point",[11,25931,25932],{},[15,25933,25934],{},"Walk before running",[76,25936,25937,25940],{},[79,25938,25939],{},"I want to build up confidence with vanilla use cases before getting too fancy",[79,25941,25942],{},"With a solid foundation in these tools, I want to learn about some of the more advanced patterns teams are adopting (Pulumi Automation API, Terragrunt for Terraform, self-mutating CDK Pipelines)",[11,25944,25945],{},[15,25946,25947],{},"12 Factor App, DevOps, and Platform Engineering",[76,25949,25950,25958],{},[79,25951,25952,25957],{},[20,25953,25956],{"href":25954,"rel":25955},"https://12factor.net/",[24],"12 Factor App"," is great and has guided how I approach both Django application development and IaC library development",[79,25959,19225,25960,25965],{},[20,25961,25964],{"href":25962,"rel":25963},"https://platformengineering.org/",[24],"platformengineering.org"," community has some good guiding principles",[56,25967,25969],{"id":25968},"cdkterraformpulumi-terminology","CDK/Terraform/Pulumi terminology",[736,25971,25973],{"id":25972},"constructs-modules-and-components","constructs, modules and components",[11,25975,25976],{},"A CDK construct, Terraform module and Pulumi component generally mean the same thing: an abstract grouping of one or more cloud resources.",[11,25978,25979,25980,25983,25984,25987,25988,25991,25992,25995],{},"In this article I will refer to ",[15,25981,25982],{},"constructs/modules/components"," as ",[15,25985,25986],{},"c/m/c"," for short, and the term ",[15,25989,25990],{},"stack"," can generally be used to refer to either a CloudFormation stack, a Pulumi Stack or a Terraform group of resources that are part of a module that has had ",[30,25993,25994],{},"apply"," ran against it.",[736,25997,25999],{"id":25998},"what-is-a-stack","what is a stack?",[11,26001,26002],{},"AWS has a resource type called CloudFormation Stacks, and Pulumi also has a concept of stacks. Terraform documentation doesn't refer to stacks, and instead in Terraform docs use the words \"Terraform configuration\" to refer to some group of resources that were built using a module.",[11,26004,26005,26006,26009],{},"CDK Constructs and Pulumi Components are somewhat similar, however CDK Constructs map to CloudFormation and the Pulumi components I'm using from the ",[30,26007,26008],{},"@pulumi/aws"," package generally map directly to Terraform resources from the AWS Provider (the Pulumi AWS Provider uses much of the same code that the Terraform AWS Provider uses).",[736,26011,26012],{"id":26012},"verbs",[11,26014,26015,26016,26019,26020,26023],{},"In CDK you ",[30,26017,26018],{},"synth"," CDK code to generate CloudFormation templates. You can also run ",[30,26021,26022],{},"diff"," to see what changes would be applied during a stack update.",[11,26025,26026,26027,26030,26031,26034,26035,26038,26039,26042],{},"In Terraform you ",[30,26028,26029],{},"init"," to download all providers and modules. This is sort of like running ",[30,26032,26033],{},"npm install"," in CDK and Pulumi. You then run ",[30,26036,26037],{},"terraform plan"," to see the changes that would result. ",[30,26040,26041],{},"terraform apply"," does CRUD operations on your cloud resources.",[11,26044,26045,26046,26048,26049,26052],{},"In Pulumi you run ",[30,26047,10638],{}," to see what changes would be made to a stack. You can use the ",[30,26050,26051],{},"--diff"," flag to see the specifics of what would change.",[11,26054,26055],{},"To summarize:",[76,26057,26058,26061,26072,26078],{},[79,26059,26060],{},"In CDK you synth CloudFormation and use these templates to deploy stacks made up of constructs. An \"app\" can contain multiple stacks, and you can deploy one or more stacks in an app at a time",[79,26062,26063,26064,26066,26067,748],{},"In Terraform you plan a configuration made up of modules, and then run ",[30,26065,26041],{}," to build the configuration/stack (",[20,26068,26071],{"href":26069,"rel":26070},"https://discuss.hashicorp.com/t/what-is-a-terraform-stack/31985",[24],"discuss.hashicorp.com/t/what-is-a-terraform-stack/31985",[79,26073,26074,26075,26077],{},"Pulumi: You preview a Pulumi stack made up of components, and then run ",[30,26076,10032],{}," to build the resources",[79,26079,26080,26081],{},"To tear down a stack in all three tools, you run ",[30,26082,26083],{},"destroy",[56,26085,26087],{"id":26086},"infrastructure-as-code-library-repos","Infrastructure as Code library repos",[11,26089,26090],{},"Let's look at the three repos that I wrote for deploying the same type of 3-tier web application to AWS using ECS Fargate.",[76,26092,26093,26102,26111],{},[79,26094,26095,26096],{},"CDK: ",[20,26097,26099],{"href":25759,"rel":26098},[24],[30,26100,26101],{},"cdk-django",[79,26103,26104,26105],{},"Terraform: ",[20,26106,26108],{"href":25769,"rel":26107},[24],[30,26109,26110],{},"terraform-aws-django",[79,26112,26113,26114],{},"Pulumi: ",[20,26115,26117],{"href":25779,"rel":26116},[24],[30,26118,26119],{},"pulumi-aws-django",[736,26121,26123],{"id":26122},"language","Language",[11,26125,26126,187,26128,26130,26131,26133,26134,26136,26137,643],{},[30,26127,26101],{},[30,26129,26119],{}," are both written in TypeScript. ",[30,26132,26110],{}," is written in HCL, a domain specific language created by HashiCorp. The ",[30,26135,26101],{}," is published to both npm and PyPI, so you can use it in JavaScript, TypeScript and Python projects, other languages are supported as well, but you need to write your library in TypeScript so it can be transpiled to other languages using ",[20,26138,26141],{"href":26139,"rel":26140},"https://github.com/aws/jsii",[24],"jsii",[11,26143,26144,26145,26150],{},"My Pulumi library is written in TypeScript and is published to NPM. For now it can only be used in JavaScript and TypeScript projects. There is a way in Pulumi to write in any language and then publish to any other major language, but I haven't  done this yet. See ",[20,26146,26149],{"href":26147,"rel":26148},"https://github.com/pulumi/pulumi-component-provider-ts-boilerplate",[24],"this GitHub repo"," for more information on this.",[11,26152,26153,26154,26159],{},"The HCL is pretty simple when you get used to it. I find that I don't like adding lots of logic in Terraform code because it takes away from the readability of a module. There is a tool called ",[20,26155,26158],{"href":26156,"rel":26157},"https://developer.hashicorp.com/terraform/cdktf",[24],"CDKTF"," which allows you to write HCL Terraform in TypeScript, but I haven't used it yet.",[736,26161,26163],{"id":26162},"release-management-versioning-and-publishing","Release management, versioning and publishing",[11,26165,26166,187,26168,26170,26171,26174,26175,26177,26178,26181,26182,26184,26185,26188],{},[30,26167,26119],{},[30,26169,26110],{}," both use ",[30,26172,26173],{},"release-please"," for automatically generating a changelog file and bumping versions. ",[30,26176,26173],{}," is an open source tool from Google that they use to version their Terraform GCP modules. Whenever I push new commits to ",[30,26179,26180],{},"main",", a new PR is created that adds changes to the CHANGELOG.md file, bumps the version of the library in ",[30,26183,21209],{}," and adds a new git tag (e.g. ",[30,26186,26187],{},"v1.2.3",") based on commit messages.",[11,26190,26191,22301,26193,26200,26201,26204,26205,106,26208,26210],{},[30,26192,26101],{},[20,26194,26197],{"href":26195,"rel":26196},"https://github.com/projen/projen",[24],[30,26198,26199],{},"projen"," for maintaining the changelog and bumping versions and publishing to npm. It is popular among developers in the CDK community and is a really awesome tool since it basically uses one file (",[30,26202,26203],{},".projenrc.ts",") to configure your entire repo, including files like ",[30,26206,26207],{},"tsconfig.json",[30,26209,21209],{},", and even GitHub Action workflows. It has a lot of configuration options, but I'm using it in a pretty simple way. It generates a new release and items to the changelog when I manually trigger a GitHub Action.",[11,26212,26213,26214,26219],{},"These tools are both based on ",[20,26215,26218],{"href":26216,"rel":26217},"https://www.conventionalcommits.org/en/v1.0.0/",[24],"conventional commits"," to automatically update the Changelog file.",[11,26221,26222,26223,26225],{},"I'm still manually publishing my ",[30,26224,26119],{}," package from the CLI. I need to add a GitHub Action to do this for me. This and other backlog items are listed at the end of the article!",[736,26227,26229],{"id":26228},"makefile-examples-and-local-development","Makefile, examples and local development",[11,26231,26232],{},"Each repo has a Makefile that includes commands that I frequently use when developing new features or fixing bugs. Each repo has commands for the following:",[76,26234,26235,26241,26246,26251,26260,26267,26273,26278,26287],{},[79,26236,26237,26238,26240],{},"synthesizing CDK to CloudFormation / running ",[30,26239,26037],{}," / previewing pulumi up for both the base and app stacks",[79,26242,26243,26244],{},"creating/updating an ad hoc base stack called ",[30,26245,10715],{},[79,26247,26248,26249],{},"destroying resources in the ad-hoc base stack called ",[30,26250,10715],{},[79,26252,26253,26254,26257,26258],{},"creating an ad hoc app stack called ",[30,26255,26256],{},"alpha"," that uses resources from ",[30,26259,10715],{},[79,26261,26262,26263,26257,26265],{},"destroying an ad hoc app stack called ",[30,26264,26256],{},[30,26266,10715],{},[79,26268,26269,26270],{},"creating/updating a prod base stack called ",[30,26271,26272],{},"stage",[79,26274,26275,26276],{},"destroying resources in the prod base stack called ",[30,26277,26272],{},[79,26279,26280,26281,26283,26284,26286],{},"creating a prod app stack using called ",[30,26282,26272],{}," that uses resources from the ",[30,26285,26272],{}," base stack",[79,26288,26289,26290],{},"destroying resources in the prod app stack called ",[30,26291,26272],{},[11,26293,26294,26295,26297],{},"Here's an example of what these commands look like in ",[30,26296,26119],{}," for prod infrastructure base and app stacks:",[459,26299,26303],{"className":26300,"code":26301,"language":26302,"meta":464,"style":464},"language-make shiki shiki-themes github-light github-dark monokai","prod-base-preview:  build\n    pulumi -C examples/prod/base --stack stage --non-interactive preview\n\nprod-base-up:   build\n    pulumi -C examples/prod/base --stack stage --non-interactive up --yes\n\nprod-base-destroy:  build\n    pulumi -C examples/prod/base --stack stage --non-interactive destroy --yes\n\nprod-app-preview:   build\n    pulumi -C examples/prod/app --stack stage --non-interactive preview\n\nprod-app-preview-diff:  build\n    pulumi -C examples/prod/app --stack stage --non-interactive preview --diff\n\nprod-app-up:    build\n    pulumi -C examples/prod/app --stack stage --non-interactive up --yes\n\nprod-app-destroy:   build\n    pulumi -C examples/prod/app --stack stage --non-interactive destroy --yes\n","make",[30,26304,26305,26313,26318,26322,26330,26335,26339,26346,26351,26355,26362,26367,26371,26378,26383,26387,26395,26400,26404,26411],{"__ignoreMap":464},[151,26306,26307,26310],{"class":469,"line":470},[151,26308,26309],{"class":473},"prod-base-preview",[151,26311,26312],{"class":503},":  build\n",[151,26314,26315],{"class":469,"line":488},[151,26316,26317],{"class":503},"    pulumi -C examples/prod/base --stack stage --non-interactive preview\n",[151,26319,26320],{"class":469,"line":500},[151,26321,1090],{"emptyLinePlaceholder":609},[151,26323,26324,26327],{"class":469,"line":509},[151,26325,26326],{"class":473},"prod-base-up",[151,26328,26329],{"class":503},":   build\n",[151,26331,26332],{"class":469,"line":517},[151,26333,26334],{"class":503},"    pulumi -C examples/prod/base --stack stage --non-interactive up --yes\n",[151,26336,26337],{"class":469,"line":534},[151,26338,1090],{"emptyLinePlaceholder":609},[151,26340,26341,26344],{"class":469,"line":1413},[151,26342,26343],{"class":473},"prod-base-destroy",[151,26345,26312],{"class":503},[151,26347,26348],{"class":469,"line":1418},[151,26349,26350],{"class":503},"    pulumi -C examples/prod/base --stack stage --non-interactive destroy --yes\n",[151,26352,26353],{"class":469,"line":2462},[151,26354,1090],{"emptyLinePlaceholder":609},[151,26356,26357,26360],{"class":469,"line":2471},[151,26358,26359],{"class":473},"prod-app-preview",[151,26361,26329],{"class":503},[151,26363,26364],{"class":469,"line":2480},[151,26365,26366],{"class":503},"    pulumi -C examples/prod/app --stack stage --non-interactive preview\n",[151,26368,26369],{"class":469,"line":2489},[151,26370,1090],{"emptyLinePlaceholder":609},[151,26372,26373,26376],{"class":469,"line":2497},[151,26374,26375],{"class":473},"prod-app-preview-diff",[151,26377,26312],{"class":503},[151,26379,26380],{"class":469,"line":3140},[151,26381,26382],{"class":503},"    pulumi -C examples/prod/app --stack stage --non-interactive preview --diff\n",[151,26384,26385],{"class":469,"line":3149},[151,26386,1090],{"emptyLinePlaceholder":609},[151,26388,26389,26392],{"class":469,"line":3158},[151,26390,26391],{"class":473},"prod-app-up",[151,26393,26394],{"class":503},":    build\n",[151,26396,26397],{"class":469,"line":3167},[151,26398,26399],{"class":503},"    pulumi -C examples/prod/app --stack stage --non-interactive up --yes\n",[151,26401,26402],{"class":469,"line":3175},[151,26403,1090],{"emptyLinePlaceholder":609},[151,26405,26406,26409],{"class":469,"line":3184},[151,26407,26408],{"class":473},"prod-app-destroy",[151,26410,26329],{"class":503},[151,26412,26413],{"class":469,"line":3193},[151,26414,26415],{"class":503},"    pulumi -C examples/prod/app --stack stage --non-interactive destroy --yes\n",[11,26417,26418,26419,26421],{},"I currently don't have tests for all of these libraries, but for now the most effective way of testing that things are working correctly is to use the ",[30,26420,25986],{},"s to create environments and smoke check the environments to make sure everything works correctly.",[11,26423,26424],{},"Adding unit tests is another item for the backlog.",[736,26426,26428],{"id":26427},"ad-hoc-vs-prod","ad-hoc vs prod",[76,26430,26431,26439,26442,26445,26448,26451],{},[79,26432,26433,26438],{},[20,26434,26437],{"href":26435,"rel":26436},"https://briancaffey.github.io/2022/03/27/ad-hoc-developer-environments-for-django-with-aws-ecs-terraform-and-github-actions",[24],"the last article I wrote was about ad hoc environments",". Also known as \"on-demand\" environments or \"preview\" environments.",[79,26440,26441],{},"the motivation for using ad-hoc environments is speed and cost (you can stand up an environment in less time and you share the costs of the base environment, including VPC, ALB, RDS)",[79,26443,26444],{},"you can completely ignore \"ad-hoc\" environments and use the \"prod\" infrastructure for any number of environments (such as dev, QA, RC, stage and prod)",[79,26446,26447],{},"prod can be used for a production environment and any number of pre-production environments",[79,26449,26450],{},"multiple environments built with \"prod\" infrastructure can be configured with a \"knobs and dials\" (e.g., how big are app and DB instances, how many tasks to run in a service, etc.)",[79,26452,26453],{},"the \"prod\" infrastructure should be the same for the \"production\" environment and the \"staging\" environment",[736,26455,26457],{"id":26456},"directory-structure","Directory structure",[11,26459,26460],{},"The directory structures for each repo are all similar with some minor differences.",[11,26462,26463,26464,187,26467,26470,26471,187,26474,643],{},"There are two types of environments: ",[30,26465,26466],{},"ad-hoc",[30,26468,26469],{},"prod",". Within ad-hoc and production, there are two directories ",[30,26472,26473],{},"base",[30,26475,26476],{},"app",[11,26478,26479,26480,26483,26484,26486,26487,26489],{},"Each repo has a directory called ",[30,26481,26482],{},"internal"," which contain building blocks used by the ",[30,26485,25986],{},"s that are exposed. The contents of the ",[30,26488,26482],{}," directories are not intended to be used by anyone who is using the libraries.",[11,26491,26492],{},[15,26493,26494],{},"CDK construct library repo structure",[459,26496,26499],{"className":26497,"code":26498,"language":997},[995],"~/git/github/cdk-django$ tree -L 4 -d src/\nsrc/\n├── constructs\n│   ├── ad-hoc\n│   │   ├── app\n│   │   └── base\n│   ├── internal\n│   │   ├── alb\n│   │   ├── bastion\n│   │   ├── customResources\n│   │   │   └── highestPriorityRule\n│   │   ├── ecs\n│   │   │   ├── iam\n│   │   │   ├── management-command\n│   │   │   ├── redis\n│   │   │   ├── scheduler\n│   │   │   ├── web\n│   │   │   └── worker\n│   │   ├── rds\n│   │   ├── sg\n│   │   └── vpc\n│   └── prod\n│       ├── app\n│       └── base\n└── examples\n    └── ad-hoc\n        ├── app\n        │   └── config\n        └── base\n            └── config\n",[30,26500,26498],{"__ignoreMap":464},[11,26502,26503],{},[15,26504,26505],{},"Terraform module library repo structure",[459,26507,26510],{"className":26508,"code":26509,"language":997},[995],"~/git/github/terraform-aws-django$ tree -L 4 -d modules\nmodules\n├── ad-hoc\n│   ├── app\n│   └── base\n├── internal\n│   ├── alb\n│   ├── autoscaling\n│   ├── bastion\n│   ├── ecs\n│   │   ├── ad-hoc\n│   │   │   ├── celery_beat\n│   │   │   ├── celery_worker\n│   │   │   ├── cluster\n│   │   │   ├── management_command\n│   │   │   ├── redis\n│   │   │   └── web\n│   │   └── prod\n│   │       ├── celery_beat\n│   │       ├── celery_worker\n│   │       ├── cluster\n│   │       ├── management_command\n│   │       └── web\n│   ├── elasticache\n│   ├── iam\n│   ├── rds\n│   ├── route53\n│   ├── s3\n│   ├── sd\n│   └── sg\n└── prod\n    ├── app\n    └── base\n",[30,26511,26509],{"__ignoreMap":464},[11,26513,26514],{},[15,26515,26516],{},"Pulumi component library repo structure",[459,26518,26521],{"className":26519,"code":26520,"language":997},[995],"~/git/github/pulumi-aws-django$ tree -L 3 src/\nsrc/\n├── components\n│   ├── ad-hoc\n│   │   ├── README.md\n│   │   ├── app\n│   │   └── base\n│   └── internal\n│       ├── README.md\n│       ├── alb\n│       ├── bastion\n│       ├── cw\n│       ├── ecs\n│       ├── iam\n│       ├── rds\n│       └── sg\n└── util\n    ├── index.ts\n    └── taggable.ts\n",[30,26522,26520],{"__ignoreMap":464},[11,26524,26525],{},[15,26526,26527],{},"Pulumi examples directory",[459,26529,26532],{"className":26530,"code":26531,"language":997},[995],"~/git/github/pulumi-aws-django$ tree -L 3 examples/\nexamples/\n└── ad-hoc\n    ├── app\n    │   ├── Pulumi.alpha.yaml\n    │   ├── Pulumi.yaml\n    │   ├── index.ts\n    │   ├── node_modules\n    │   ├── package-lock.json\n    │   ├── package.json\n    │   └── tsconfig.json\n    └── base\n        ├── Pulumi.yaml\n        ├── bin\n        ├── index.ts\n        ├── package-lock.json\n        ├── package.json\n        └── tsconfig.json\n",[30,26533,26531],{"__ignoreMap":464},[736,26535,26537],{"id":26536},"cloc","CLOC",[11,26539,26540,26541,26543],{},"Let's use CLOC (count lines of code) to compare the lines of code used in the ",[30,26542,25986],{}," of CDK/CloudFormation/Terraform/Pulumi.",[11,26545,26546],{},[15,26547,26548],{},[30,26549,26101],{},[459,26551,26554],{"className":26552,"code":26553,"language":997},[995],"~/git/github/cdk-django$ cloc src/constructs/\n      14 text files.\n      14 unique files.\n       0 files ignored.\n\ngithub.com/AlDanial/cloc v 1.94  T=0.04 s (356.1 files/s, 30040.9 lines/s)\n-------------------------------------------------------------------------------\nLanguage                     files          blank        comment           code\n-------------------------------------------------------------------------------\nTypeScript                      13            155             59            908\nPython                           1             18              8             33\n-------------------------------------------------------------------------------\nSUM:                            14            173             67            941\n-------------------------------------------------------------------------------\n",[30,26555,26553],{"__ignoreMap":464},[11,26557,26558],{},[15,26559,26560],{},[30,26561,26110],{},[459,26563,26566],{"className":26564,"code":26565,"language":997},[995],"~/git/github/terraform-aws-django$ cloc modules/\n      68 text files.\n      58 unique files.\n      11 files ignored.\n\ngithub.com/AlDanial/cloc v 1.94  T=0.15 s (385.9 files/s, 20585.1 lines/s)\n-------------------------------------------------------------------------------\nLanguage                     files          blank        comment           code\n-------------------------------------------------------------------------------\nHCL                             55            472            205           2390\nMarkdown                         3              7              0             20\n-------------------------------------------------------------------------------\nSUM:                            58            479            205           2410\n-------------------------------------------------------------------------------\n",[30,26567,26565],{"__ignoreMap":464},[11,26569,26570],{},[15,26571,26572],{},[30,26573,26119],{},[459,26575,26578],{"className":26576,"code":26577,"language":997},[995],"~/git/github/pulumi-aws-django$ cloc src/components/\n      15 text files.\n      15 unique files.\n       0 files ignored.\n\ngithub.com/AlDanial/cloc v 1.94  T=0.11 s (134.5 files/s, 12924.2 lines/s)\n-------------------------------------------------------------------------------\nLanguage                     files          blank        comment           code\n-------------------------------------------------------------------------------\nTypeScript                      13            110            176           1119\nMarkdown                         2              6              0             30\n-------------------------------------------------------------------------------\nSUM:                            15            116            176           1149\n-------------------------------------------------------------------------------\n",[30,26579,26577],{"__ignoreMap":464},[56,26581,26583],{"id":26582},"communities","Communities",[11,26585,26586],{},"The CDK, Terraform and Pulumi communities are all great and a lot of people helped when I got stuck on issues writing these libraries. Thank you!",[76,26588,26589,26596,26603],{},[79,26590,26591],{},[20,26592,26595],{"href":26593,"rel":26594},"https://cdk.dev/",[24],"cdk.dev",[79,26597,26598],{},[20,26599,26602],{"href":26600,"rel":26601},"https://discuss.hashicorp.com/c/terraform-core/27",[24],"Terraform Section of HashiCorp Discuss Forum",[79,26604,26605],{},[20,26606,26609],{"href":26607,"rel":26608},"https://slack.pulumi.com/",[24],"Pulumi Slack",[56,26611,26612],{"id":26612},"μblog",[11,26614,26615],{},"μblog is a micro blogging application that I have written using Django and Vue.js. Here's a screenshot of the homepage:",[11,26617,26618],{},[2718,26619],{"alt":26620,"src":26621},"ublog","/static/ublog_screenshot.png",[11,26623,26624],{},"It is a pretty simple app. Users can write posts with text and an optional images. Logged in users can write posts and like posts.",[736,26626,26628],{"id":26627},"mono-repo-structure","Mono-repo structure",[11,26630,26631,26632,26635],{},"It lives in a GitHub mono repo called ",[30,26633,26634],{},"django-step-by-step",". This mono repo contains a few different things:",[76,26637,26638,26641,26644,26653],{},[79,26639,26640],{},"backend Django application",[79,26642,26643],{},"frontend Vue.js application",[79,26645,26646,26647,106,26649,187,26651],{},"IaC code that uses c/m/c from ",[30,26648,26101],{},[30,26650,26110],{},[30,26652,26119],{},[79,26654,26655],{},"GitHub Actions workflows for both Infrastructure deployments and application deployments",[11,26657,26658],{},"μblog is the reference application that I deploy to infrastructure created with CDK, Terraform and Pulumi. μblog is meant to represent a generic 12 Factor application that uses:",[76,26660,26661,26664,26667,26670,26673,26676,26679,26682],{},[79,26662,26663],{},"gunicorn for a backend API",[79,26665,26666],{},"Vue.js for a client that consumes the backend API",[79,26668,26669],{},"celery for async task processing",[79,26671,26672],{},"celery beat for scheduling tasks",[79,26674,26675],{},"Postgres for relational data",[79,26677,26678],{},"Redis for caching and message brokering",[79,26680,26681],{},"S3 for object storage",[79,26683,26684],{},"Django admin for a simple admin interface",[11,26686,26687],{},"There is a lot more I could add on μblog. For now I'll just mention that it:",[76,26689,26690,26693,26696,26699,26702],{},[79,26691,26692],{},"has a great local development environment (supports both docker-compose and virtual environments)",[79,26694,26695],{},"demonstrates how to use Django in different ways. It implements the same application using Function Based View and Class Based Views, and implements both a REST API (both with FBV and CBV) and GraphQL.",[79,26697,26698],{},"GitHub Actions for running unit tests",[79,26700,26701],{},"k6 for load testing",[79,26703,26704,26705],{},"contains a documentation site deployed to GitHub pages (made with VuePress) can be found here: ",[20,26706,25791],{"href":25791,"rel":26707},[24],[14063,26709,26711],{"id":26710},"infrastructure-deep-dive","Infrastructure Deep Dive",[11,26713,26714,26715,26717],{},"Let's go through each of the ",[30,26716,25986],{},"s used in the three libraries. I'll cover some of the organizational decisions, dependencies and differences between how things are done between CDK, Terraform and Pulumi.",[11,26719,26720,26721,187,26723,26725,26726,187,26728,26730],{},"I'll first talk about the two stacks used in ad hoc environments: ",[30,26722,26473],{},[30,26724,26476],{},". Then I'll talk about the prod environments which are also composed of ",[30,26727,26473],{},[30,26729,26476],{}," stacks.",[11,26732,26733,26734,26737],{},"Keep in mind that there aren't that many differences between the ad hoc environment base and app stacks and the prod environment app and base stacks. A future optimization could be to use a single base and app stack, but I think there is a trade-off between readability and DRYness of infrastructure code, ",[15,26735,26736],{},"especially with Terraform",". In general I try to use very little conditionals and logic with Terraform code. It is much easier to have dynamic configuration in CDK and Pulumi, and probably also for other tools like CDKTF (that I have not yet tried).",[56,26739,26741],{"id":26740},"splitting-up-the-stacks","Splitting up the stacks",[11,26743,26744],{},"While it is possible to put all resources in a single stack with both Terraform, CDK and Pulumi, it is not recommended to do so.",[76,26746,26747,26753,26761],{},[79,26748,26749,26750],{},"Terraform enables this with outputs and ",[30,26751,26752],{},"terraform_remote_state",[79,26754,26755,26756],{},"Pulumi encourages the use of ",[20,26757,26760],{"href":26758,"rel":26759},"https://www.pulumi.com/docs/guides/organizing-projects-stacks/",[24],"micro stacks",[79,26762,26763,26764],{},"CDK has an article on how to ",[20,26765,26768],{"href":26766,"rel":26767},"https://docs.aws.amazon.com/cdk/v2/guide/stack_how_to_create_multiple_stacks.html",[24],"create an app with multiple stacks",[11,26770,26771],{},"My design decision was to keep things limited to 2 stacks. Later on it would be interesting to try splitting out another stack.",[11,26773,26774],{},"Also, on-demand environments really lends itself to stacks that are split up.",[11,26776,26777,26778,26783],{},"In the section ",[20,26779,26782],{"href":26780,"rel":26781},"https://docs.aws.amazon.com/cdk/v2/guide/resources.html",[24],"\"Passing unique identifiers\"",", the CDK recommends that we keep the two stacks in the same app. In Terraform and Pulumi, each stack environment is in its own app.",[11,26785,26786,26787,26789,26790,26792,26793,26795,26796,187,26799,26802,26803,26805,26806,106,26809,187,26812,26815],{},"There is a balance to be found between single stacks vs micro stacks. Both the base and app ",[30,26788,25986],{},"s could be split out further. For example, the ",[30,26791,26473],{}," ",[30,26794,25986],{},"s could be split into ",[30,26797,26798],{},"networking",[30,26800,26801],{},"rds",". The ",[30,26804,26476],{}," stack could be split into different ECS services so that their infrastructure can be deployed independently, like ",[30,26807,26808],{},"cluster",[30,26810,26811],{},"backend",[30,26813,26814],{},"frontend",". The more resources that a stack has, the longer it takes to deploy and the more risky it gets, but adding lots of stacks can add to mental overhead, and pipeline complexity. Each tool has ways of dealing with these complexities (CDK Pipelines, Terragrunt, Pulumi Automation API), but I won't be getting into any of these options in this article. I would like to try these out and share in a future article.",[11,26817,26818],{},"My rules of thumbs are:",[76,26820,26821,26824],{},[79,26822,26823],{},"single stacks are bad because you don't want to put all your eggs in one basket, however your IaC tool should give you confidence about what is going to change when you try to make a change",[79,26825,26826],{},"Lots of small stacks can cause overhead and make things more complex than they need to be",[56,26828,26830],{"id":26829},"ad-hoc-base-overview","Ad hoc base overview",[11,26832,26833],{},"Here's an overview of the resources used in an ad hoc base environment.",[76,26835,26836,26839,26842,26845,26848,26851,26854,26857],{},[79,26837,26838],{},"(Inputs)",[79,26840,26841],{},"(Optional environment configs)",[79,26843,26844],{},"VPC and Service Discovery",[79,26846,26847],{},"S3",[79,26849,26850],{},"Security Groups",[79,26852,26853],{},"Load Balancer",[79,26855,26856],{},"RDS",[79,26858,26859],{},"Bastion Host",[736,26861,26863],{"id":26862},"visualization","Visualization",[11,26865,26866,26867,26870],{},"Here's a dependency graph showing all of the resources in ad hoc base stack. This can be found on the ",[30,26868,26869],{},"Resources"," tab of the ad hoc base stack in the Pulumi console.",[11,26872,26873],{},[2718,26874],{"alt":26875,"src":26876},"Graph view of ad hoc base infrastructure","/static/pulumi_ad_hoc_base_dep_graph.png",[736,26878,26880],{"id":26879},"inputs","Inputs",[11,26882,26883],{},"There are only two required inputs for the ad hoc base stack",[76,26885,26886,26889],{},[79,26887,26888],{},"ACM certificate ARN",[79,26890,26891],{},"Domain Name",[11,26893,26894],{},"I store these values in environment variables for the pipelines in CDK, Terraform and Pulumi. When running pipelines from my local environment, they are exported in my shell before running deploy/apply/up or synth/plan/preview.",[736,26896,26898],{"id":26897},"vpc","VPC",[11,26900,26901,26902,26904],{},"The VPC is the first resource that is created as part of the ",[30,26903,26473],{}," stack. There official, high-level constructs in each IaC tool for building VPCs and all related networking resources.",[76,26906,26907,26913,26919],{},[79,26908,26909,26912],{},[30,26910,26911],{},"awsx"," has a VPC module",[79,26914,26915,26918],{},[30,26916,26917],{},"terraform-aws-vpc"," module",[79,26920,26921],{},"L2 VPC Construct in CDK",[11,26923,26924,26925,26928,26929,26932],{},"The setting in the Terraform VPC module ",[30,26926,26927],{},"one_nat_gateway_per_az = false"," doesn't seem to exist on the ",[30,26930,26931],{},"awsx.ec2.Vpc"," module. This will add to cost savings since it will use 1 NAT Gateway instead of 2 or 3.",[736,26934,26850],{"id":26935},"security-groups",[11,26937,26938],{},"Pulumi and Terraform can be used in a similar way to define security groups. CDK has a much more concise option for defining ingress and egress rules for security groups.",[459,26940,26944],{"className":26941,"code":26942,"language":26943,"meta":464,"style":464},"language-ts shiki shiki-themes github-light github-dark monokai","    const albSecurityGroup = new SecurityGroup(scope, 'AlbSecurityGroup', {\n      vpc: props.vpc,\n    });\n\n    albSecurityGroup.addIngressRule(Peer.anyIpv4(), Port.tcp(443), 'HTTPS');\n    albSecurityGroup.addIngressRule(Peer.anyIpv4(), Port.tcp(80), 'HTTP');\n","ts",[30,26945,26946,26969,26974,26979,26983,27015],{"__ignoreMap":464},[151,26947,26948,26950,26953,26955,26957,26960,26963,26966],{"class":469,"line":470},[151,26949,19860],{"class":12347},[151,26951,26952],{"class":12360}," albSecurityGroup",[151,26954,19865],{"class":1869},[151,26956,4236],{"class":1869},[151,26958,26959],{"class":473}," SecurityGroup",[151,26961,26962],{"class":503},"(scope, ",[151,26964,26965],{"class":481},"'AlbSecurityGroup'",[151,26967,26968],{"class":503},", {\n",[151,26970,26971],{"class":469,"line":488},[151,26972,26973],{"class":503},"      vpc: props.vpc,\n",[151,26975,26976],{"class":469,"line":500},[151,26977,26978],{"class":503},"    });\n",[151,26980,26981],{"class":469,"line":509},[151,26982,1090],{"emptyLinePlaceholder":609},[151,26984,26985,26988,26991,26994,26997,27000,27003,27005,27008,27010,27013],{"class":469,"line":517},[151,26986,26987],{"class":503},"    albSecurityGroup.",[151,26989,26990],{"class":473},"addIngressRule",[151,26992,26993],{"class":503},"(Peer.",[151,26995,26996],{"class":473},"anyIpv4",[151,26998,26999],{"class":503},"(), Port.",[151,27001,27002],{"class":473},"tcp",[151,27004,12386],{"class":503},[151,27006,27007],{"class":477},"443",[151,27009,24817],{"class":503},[151,27011,27012],{"class":481},"'HTTPS'",[151,27014,20129],{"class":503},[151,27016,27017,27019,27021,27023,27025,27027,27029,27031,27034,27036,27039],{"class":469,"line":534},[151,27018,26987],{"class":503},[151,27020,26990],{"class":473},[151,27022,26993],{"class":503},[151,27024,26996],{"class":473},[151,27026,26999],{"class":503},[151,27028,27002],{"class":473},[151,27030,12386],{"class":503},[151,27032,27033],{"class":477},"80",[151,27035,24817],{"class":503},[151,27037,27038],{"class":481},"'HTTP'",[151,27040,20129],{"class":503},[736,27042,27044],{"id":27043},"load-balancer-resources","Load Balancer Resources",[11,27046,27047],{},"There's not much to comment on here. In each library I have resource group that defines the following:",[76,27049,27050,27053,27056,27059],{},[79,27051,27052],{},"Application Load Balancer",[79,27054,27055],{},"A default target group",[79,27057,27058],{},"An HTTP listener that redirects to HTTPS",[79,27060,27061],{},"An HTTPS listener with a default \"fixed-response\" action",[11,27063,27064],{},"Properties from these resources are used in the \"app\" stack to build listener rules for ECS services that are configured with load balancers, such as the backend and frontend web services.",[11,27066,27067],{},"Ad hoc app environments all share a common load balancer from the base stack that they use.",[736,27069,27071],{"id":27070},"rds-resources","RDS Resources",[11,27073,27074,27075,27077],{},"All three libraries have the RDS security group and Subnet Group in the same ",[30,27076,25986],{}," as the RDS instance. The SG and DB Subnet group could alternatively be grouped closer to the other network resources.",[11,27079,27080],{},"Currently the RDS resources are part of the \"base\" stack for each library. A future optimization may be to break the RDS instance out of the \"base\" stack and put it in its own stack. The \"RDS\" stack would be dependent on the \"base\" stack, and then \"app\" stack would then be dependent on both the \"base\" stack and the \"RDS\" stack. More stacks isn't necessarily a bad thing, but for my initial implementation of these libraries I have decided to keep the \"micro stacks\" approach limited to only 2 stacks for an environment.",[11,27082,27083],{},"The way that database secrets are handled is another difference between CDK and Terraform and Pulumi. I am currently \"hardcoding\" the RDS password for Terraform and Pulumi, and in CDK I am using a Secrets Manager Secret for the database credential.",[459,27085,27087],{"className":26941,"code":27086,"language":26943,"meta":464,"style":464},"    const secret = new Secret(scope, 'dbSecret', {\n      secretName: props.dbSecretName,\n      description: 'secret for rds',\n      generateSecretString: {\n        secretStringTemplate: JSON.stringify({ username: 'postgres' }),\n        generateStringKey: 'password',\n        excludePunctuation: true,\n        includeSpace: false,\n      },\n    });\n",[30,27088,27089,27109,27114,27124,27129,27151,27161,27170,27179,27184],{"__ignoreMap":464},[151,27090,27091,27093,27095,27097,27099,27102,27104,27107],{"class":469,"line":470},[151,27092,19860],{"class":12347},[151,27094,4673],{"class":12360},[151,27096,19865],{"class":1869},[151,27098,4236],{"class":1869},[151,27100,27101],{"class":473}," Secret",[151,27103,26962],{"class":503},[151,27105,27106],{"class":481},"'dbSecret'",[151,27108,26968],{"class":503},[151,27110,27111],{"class":469,"line":488},[151,27112,27113],{"class":503},"      secretName: props.dbSecretName,\n",[151,27115,27116,27119,27122],{"class":469,"line":500},[151,27117,27118],{"class":503},"      description: ",[151,27120,27121],{"class":481},"'secret for rds'",[151,27123,9417],{"class":503},[151,27125,27126],{"class":469,"line":509},[151,27127,27128],{"class":503},"      generateSecretString: {\n",[151,27130,27131,27134,27137,27139,27142,27145,27148],{"class":469,"line":517},[151,27132,27133],{"class":503},"        secretStringTemplate: ",[151,27135,27136],{"class":12360},"JSON",[151,27138,643],{"class":503},[151,27140,27141],{"class":473},"stringify",[151,27143,27144],{"class":503},"({ username: ",[151,27146,27147],{"class":481},"'postgres'",[151,27149,27150],{"class":503}," }),\n",[151,27152,27153,27156,27159],{"class":469,"line":534},[151,27154,27155],{"class":503},"        generateStringKey: ",[151,27157,27158],{"class":481},"'password'",[151,27160,9417],{"class":503},[151,27162,27163,27166,27168],{"class":469,"line":1413},[151,27164,27165],{"class":503},"        excludePunctuation: ",[151,27167,19726],{"class":477},[151,27169,9417],{"class":503},[151,27171,27172,27175,27177],{"class":469,"line":1418},[151,27173,27174],{"class":503},"        includeSpace: ",[151,27176,9522],{"class":477},[151,27178,9417],{"class":503},[151,27180,27181],{"class":469,"line":2462},[151,27182,27183],{"class":503},"      },\n",[151,27185,27186],{"class":469,"line":2471},[151,27187,26978],{"class":503},[11,27189,27190,27191,27194],{},"In the ",[30,27192,27193],{},"DatabaseInstance"," props we can then use this secret like so:",[459,27196,27199],{"className":27197,"code":27198,"language":997},[995],"    credentials: Credentials.fromSecret(secret),\n",[30,27200,27198],{"__ignoreMap":464},[11,27202,27203,27204,27207],{},"In the application deployed with CDK, I use a Django settings module package that uses a package called ",[30,27205,27206],{},"aws_secretsmanager_caching"," to get and cache the secrets manager secret for the database, whereas in the apps deployed with Terraform and Pulumi I read in the password from an environment variable.",[11,27209,27210,27211,27214,27215,187,27222,27229],{},"The Terraform and Pulumi database instance arguments simply accept a ",[30,27212,27213],{},"password"," field. This will be another item for the backlog for Terraform and Pulumi. The ",[20,27216,27219],{"href":27217,"rel":27218},"https://www.pulumi.com/registry/packages/random/api-docs/randompassword/",[24],[30,27220,27221],{},"randompassword",[20,27223,27226],{"href":27224,"rel":27225},"https://www.pulumi.com/registry/packages/aws/api-docs/secretsmanager/secretversion/",[24],[30,27227,27228],{},"secretversion"," components can be used to do this.",[736,27231,26859],{"id":27232},"bastion-host",[11,27234,27235],{},"There are two main use cases for the bastion host in ad-hoc environments.",[76,27237,27238,27245],{},[79,27239,27240,27241,27244],{},"When creating a new ad hoc app environment, the bastion host is used to create a new database called ",[30,27242,27243],{},"{ad-hoc-env-name}-db"," that the new ad hoc environment will use. (There might be another way of doing this, but using a bastion host is working well for now).",[79,27246,27247,27248,27251],{},"If you using a database management tool on you local machine like DBeaver, the bastion host can help you connect to the RDS instance in a private subnet. The bastion host instance is configured to run a service that forwards traffic on port 5432 to the RDS instance. If you port forward from your local machine to the bastion host on port 5432, you can connect RDS by simple connecting to ",[30,27249,27250],{},"localhost:5432"," on your local machine.",[11,27253,27254],{},"You don't need to manage SSH keys since you connect to the instance in a private subnet using SSM:",[459,27256,27258],{"className":461,"code":27257,"language":463,"meta":464,"style":464},"aws ssm start-session --target $INSTANCE_ID\n",[30,27259,27260],{"__ignoreMap":464},[151,27261,27262,27265,27268,27271,27274],{"class":469,"line":470},[151,27263,27264],{"class":473},"aws",[151,27266,27267],{"class":481}," ssm",[151,27269,27270],{"class":481}," start-session",[151,27272,27273],{"class":477}," --target",[151,27275,27276],{"class":503}," $INSTANCE_ID\n",[736,27278,27280],{"id":27279},"outputs","Outputs",[11,27282,27283],{},"Here are the outputs for the ad hoc base stack used in Terraform and Pulumi:",[76,27285,27286,27289,27292,27295,27298,27301,27304,27307,27310,27313],{},[79,27287,27288],{},"vpc_id",[79,27290,27291],{},"assets_bucket_name",[79,27293,27294],{},"private_subnet_ids",[79,27296,27297],{},"app_sg_id",[79,27299,27300],{},"alb_sg_id",[79,27302,27303],{},"listener_arn",[79,27305,27306],{},"alb_dns_name",[79,27308,27309],{},"task_role_arn",[79,27311,27312],{},"execution_role_arn",[79,27314,27315],{},"rds_address",[11,27317,27318,27319,187,27322,27325,27326,643],{},"In CDK, the stack references in the app stack don't reference the unique identifiers from the base stack (such as the VPC id or bastion host instance id), but instead they reference the properties of the stack which have types like ",[30,27320,27321],{},"Vpc",[30,27323,27324],{},"RdsInstance",". More on this later in the following section ",[15,27327,27328],{},"Passing data between stacks",[56,27330,27332],{"id":27331},"ad-hoc-app-overview","Ad hoc app overview",[11,27334,27335],{},"The ad hoc app is an group of resources that powers an on-demand environment that is meant to be short lived for testing, QA, validation, demos, etc.",[11,27337,27338],{},"This visualization shows all of the resources in the ad hoc app stack. It also comes from the Pulumi console.",[11,27340,27341],{},[2718,27342],{"alt":27343,"src":27344},"Graph view of ad hoc app infrastructure","/static/pulumi_ad_hoc_app_dep_graph.png",[736,27346,27348],{"id":27347},"ecs-cluster","ECS Cluster",[76,27350,27351,27354],{},[79,27352,27353],{},"This is a small component that defines both ECS Cluster and the default capacity providers",[79,27355,27356,27357,27360,27361,27363],{},"It defaults to not using ",[30,27358,27359],{},"FARGATE_SPOT","; ad hoc environments do use ",[30,27362,27359],{}," for cost savings",[210,27365,27366],{},[11,27367,27368,27369,748],{},"NOTE: defaultCapacityProviderStrategy on cluster not currently supported. (",[20,27370,19750],{"href":27371,"rel":27372},"https://docs.aws.amazon.com/cdk/api/v1/docs/@aws-cdk_aws-ecs.CapacityProviderStrategy.html",[24],[736,27374,27376],{"id":27375},"shared-environment-variables","Shared environment variables",[11,27378,27379,27380,27382],{},"The backend containers should all have the same environment variables, so I define them once in the app stack and pass these into the service resource ",[30,27381,25986],{},"s.",[76,27384,27385,27399,27405,27411,27417],{},[79,27386,27387,27388,27391,27392,27395,27396,643],{},"I struggled to get this right in pulumi. A lot of Pulumi examples used ",[30,27389,27390],{},"JSON.stringify"," for containerDefinitions in task definitions. I was able to get help from the Pulumi Slack channel; someone recommended that I use ",[30,27393,27394],{},"pulumi.jsonStringify"," which was added in a relatively recent version of ",[30,27397,27398],{},"pulumi/pulumi",[79,27400,27401,27402],{},"CDK allows you to declare environment variables for a containerDefinition like ",[30,27403,27404],{},"{ FOO: \"bar\" }",[79,27406,27407,27408],{},"Pulumi and Terraform require that values are passed like ",[30,27409,27410],{},"{ name: \"FOO\", value: \"bar\"}",[79,27412,27413,27414,27416],{},"You could transform ",[30,27415,27404],{}," into the name/value format, but I didn't bother to do this",[79,27418,27419,27420,27423],{},"extra env vars in Terraform to allow for dynamically passing extra environment variables, and I used the ",[30,27421,27422],{},"concat"," function to add these to the list of default environment variables.",[11,27425,27426],{},"Here's what the code looks like for joining extra environment variables to the default environment variables:",[459,27428,27430],{"className":26941,"code":27429,"language":26943,"meta":464,"style":464},"    // CDK\n    if (extraEnvVars) {\n      environmentVariables = { ...extraEnvVars, ...environmentVariables };\n    }\n",[30,27431,27432,27437,27444,27464],{"__ignoreMap":464},[151,27433,27434],{"class":469,"line":470},[151,27435,27436],{"class":1527},"    // CDK\n",[151,27438,27439,27441],{"class":469,"line":488},[151,27440,23327],{"class":1869},[151,27442,27443],{"class":503}," (extraEnvVars) {\n",[151,27445,27446,27449,27451,27453,27456,27459,27461],{"class":469,"line":500},[151,27447,27448],{"class":503},"      environmentVariables ",[151,27450,1876],{"class":1869},[151,27452,12351],{"class":503},[151,27454,27455],{"class":1869},"...",[151,27457,27458],{"class":503},"extraEnvVars, ",[151,27460,27455],{"class":1869},[151,27462,27463],{"class":503},"environmentVariables };\n",[151,27465,27466],{"class":469,"line":509},[151,27467,9461],{"class":503},[459,27469,27472],{"className":27470,"code":27471,"language":997},[995],"    # terraform\n    env_vars = concat(local.env_vars, var.extra_env_vars)\n",[30,27473,27471],{"__ignoreMap":464},[459,27475,27478],{"className":27476,"code":27477,"language":997},[995],"    // Pulumi\n    if (extraEnvVars) {\n      envVars = envVars.apply(x => x.concat(extraEnvVars!))\n    }\n",[30,27479,27477],{"__ignoreMap":464},[736,27481,27483],{"id":27482},"route53-record","Route53 Record",[11,27485,27486],{},"This is pretty straightforward in each library. Each ad hoc environment gets a Route 53 record, and listener rules for the web services (Django and Vue.js SPA) match on a combination of the host header and path patterns.",[11,27488,27489,27490,27493,27494,27497],{},"This part is pretty opinionated in that it assumes you want to host the frontend and backend services on the same URL. For example, requests matching ",[30,27491,27492],{},"example.com/api/*"," are routed to the backend API and all other requests matching ",[30,27495,27496],{},"example.com/*"," are routed to the frontend service.",[736,27499,27501],{"id":27500},"redis","Redis",[11,27503,27504],{},"I go into more depth about why I run a Redis instance in an ECS service in my other article. This is only for the ad hoc environments. Production environments are configured with ElastiCache running Redis.",[11,27506,27507],{},"I decided to not make this service use any persistent storage. It may be a good idea to not use FARGATE_SPOT for this service, since restarts to the redis service could cause issues in ad hoc environments. For example, you may get a lot of celery errors in ad hoc environments if redis is not reachable.",[736,27509,27511],{"id":27510},"web-service","Web Service",[11,27513,27514,27515,27517,27518,27521,27522,187,27525,27528,27529,13576],{},"The web service is what defines the main Django application as well as the frontend website (JavaScript SPA or SSR site). I designed the Web Service resources group to be able to support both traditional Django apps (powered by templates), or for Django apps that service only a limited number of endpoints. This ",[30,27516,25986],{}," has an input parameter called ",[30,27519,27520],{},"pathPatterns"," which determines which paths it serves. For example, the API container may serve traffic for ",[30,27523,27524],{},"/api/*",[30,27526,27527],{},"/admin/*"," only, or it may want to serve all traffic (",[30,27530,27531],{},"/*",[11,27533,27534],{},"The way I use these components in ad hoc and prod environments is heavily opinionated in that:",[76,27536,27537],{},[79,27538,27539,27540,27542,27543,106,27545,106,27547,27550],{},"it assumes that the frontend SPA/SSR site should have a lower priority rule than the backend service and should route request paths matching ",[30,27541,27531],{},", while the backend service routes requests for a specific list of path patterns (",[30,27544,27524],{},[30,27546,27527],{},[30,27548,27549],{},"/graphql/*",", etc.).",[11,27552,27553],{},"You may want Django to handle most of your routes and 404 pages, in which case you would want the SPA to only handle requests matching certain paths. This would require some more consideration and careful refactoring.",[736,27555,27557],{"id":27556},"celery","Celery",[76,27559,27560,27563],{},[79,27561,27562],{},"The reason for having a celery service is to be able to have potentially multiple workers that scale independently",[79,27564,27565],{},"I use the same Pulumi component for both works and schedulers",[11,27567,27568,27569,27572,27573,643],{},"The terminology for this resource group could be better. Celery is one of many options for running async task workers, so it should probably be called something like ",[30,27570,27571],{},"AsyncWorker"," across the board rather than using the term ",[30,27574,27556],{},[736,27576,27578],{"id":27577},"management-command","Management Command",[76,27580,27581,27590],{},[79,27582,27583,27584,187,27587],{},"Defines a task that can be used to run commands like ",[30,27585,27586],{},"collectstatic",[30,27588,27589],{},"migrate",[79,27591,27592,27593,27595],{},"These tasks are ran both after the initial ",[30,27594,26476],{}," stack deployment and before rolling application upgrades",[11,27597,27598,27599,187,27601,27603],{},"In my Django app I have a single management command that calls ",[30,27600,27589],{},[30,27602,27586],{}," and runs them in the same process one after another. This management command could also be used for clearing caches during updates, loading fixtures, etc.",[11,27605,27606,27607,27609],{},"One other thing to note about this ",[30,27608,25986],{}," is that it outputs a complete script that can be used in GitHub Actions (or on your CLI when testing locally) that does the following:",[76,27611,27612,27619,27622,27625,27630],{},[79,27613,27614,27615,27618],{},"saves the ",[30,27616,27617],{},"START"," timestamp",[79,27620,27621],{},"runs the task with the required settings",[79,27623,27624],{},"waits for the task to complete",[79,27626,27614,27627,27618],{},[30,27628,27629],{},"END",[79,27631,27632,27633,187,27635,27637,27638],{},"collects the logs for the task between ",[30,27634,27617],{},[30,27636,27629],{}," and prints them to ",[30,27639,27640],{},"stdout",[11,27642,27643],{},"Here's an example of what the script looks like in Pulumi:",[459,27645,27647],{"className":26941,"code":27646,"language":26943,"meta":464,"style":464},"    const executionScript = pulumi.interpolate`#!/bin/bash\nSTART_TIME=$(date +%s000)\nTASK_ID=$(aws ecs run-task --cluster ${props.ecsClusterId} --task-definition ${taskDefinition.arn} --launch-type FARGATE --network-configuration \"awsvpcConfiguration={subnets=[${props.privateSubnetIds.apply(x => x.join(\",\"))}],securityGroups=[${props.appSgId}],assignPublicIp=ENABLED}\" | jq -r '.tasks[0].taskArn')\naws ecs wait tasks-stopped --tasks $TASK_ID --cluster ${props.ecsClusterId}\nEND_TIME=$(date +%s000)\naws logs get-log-events --log-group-name ${cwLoggingResources.cwLogGroupName} --log-stream-name ${props.name}/${props.name}/\\${TASK_ID##*/} --start-time $START_TIME --end-time $END_TIME | jq -r '.events[].message'\n`;\n    this.executionScript = executionScript;\n",[30,27648,27649,27667,27672,27762,27777,27782,27832,27838],{"__ignoreMap":464},[151,27650,27651,27653,27656,27658,27661,27664],{"class":469,"line":470},[151,27652,19860],{"class":12347},[151,27654,27655],{"class":12360}," executionScript",[151,27657,19865],{"class":1869},[151,27659,27660],{"class":503}," pulumi.",[151,27662,27663],{"class":473},"interpolate",[151,27665,27666],{"class":481},"`#!/bin/bash\n",[151,27668,27669],{"class":469,"line":488},[151,27670,27671],{"class":481},"START_TIME=$(date +%s000)\n",[151,27673,27674,27677,27679,27682,27684,27687,27689,27692,27694,27697,27699,27702,27704,27707,27709,27711,27713,27716,27718,27720,27722,27725,27727,27730,27732,27735,27737,27740,27743,27745,27748,27750,27752,27754,27757,27759],{"class":469,"line":500},[151,27675,27676],{"class":481},"TASK_ID=$(aws ecs run-task --cluster ",[151,27678,19871],{"class":19870},[151,27680,27681],{"class":503},"props",[151,27683,643],{"class":4828},[151,27685,27686],{"class":503},"ecsClusterId",[151,27688,2001],{"class":19870},[151,27690,27691],{"class":481}," --task-definition ",[151,27693,19871],{"class":19870},[151,27695,27696],{"class":503},"taskDefinition",[151,27698,643],{"class":4828},[151,27700,27701],{"class":503},"arn",[151,27703,2001],{"class":19870},[151,27705,27706],{"class":481}," --launch-type FARGATE --network-configuration \"awsvpcConfiguration={subnets=[",[151,27708,19871],{"class":19870},[151,27710,27681],{"class":503},[151,27712,643],{"class":4828},[151,27714,27715],{"class":503},"privateSubnetIds",[151,27717,643],{"class":4828},[151,27719,25994],{"class":473},[151,27721,12386],{"class":4828},[151,27723,11126],{"class":27724},"sdpu8",[151,27726,20832],{"class":12347},[151,27728,27729],{"class":503}," x",[151,27731,643],{"class":4828},[151,27733,27734],{"class":473},"join",[151,27736,12386],{"class":4828},[151,27738,27739],{"class":481},"\",\"",[151,27741,27742],{"class":4828},"))",[151,27744,2001],{"class":19870},[151,27746,27747],{"class":481},"],securityGroups=[",[151,27749,19871],{"class":19870},[151,27751,27681],{"class":503},[151,27753,643],{"class":4828},[151,27755,27756],{"class":503},"appSgId",[151,27758,2001],{"class":19870},[151,27760,27761],{"class":481},"],assignPublicIp=ENABLED}\" | jq -r '.tasks[0].taskArn')\n",[151,27763,27764,27767,27769,27771,27773,27775],{"class":469,"line":509},[151,27765,27766],{"class":481},"aws ecs wait tasks-stopped --tasks $TASK_ID --cluster ",[151,27768,19871],{"class":19870},[151,27770,27681],{"class":503},[151,27772,643],{"class":4828},[151,27774,27686],{"class":503},[151,27776,6274],{"class":19870},[151,27778,27779],{"class":469,"line":517},[151,27780,27781],{"class":481},"END_TIME=$(date +%s000)\n",[151,27783,27784,27787,27789,27792,27794,27797,27799,27802,27804,27806,27808,27810,27812,27814,27816,27818,27820,27822,27824,27826,27829],{"class":469,"line":534},[151,27785,27786],{"class":481},"aws logs get-log-events --log-group-name ",[151,27788,19871],{"class":19870},[151,27790,27791],{"class":503},"cwLoggingResources",[151,27793,643],{"class":4828},[151,27795,27796],{"class":503},"cwLogGroupName",[151,27798,2001],{"class":19870},[151,27800,27801],{"class":481}," --log-stream-name ",[151,27803,19871],{"class":19870},[151,27805,27681],{"class":503},[151,27807,643],{"class":4828},[151,27809,20415],{"class":503},[151,27811,2001],{"class":19870},[151,27813,19883],{"class":481},[151,27815,19871],{"class":19870},[151,27817,27681],{"class":503},[151,27819,643],{"class":4828},[151,27821,20415],{"class":503},[151,27823,2001],{"class":19870},[151,27825,19883],{"class":481},[151,27827,27828],{"class":477},"\\$",[151,27830,27831],{"class":481},"{TASK_ID##*/} --start-time $START_TIME --end-time $END_TIME | jq -r '.events[].message'\n",[151,27833,27834,27836],{"class":469,"line":1413},[151,27835,2798],{"class":481},[151,27837,20086],{"class":503},[151,27839,27840,27843,27846,27848],{"class":469,"line":1418},[151,27841,27842],{"class":15289},"    this",[151,27844,27845],{"class":503},".executionScript ",[151,27847,1876],{"class":1869},[151,27849,27850],{"class":503}," executionScript;\n",[11,27852,27853],{},"In GitHub Actions we get this command as a stack output, save it to a file, make it executable and then run it. This is what it looks like with CDK as a CloudFormation stack output:",[459,27855,27857],{"className":14359,"code":27856,"language":14361,"meta":464,"style":464},"      - name: \"Run backend update command\"\n        id: run_backend_update\n        run: |\n          # get the script from the stack output with an output key that contains the string `backendUpdate`\n          BACKEND_UPDATE_SCRIPT=$(aws cloudformation describe-stacks \\\n            --stack-name $AD_HOC_APP_NAME \\\n            | jq -r '.Stacks[0].Outputs[]|select(.OutputKey | contains(\"backendUpdate\")) | .OutputValue' \\\n          )\n\n          echo \"$BACKEND_UPDATE_SCRIPT\" > backend_update_command.sh\n          cat backend_update_command.sh\n          sudo chmod +x backend_update_command.sh\n          ./backend_update_command.sh\n",[30,27858,27859,27870,27880,27889,27894,27899,27904,27909,27914,27918,27923,27928,27933],{"__ignoreMap":464},[151,27860,27861,27863,27865,27867],{"class":469,"line":470},[151,27862,14459],{"class":503},[151,27864,20415],{"class":14368},[151,27866,6208],{"class":503},[151,27868,27869],{"class":481},"\"Run backend update command\"\n",[151,27871,27872,27875,27877],{"class":469,"line":488},[151,27873,27874],{"class":14368},"        id",[151,27876,6208],{"class":503},[151,27878,27879],{"class":481},"run_backend_update\n",[151,27881,27882,27885,27887],{"class":469,"line":500},[151,27883,27884],{"class":14368},"        run",[151,27886,6208],{"class":503},[151,27888,20607],{"class":1869},[151,27890,27891],{"class":469,"line":509},[151,27892,27893],{"class":481},"          # get the script from the stack output with an output key that contains the string `backendUpdate`\n",[151,27895,27896],{"class":469,"line":517},[151,27897,27898],{"class":481},"          BACKEND_UPDATE_SCRIPT=$(aws cloudformation describe-stacks \\\n",[151,27900,27901],{"class":469,"line":534},[151,27902,27903],{"class":481},"            --stack-name $AD_HOC_APP_NAME \\\n",[151,27905,27906],{"class":469,"line":1413},[151,27907,27908],{"class":481},"            | jq -r '.Stacks[0].Outputs[]|select(.OutputKey | contains(\"backendUpdate\")) | .OutputValue' \\\n",[151,27910,27911],{"class":469,"line":1418},[151,27912,27913],{"class":481},"          )\n",[151,27915,27916],{"class":469,"line":2462},[151,27917,1090],{"emptyLinePlaceholder":609},[151,27919,27920],{"class":469,"line":2471},[151,27921,27922],{"class":481},"          echo \"$BACKEND_UPDATE_SCRIPT\" > backend_update_command.sh\n",[151,27924,27925],{"class":469,"line":2480},[151,27926,27927],{"class":481},"          cat backend_update_command.sh\n",[151,27929,27930],{"class":469,"line":2489},[151,27931,27932],{"class":481},"          sudo chmod +x backend_update_command.sh\n",[151,27934,27935],{"class":469,"line":2497},[151,27936,27937],{"class":481},"          ./backend_update_command.sh\n",[736,27939,27328],{"id":27940},"passing-data-between-stacks",[11,27942,27943],{},"Pulumi uses stack references, Terraform uses remote state and CDK uses Stack Outputs or Stack References.",[11,27945,27946],{},"Here's what this looks like in Terraform",[459,27948,27952],{"className":27949,"code":27950,"language":27951,"meta":464,"style":464},"language-terraform shiki shiki-themes github-light github-dark monokai","data \"terraform_remote_state\" \"this\" {\n  backend = \"local\"\n\n  config = {\n    path = \"../base/terraform.tfstate\"\n  }\n}\n\nmodule \"main\" {\n  source = \"../../../modules/ad-hoc/app\"\n\n  vpc_id                         = data.terraform_remote_state.this.outputs.vpc_id\n  assets_bucket_name             = data.terraform_remote_state.this.outputs.assets_bucket_name\n  private_subnet_ids             = data.terraform_remote_state.this.outputs.private_subnet_ids\n  app_sg_id                      = data.terraform_remote_state.this.outputs.app_sg_id\n  alb_sg_id                      = data.terraform_remote_state.this.outputs.alb_sg_id\n  listener_arn                   = data.terraform_remote_state.this.outputs.listener_arn\n  alb_dns_name                   = data.terraform_remote_state.this.outputs.alb_dns_name\n  service_discovery_namespace_id = data.terraform_remote_state.this.outputs.service_discovery_namespace_id\n  rds_address                    = data.terraform_remote_state.this.outputs.rds_address\n  domain_name                    = data.terraform_remote_state.this.outputs.domain_name\n  base_stack_name                = data.terraform_remote_state.this.outputs.base_stack_name\n  region                         = var.region\n}\n","terraform",[30,27953,27954,27966,27976,27980,27989,27999,28003,28007,28011,28021,28031,28035,28063,28090,28116,28143,28169,28196,28222,28248,28275,28301,28328,28343],{"__ignoreMap":464},[151,27955,27956,27958,27961,27964],{"class":469,"line":470},[151,27957,12355],{"class":15254},[151,27959,27960],{"class":12360}," \"terraform_remote_state\"",[151,27962,27963],{"class":12360}," \"this\"",[151,27965,19833],{"class":503},[151,27967,27968,27971,27973],{"class":469,"line":488},[151,27969,27970],{"class":503},"  backend",[151,27972,19865],{"class":1869},[151,27974,27975],{"class":481}," \"local\"\n",[151,27977,27978],{"class":469,"line":500},[151,27979,1090],{"emptyLinePlaceholder":609},[151,27981,27982,27985,27987],{"class":469,"line":509},[151,27983,27984],{"class":503},"  config",[151,27986,19865],{"class":1869},[151,27988,19833],{"class":503},[151,27990,27991,27994,27996],{"class":469,"line":517},[151,27992,27993],{"class":503},"    path ",[151,27995,1876],{"class":1869},[151,27997,27998],{"class":481}," \"../base/terraform.tfstate\"\n",[151,28000,28001],{"class":469,"line":534},[151,28002,19957],{"class":503},[151,28004,28005],{"class":469,"line":1413},[151,28006,6274],{"class":503},[151,28008,28009],{"class":469,"line":1418},[151,28010,1090],{"emptyLinePlaceholder":609},[151,28012,28013,28016,28019],{"class":469,"line":2462},[151,28014,28015],{"class":15254},"module",[151,28017,28018],{"class":12360}," \"main\"",[151,28020,19833],{"class":503},[151,28022,28023,28026,28028],{"class":469,"line":2471},[151,28024,28025],{"class":503},"  source",[151,28027,19865],{"class":1869},[151,28029,28030],{"class":481}," \"../../../modules/ad-hoc/app\"\n",[151,28032,28033],{"class":469,"line":2480},[151,28034,1090],{"emptyLinePlaceholder":609},[151,28036,28037,28040,28043,28046,28048,28050,28052,28054,28056,28058,28060],{"class":469,"line":2489},[151,28038,28039],{"class":503},"  vpc_id",[151,28041,28042],{"class":1869},"                         =",[151,28044,28045],{"class":503}," data",[151,28047,643],{"class":1869},[151,28049,26752],{"class":503},[151,28051,643],{"class":1869},[151,28053,23252],{"class":503},[151,28055,643],{"class":1869},[151,28057,27279],{"class":503},[151,28059,643],{"class":1869},[151,28061,28062],{"class":503},"vpc_id\n",[151,28064,28065,28068,28071,28073,28075,28077,28079,28081,28083,28085,28087],{"class":469,"line":2497},[151,28066,28067],{"class":503},"  assets_bucket_name",[151,28069,28070],{"class":1869},"             =",[151,28072,28045],{"class":503},[151,28074,643],{"class":1869},[151,28076,26752],{"class":503},[151,28078,643],{"class":1869},[151,28080,23252],{"class":503},[151,28082,643],{"class":1869},[151,28084,27279],{"class":503},[151,28086,643],{"class":1869},[151,28088,28089],{"class":503},"assets_bucket_name\n",[151,28091,28092,28095,28097,28099,28101,28103,28105,28107,28109,28111,28113],{"class":469,"line":3140},[151,28093,28094],{"class":503},"  private_subnet_ids",[151,28096,28070],{"class":1869},[151,28098,28045],{"class":503},[151,28100,643],{"class":1869},[151,28102,26752],{"class":503},[151,28104,643],{"class":1869},[151,28106,23252],{"class":503},[151,28108,643],{"class":1869},[151,28110,27279],{"class":503},[151,28112,643],{"class":1869},[151,28114,28115],{"class":503},"private_subnet_ids\n",[151,28117,28118,28121,28124,28126,28128,28130,28132,28134,28136,28138,28140],{"class":469,"line":3149},[151,28119,28120],{"class":503},"  app_sg_id",[151,28122,28123],{"class":1869},"                      =",[151,28125,28045],{"class":503},[151,28127,643],{"class":1869},[151,28129,26752],{"class":503},[151,28131,643],{"class":1869},[151,28133,23252],{"class":503},[151,28135,643],{"class":1869},[151,28137,27279],{"class":503},[151,28139,643],{"class":1869},[151,28141,28142],{"class":503},"app_sg_id\n",[151,28144,28145,28148,28150,28152,28154,28156,28158,28160,28162,28164,28166],{"class":469,"line":3158},[151,28146,28147],{"class":503},"  alb_sg_id",[151,28149,28123],{"class":1869},[151,28151,28045],{"class":503},[151,28153,643],{"class":1869},[151,28155,26752],{"class":503},[151,28157,643],{"class":1869},[151,28159,23252],{"class":503},[151,28161,643],{"class":1869},[151,28163,27279],{"class":503},[151,28165,643],{"class":1869},[151,28167,28168],{"class":503},"alb_sg_id\n",[151,28170,28171,28174,28177,28179,28181,28183,28185,28187,28189,28191,28193],{"class":469,"line":3167},[151,28172,28173],{"class":503},"  listener_arn",[151,28175,28176],{"class":1869},"                   =",[151,28178,28045],{"class":503},[151,28180,643],{"class":1869},[151,28182,26752],{"class":503},[151,28184,643],{"class":1869},[151,28186,23252],{"class":503},[151,28188,643],{"class":1869},[151,28190,27279],{"class":503},[151,28192,643],{"class":1869},[151,28194,28195],{"class":503},"listener_arn\n",[151,28197,28198,28201,28203,28205,28207,28209,28211,28213,28215,28217,28219],{"class":469,"line":3175},[151,28199,28200],{"class":503},"  alb_dns_name",[151,28202,28176],{"class":1869},[151,28204,28045],{"class":503},[151,28206,643],{"class":1869},[151,28208,26752],{"class":503},[151,28210,643],{"class":1869},[151,28212,23252],{"class":503},[151,28214,643],{"class":1869},[151,28216,27279],{"class":503},[151,28218,643],{"class":1869},[151,28220,28221],{"class":503},"alb_dns_name\n",[151,28223,28224,28227,28229,28231,28233,28235,28237,28239,28241,28243,28245],{"class":469,"line":3184},[151,28225,28226],{"class":503},"  service_discovery_namespace_id",[151,28228,19865],{"class":1869},[151,28230,28045],{"class":503},[151,28232,643],{"class":1869},[151,28234,26752],{"class":503},[151,28236,643],{"class":1869},[151,28238,23252],{"class":503},[151,28240,643],{"class":1869},[151,28242,27279],{"class":503},[151,28244,643],{"class":1869},[151,28246,28247],{"class":503},"service_discovery_namespace_id\n",[151,28249,28250,28253,28256,28258,28260,28262,28264,28266,28268,28270,28272],{"class":469,"line":3193},[151,28251,28252],{"class":503},"  rds_address",[151,28254,28255],{"class":1869},"                    =",[151,28257,28045],{"class":503},[151,28259,643],{"class":1869},[151,28261,26752],{"class":503},[151,28263,643],{"class":1869},[151,28265,23252],{"class":503},[151,28267,643],{"class":1869},[151,28269,27279],{"class":503},[151,28271,643],{"class":1869},[151,28273,28274],{"class":503},"rds_address\n",[151,28276,28277,28280,28282,28284,28286,28288,28290,28292,28294,28296,28298],{"class":469,"line":3720},[151,28278,28279],{"class":503},"  domain_name",[151,28281,28255],{"class":1869},[151,28283,28045],{"class":503},[151,28285,643],{"class":1869},[151,28287,26752],{"class":503},[151,28289,643],{"class":1869},[151,28291,23252],{"class":503},[151,28293,643],{"class":1869},[151,28295,27279],{"class":503},[151,28297,643],{"class":1869},[151,28299,28300],{"class":503},"domain_name\n",[151,28302,28303,28306,28309,28311,28313,28315,28317,28319,28321,28323,28325],{"class":469,"line":3729},[151,28304,28305],{"class":503},"  base_stack_name",[151,28307,28308],{"class":1869},"                =",[151,28310,28045],{"class":503},[151,28312,643],{"class":1869},[151,28314,26752],{"class":503},[151,28316,643],{"class":1869},[151,28318,23252],{"class":503},[151,28320,643],{"class":1869},[151,28322,27279],{"class":503},[151,28324,643],{"class":1869},[151,28326,28327],{"class":503},"base_stack_name\n",[151,28329,28330,28333,28335,28338,28340],{"class":469,"line":3735},[151,28331,28332],{"class":503},"  region",[151,28334,28042],{"class":1869},[151,28336,28337],{"class":503}," var",[151,28339,643],{"class":1869},[151,28341,28342],{"class":503},"region\n",[151,28344,28345],{"class":469,"line":3745},[151,28346,6274],{"class":503},[11,28348,28349],{},"In CDK:",[459,28351,28353],{"className":26941,"code":28352,"language":26943,"meta":464,"style":464},"const baseStack = new Stack(app, 'ExampleAdHocBaseStack', { env, stackName: adHocBaseEnvName });\nbaseStack.node.setContext('config', adHocBaseEnvConfig);\n\nconst appStack = new Stack(app, 'ExampleAdHocAppStack', { env, stackName: adHocAppEnvName });\nappStack.node.setContext('config', adHocAppEnvConfig);\n\nconst adHocBase = new AdHocBase(baseStack, 'AdHocBase', { certificateArn, domainName });\n\nconst addHocApp = new AdHocApp(appStack, 'AdHocApp', {\n  baseStackName: adHocBaseEnvName,\n  vpc: adHocBase.vpc,\n  alb: adHocBase.alb,\n  appSecurityGroup: adHocBase.appSecurityGroup,\n  serviceDiscoveryNamespace: adHocBase.serviceDiscoveryNamespace,\n  rdsInstance: adHocBase.databaseInstance,\n  assetsBucket: adHocBase.assetsBucket,\n  domainName: adHocBase.domainName,\n  listener: adHocBase.listener,\n});\n",[30,28354,28355,28378,28394,28398,28419,28433,28437,28460,28464,28486,28491,28496,28501,28506,28511,28516,28521,28526,28531],{"__ignoreMap":464},[151,28356,28357,28359,28362,28364,28366,28369,28372,28375],{"class":469,"line":470},[151,28358,12348],{"class":12347},[151,28360,28361],{"class":12360}," baseStack",[151,28363,19865],{"class":1869},[151,28365,4236],{"class":1869},[151,28367,28368],{"class":473}," Stack",[151,28370,28371],{"class":503},"(app, ",[151,28373,28374],{"class":481},"'ExampleAdHocBaseStack'",[151,28376,28377],{"class":503},", { env, stackName: adHocBaseEnvName });\n",[151,28379,28380,28383,28386,28388,28391],{"class":469,"line":488},[151,28381,28382],{"class":503},"baseStack.node.",[151,28384,28385],{"class":473},"setContext",[151,28387,12386],{"class":503},[151,28389,28390],{"class":481},"'config'",[151,28392,28393],{"class":503},", adHocBaseEnvConfig);\n",[151,28395,28396],{"class":469,"line":500},[151,28397,1090],{"emptyLinePlaceholder":609},[151,28399,28400,28402,28405,28407,28409,28411,28413,28416],{"class":469,"line":509},[151,28401,12348],{"class":12347},[151,28403,28404],{"class":12360}," appStack",[151,28406,19865],{"class":1869},[151,28408,4236],{"class":1869},[151,28410,28368],{"class":473},[151,28412,28371],{"class":503},[151,28414,28415],{"class":481},"'ExampleAdHocAppStack'",[151,28417,28418],{"class":503},", { env, stackName: adHocAppEnvName });\n",[151,28420,28421,28424,28426,28428,28430],{"class":469,"line":517},[151,28422,28423],{"class":503},"appStack.node.",[151,28425,28385],{"class":473},[151,28427,12386],{"class":503},[151,28429,28390],{"class":481},[151,28431,28432],{"class":503},", adHocAppEnvConfig);\n",[151,28434,28435],{"class":469,"line":534},[151,28436,1090],{"emptyLinePlaceholder":609},[151,28438,28439,28441,28444,28446,28448,28451,28454,28457],{"class":469,"line":1413},[151,28440,12348],{"class":12347},[151,28442,28443],{"class":12360}," adHocBase",[151,28445,19865],{"class":1869},[151,28447,4236],{"class":1869},[151,28449,28450],{"class":473}," AdHocBase",[151,28452,28453],{"class":503},"(baseStack, ",[151,28455,28456],{"class":481},"'AdHocBase'",[151,28458,28459],{"class":503},", { certificateArn, domainName });\n",[151,28461,28462],{"class":469,"line":1418},[151,28463,1090],{"emptyLinePlaceholder":609},[151,28465,28466,28468,28471,28473,28475,28478,28481,28484],{"class":469,"line":2462},[151,28467,12348],{"class":12347},[151,28469,28470],{"class":12360}," addHocApp",[151,28472,19865],{"class":1869},[151,28474,4236],{"class":1869},[151,28476,28477],{"class":473}," AdHocApp",[151,28479,28480],{"class":503},"(appStack, ",[151,28482,28483],{"class":481},"'AdHocApp'",[151,28485,26968],{"class":503},[151,28487,28488],{"class":469,"line":2471},[151,28489,28490],{"class":503},"  baseStackName: adHocBaseEnvName,\n",[151,28492,28493],{"class":469,"line":2480},[151,28494,28495],{"class":503},"  vpc: adHocBase.vpc,\n",[151,28497,28498],{"class":469,"line":2489},[151,28499,28500],{"class":503},"  alb: adHocBase.alb,\n",[151,28502,28503],{"class":469,"line":2497},[151,28504,28505],{"class":503},"  appSecurityGroup: adHocBase.appSecurityGroup,\n",[151,28507,28508],{"class":469,"line":3140},[151,28509,28510],{"class":503},"  serviceDiscoveryNamespace: adHocBase.serviceDiscoveryNamespace,\n",[151,28512,28513],{"class":469,"line":3149},[151,28514,28515],{"class":503},"  rdsInstance: adHocBase.databaseInstance,\n",[151,28517,28518],{"class":469,"line":3158},[151,28519,28520],{"class":503},"  assetsBucket: adHocBase.assetsBucket,\n",[151,28522,28523],{"class":469,"line":3167},[151,28524,28525],{"class":503},"  domainName: adHocBase.domainName,\n",[151,28527,28528],{"class":469,"line":3175},[151,28529,28530],{"class":503},"  listener: adHocBase.listener,\n",[151,28532,28533],{"class":469,"line":3184},[151,28534,20850],{"class":503},[11,28536,28537],{},"and in Pulumi:",[459,28539,28541],{"className":26941,"code":28540,"language":26943,"meta":464,"style":464},"const stackReference = new pulumi.StackReference(`${org}/ad-hoc-base/${environment}`)\n\nconst vpcId = stackReference.getOutput(\"vpcId\") as pulumi.Output\u003Cstring>;\nconst assetsBucketName = stackReference.getOutput(\"assetsBucketName\") as pulumi.Output\u003Cstring>;\nconst privateSubnets = stackReference.getOutput(\"privateSubnetIds\") as pulumi.Output\u003Cstring[]>;\nconst appSgId = stackReference.getOutput(\"appSgId\") as pulumi.Output\u003Cstring>;\nconst albSgId = stackReference.getOutput(\"albSgId\") as pulumi.Output\u003Cstring>;\nconst listenerArn = stackReference.getOutput(\"listenerArn\") as pulumi.Output\u003Cstring>;\nconst albDnsName = stackReference.getOutput(\"albDnsName\") as pulumi.Output\u003Cstring>;\nconst serviceDiscoveryNamespaceId = stackReference.getOutput(\"serviceDiscoveryNamespaceId\") as pulumi.Output\u003Cstring>;\nconst rdsAddress = stackReference.getOutput(\"rdsAddress\") as pulumi.Output\u003Cstring>;\nconst domainName = stackReference.getOutput(\"domainName\") as pulumi.Output\u003Cstring>;\nconst baseStackName = stackReference.getOutput(\"baseStackName\") as pulumi.Output\u003Cstring>;\n\n// ad hoc app env\nconst adHocAppComponent = new AdHocAppComponent(\"AdHocAppComponent\", {\n  vpcId,\n  assetsBucketName,\n  privateSubnets,\n  appSgId,\n  albSgId,\n  listenerArn,\n  albDnsName,\n  serviceDiscoveryNamespaceId,\n  rdsAddress,\n  domainName,\n  baseStackName\n});\n",[30,28542,28543,28584,28588,28627,28661,28696,28730,28764,28798,28832,28866,28900,28934,28968,28972,28977,28998,29003,29008,29013,29018,29023,29028,29033,29038,29043,29048,29053],{"__ignoreMap":464},[151,28544,28545,28547,28550,28552,28554,28556,28559,28561,28563,28565,28568,28570,28573,28575,28578,28580,28582],{"class":469,"line":470},[151,28546,12348],{"class":12347},[151,28548,28549],{"class":12360}," stackReference",[151,28551,19865],{"class":1869},[151,28553,4236],{"class":1869},[151,28555,27660],{"class":503},[151,28557,28558],{"class":473},"StackReference",[151,28560,12386],{"class":503},[151,28562,2798],{"class":481},[151,28564,19871],{"class":19870},[151,28566,28567],{"class":503},"org",[151,28569,2001],{"class":19870},[151,28571,28572],{"class":481},"/ad-hoc-base/",[151,28574,19871],{"class":19870},[151,28576,28577],{"class":503},"environment",[151,28579,2001],{"class":19870},[151,28581,2798],{"class":481},[151,28583,3640],{"class":503},[151,28585,28586],{"class":469,"line":488},[151,28587,1090],{"emptyLinePlaceholder":609},[151,28589,28590,28592,28595,28597,28600,28603,28605,28608,28610,28612,28615,28617,28620,28622,28624],{"class":469,"line":500},[151,28591,12348],{"class":12347},[151,28593,28594],{"class":12360}," vpcId",[151,28596,19865],{"class":1869},[151,28598,28599],{"class":503}," stackReference.",[151,28601,28602],{"class":473},"getOutput",[151,28604,12386],{"class":503},[151,28606,28607],{"class":481},"\"vpcId\"",[151,28609,16995],{"class":503},[151,28611,16998],{"class":1869},[151,28613,28614],{"class":15254}," pulumi",[151,28616,643],{"class":503},[151,28618,28619],{"class":15254},"Output",[151,28621,3613],{"class":503},[151,28623,23282],{"class":6205},[151,28625,28626],{"class":503},">;\n",[151,28628,28629,28631,28634,28636,28638,28640,28642,28645,28647,28649,28651,28653,28655,28657,28659],{"class":469,"line":509},[151,28630,12348],{"class":12347},[151,28632,28633],{"class":12360}," assetsBucketName",[151,28635,19865],{"class":1869},[151,28637,28599],{"class":503},[151,28639,28602],{"class":473},[151,28641,12386],{"class":503},[151,28643,28644],{"class":481},"\"assetsBucketName\"",[151,28646,16995],{"class":503},[151,28648,16998],{"class":1869},[151,28650,28614],{"class":15254},[151,28652,643],{"class":503},[151,28654,28619],{"class":15254},[151,28656,3613],{"class":503},[151,28658,23282],{"class":6205},[151,28660,28626],{"class":503},[151,28662,28663,28665,28668,28670,28672,28674,28676,28679,28681,28683,28685,28687,28689,28691,28693],{"class":469,"line":517},[151,28664,12348],{"class":12347},[151,28666,28667],{"class":12360}," privateSubnets",[151,28669,19865],{"class":1869},[151,28671,28599],{"class":503},[151,28673,28602],{"class":473},[151,28675,12386],{"class":503},[151,28677,28678],{"class":481},"\"privateSubnetIds\"",[151,28680,16995],{"class":503},[151,28682,16998],{"class":1869},[151,28684,28614],{"class":15254},[151,28686,643],{"class":503},[151,28688,28619],{"class":15254},[151,28690,3613],{"class":503},[151,28692,23282],{"class":6205},[151,28694,28695],{"class":503},"[]>;\n",[151,28697,28698,28700,28703,28705,28707,28709,28711,28714,28716,28718,28720,28722,28724,28726,28728],{"class":469,"line":534},[151,28699,12348],{"class":12347},[151,28701,28702],{"class":12360}," appSgId",[151,28704,19865],{"class":1869},[151,28706,28599],{"class":503},[151,28708,28602],{"class":473},[151,28710,12386],{"class":503},[151,28712,28713],{"class":481},"\"appSgId\"",[151,28715,16995],{"class":503},[151,28717,16998],{"class":1869},[151,28719,28614],{"class":15254},[151,28721,643],{"class":503},[151,28723,28619],{"class":15254},[151,28725,3613],{"class":503},[151,28727,23282],{"class":6205},[151,28729,28626],{"class":503},[151,28731,28732,28734,28737,28739,28741,28743,28745,28748,28750,28752,28754,28756,28758,28760,28762],{"class":469,"line":1413},[151,28733,12348],{"class":12347},[151,28735,28736],{"class":12360}," albSgId",[151,28738,19865],{"class":1869},[151,28740,28599],{"class":503},[151,28742,28602],{"class":473},[151,28744,12386],{"class":503},[151,28746,28747],{"class":481},"\"albSgId\"",[151,28749,16995],{"class":503},[151,28751,16998],{"class":1869},[151,28753,28614],{"class":15254},[151,28755,643],{"class":503},[151,28757,28619],{"class":15254},[151,28759,3613],{"class":503},[151,28761,23282],{"class":6205},[151,28763,28626],{"class":503},[151,28765,28766,28768,28771,28773,28775,28777,28779,28782,28784,28786,28788,28790,28792,28794,28796],{"class":469,"line":1418},[151,28767,12348],{"class":12347},[151,28769,28770],{"class":12360}," listenerArn",[151,28772,19865],{"class":1869},[151,28774,28599],{"class":503},[151,28776,28602],{"class":473},[151,28778,12386],{"class":503},[151,28780,28781],{"class":481},"\"listenerArn\"",[151,28783,16995],{"class":503},[151,28785,16998],{"class":1869},[151,28787,28614],{"class":15254},[151,28789,643],{"class":503},[151,28791,28619],{"class":15254},[151,28793,3613],{"class":503},[151,28795,23282],{"class":6205},[151,28797,28626],{"class":503},[151,28799,28800,28802,28805,28807,28809,28811,28813,28816,28818,28820,28822,28824,28826,28828,28830],{"class":469,"line":2462},[151,28801,12348],{"class":12347},[151,28803,28804],{"class":12360}," albDnsName",[151,28806,19865],{"class":1869},[151,28808,28599],{"class":503},[151,28810,28602],{"class":473},[151,28812,12386],{"class":503},[151,28814,28815],{"class":481},"\"albDnsName\"",[151,28817,16995],{"class":503},[151,28819,16998],{"class":1869},[151,28821,28614],{"class":15254},[151,28823,643],{"class":503},[151,28825,28619],{"class":15254},[151,28827,3613],{"class":503},[151,28829,23282],{"class":6205},[151,28831,28626],{"class":503},[151,28833,28834,28836,28839,28841,28843,28845,28847,28850,28852,28854,28856,28858,28860,28862,28864],{"class":469,"line":2471},[151,28835,12348],{"class":12347},[151,28837,28838],{"class":12360}," serviceDiscoveryNamespaceId",[151,28840,19865],{"class":1869},[151,28842,28599],{"class":503},[151,28844,28602],{"class":473},[151,28846,12386],{"class":503},[151,28848,28849],{"class":481},"\"serviceDiscoveryNamespaceId\"",[151,28851,16995],{"class":503},[151,28853,16998],{"class":1869},[151,28855,28614],{"class":15254},[151,28857,643],{"class":503},[151,28859,28619],{"class":15254},[151,28861,3613],{"class":503},[151,28863,23282],{"class":6205},[151,28865,28626],{"class":503},[151,28867,28868,28870,28873,28875,28877,28879,28881,28884,28886,28888,28890,28892,28894,28896,28898],{"class":469,"line":2480},[151,28869,12348],{"class":12347},[151,28871,28872],{"class":12360}," rdsAddress",[151,28874,19865],{"class":1869},[151,28876,28599],{"class":503},[151,28878,28602],{"class":473},[151,28880,12386],{"class":503},[151,28882,28883],{"class":481},"\"rdsAddress\"",[151,28885,16995],{"class":503},[151,28887,16998],{"class":1869},[151,28889,28614],{"class":15254},[151,28891,643],{"class":503},[151,28893,28619],{"class":15254},[151,28895,3613],{"class":503},[151,28897,23282],{"class":6205},[151,28899,28626],{"class":503},[151,28901,28902,28904,28907,28909,28911,28913,28915,28918,28920,28922,28924,28926,28928,28930,28932],{"class":469,"line":2489},[151,28903,12348],{"class":12347},[151,28905,28906],{"class":12360}," domainName",[151,28908,19865],{"class":1869},[151,28910,28599],{"class":503},[151,28912,28602],{"class":473},[151,28914,12386],{"class":503},[151,28916,28917],{"class":481},"\"domainName\"",[151,28919,16995],{"class":503},[151,28921,16998],{"class":1869},[151,28923,28614],{"class":15254},[151,28925,643],{"class":503},[151,28927,28619],{"class":15254},[151,28929,3613],{"class":503},[151,28931,23282],{"class":6205},[151,28933,28626],{"class":503},[151,28935,28936,28938,28941,28943,28945,28947,28949,28952,28954,28956,28958,28960,28962,28964,28966],{"class":469,"line":2497},[151,28937,12348],{"class":12347},[151,28939,28940],{"class":12360}," baseStackName",[151,28942,19865],{"class":1869},[151,28944,28599],{"class":503},[151,28946,28602],{"class":473},[151,28948,12386],{"class":503},[151,28950,28951],{"class":481},"\"baseStackName\"",[151,28953,16995],{"class":503},[151,28955,16998],{"class":1869},[151,28957,28614],{"class":15254},[151,28959,643],{"class":503},[151,28961,28619],{"class":15254},[151,28963,3613],{"class":503},[151,28965,23282],{"class":6205},[151,28967,28626],{"class":503},[151,28969,28970],{"class":469,"line":3140},[151,28971,1090],{"emptyLinePlaceholder":609},[151,28973,28974],{"class":469,"line":3149},[151,28975,28976],{"class":1527},"// ad hoc app env\n",[151,28978,28979,28981,28984,28986,28988,28991,28993,28996],{"class":469,"line":3158},[151,28980,12348],{"class":12347},[151,28982,28983],{"class":12360}," adHocAppComponent",[151,28985,19865],{"class":1869},[151,28987,4236],{"class":1869},[151,28989,28990],{"class":473}," AdHocAppComponent",[151,28992,12386],{"class":503},[151,28994,28995],{"class":481},"\"AdHocAppComponent\"",[151,28997,26968],{"class":503},[151,28999,29000],{"class":469,"line":3167},[151,29001,29002],{"class":503},"  vpcId,\n",[151,29004,29005],{"class":469,"line":3175},[151,29006,29007],{"class":503},"  assetsBucketName,\n",[151,29009,29010],{"class":469,"line":3184},[151,29011,29012],{"class":503},"  privateSubnets,\n",[151,29014,29015],{"class":469,"line":3193},[151,29016,29017],{"class":503},"  appSgId,\n",[151,29019,29020],{"class":469,"line":3720},[151,29021,29022],{"class":503},"  albSgId,\n",[151,29024,29025],{"class":469,"line":3729},[151,29026,29027],{"class":503},"  listenerArn,\n",[151,29029,29030],{"class":469,"line":3735},[151,29031,29032],{"class":503},"  albDnsName,\n",[151,29034,29035],{"class":469,"line":3745},[151,29036,29037],{"class":503},"  serviceDiscoveryNamespaceId,\n",[151,29039,29040],{"class":469,"line":3754},[151,29041,29042],{"class":503},"  rdsAddress,\n",[151,29044,29045],{"class":469,"line":3760},[151,29046,29047],{"class":503},"  domainName,\n",[151,29049,29050],{"class":469,"line":3773},[151,29051,29052],{"class":503},"  baseStackName\n",[151,29054,29055],{"class":469,"line":3782},[151,29056,20850],{"class":503},[56,29058,29060],{"id":29059},"cli-scaffolding","CLI scaffolding",[11,29062,29063],{},"CDK and Pulumi have some good options for how to scaffold a project.",[76,29065,29066,29080,29086,29094],{},[79,29067,29068,29069,29072,29073,29076,29077,29079],{},"Pulumi has ",[30,29070,29071],{},"pulumi new aws-typescript"," among lots of other options (run ",[30,29074,29075],{},"pulumi new -l"," to see over 200 project types). I used this to create the library itself, the examples and the pulumi projects that I use in ",[30,29078,26634],{}," that consume the library.",[79,29081,29082,29083,29085],{},"CDK has ",[30,29084,26199],{}," CLI commands which can help set up either library code or project code",[79,29087,29088,29089,187,29091,29093],{},"The major benefits of these tools is setting up ",[30,29090,26207],{},[30,29092,21209],{}," correctly",[79,29095,29096],{},"Terraform is so simple that it doesn't really need tooling for scaffolding",[56,29098,29100],{"id":29099},"best-practices","Best practices",[11,29102,29103,29104,29106,29107,29112],{},"For ",[30,29105,26110],{},", I tried to follow the recommendations from ",[20,29108,29111],{"href":29109,"rel":29110},"https://www.terraform-best-practices.com/",[24],"terraform-best-practices.com"," which helped me a lot with things like consistent naming patterns and directory structures. For example:",[76,29114,29115],{},[79,29116,29117,29118,29120],{},"use the name ",[30,29119,23252],{}," for resources in a module where that resource is the only resource of its type",[11,29122,29123],{},"CDK and Pulumi lend themselves to more nesting and abstractions because they can be written in more familiar programming languages with better abstractions, functions, loops, classes, etc., so there are some differences in directory structure of my libraries when comparing Terraform to both CDK and Pulumi.",[11,29125,29126,29127,106,29129,106,29131,106,29134,106,29137,29140,29141,187,29144,29147],{},"For Pulumi and CDK, I mostly tried to follow along with recommendations from their documentation and example projects. While working with Pulumi I struggled a bit with the concepts of ",[30,29128,26880],{},[30,29130,27280],{},[30,29132,29133],{},"pulumi.interpolate",[30,29135,29136],{},"apply()",[30,29138,29139],{},"all()"," and the differences between ",[30,29142,29143],{},"getX",[30,29145,29146],{},"getXOutput",". There is a little bit of a learning curve here, but the documentation and examples go a long way in showing how to do things the right way.",[56,29149,29151],{"id":29150},"environment-configuration","Environment configuration",[11,29153,29154],{},"Environment configuration allows for either a base or app stack to be configured with non-default values. For example:",[76,29156,29157,29160],{},[79,29158,29159],{},"you may decide to start a new base environment but you want to provision a powerful database instance class and size. You would change this using environment configuration",[79,29161,29162],{},"You might want to create an ad hoc app environment but you need it to include some special environment variables, you could set these in environment config.",[11,29164,29165],{},"In the examples above, our IaC can optionally take environment configuration values that overwrite default values, or extend default values.",[76,29167,29168,29179,29186],{},[79,29169,29170,29171,129,29174,748],{},"Pulumi defines environment-specific config in files called ",[30,29172,29173],{},"Pulumi.{env}.yaml",[20,29175,29178],{"href":29176,"rel":29177},"https://www.pulumi.com/docs/intro/concepts/config/",[24],"Pulumi article on configuration",[79,29180,29181,29182,29185],{},"Terraform uses ",[30,29183,29184],{},"{env}.tfvars"," for this type of configuration",[79,29187,29188,29189,29192],{},"CDK has several options for this type of configuration (",[30,29190,29191],{},"cdk.context.json",", extending stack props, etc.)",[11,29194,29195,29196,29198,29199,29202],{},"For CDK I have been using ",[30,29197,28385],{}," and the ",[30,29200,29201],{},"tryGetContext"," method:",[11,29204,29205,29207],{},[30,29206,28385],{}," needs to be set on the node before any child nodes are added:",[459,29209,29211],{"className":26941,"code":29210,"language":26943,"meta":464,"style":464},"const baseStack = new Stack(app, 'ExampleAdHocBaseStack', { env, stackName: adHocBaseEnvName });\nbaseStack.node.setContext('config', adHocBaseEnvConfig);\n\nconst appStack = new Stack(app, 'ExampleAdHocAppStack', { env, stackName: adHocAppEnvName });\nappStack.node.setContext('config', adHocAppEnvConfig);\n",[30,29212,29213,29231,29243,29247,29265],{"__ignoreMap":464},[151,29214,29215,29217,29219,29221,29223,29225,29227,29229],{"class":469,"line":470},[151,29216,12348],{"class":12347},[151,29218,28361],{"class":12360},[151,29220,19865],{"class":1869},[151,29222,4236],{"class":1869},[151,29224,28368],{"class":473},[151,29226,28371],{"class":503},[151,29228,28374],{"class":481},[151,29230,28377],{"class":503},[151,29232,29233,29235,29237,29239,29241],{"class":469,"line":488},[151,29234,28382],{"class":503},[151,29236,28385],{"class":473},[151,29238,12386],{"class":503},[151,29240,28390],{"class":481},[151,29242,28393],{"class":503},[151,29244,29245],{"class":469,"line":500},[151,29246,1090],{"emptyLinePlaceholder":609},[151,29248,29249,29251,29253,29255,29257,29259,29261,29263],{"class":469,"line":509},[151,29250,12348],{"class":12347},[151,29252,28404],{"class":12360},[151,29254,19865],{"class":1869},[151,29256,4236],{"class":1869},[151,29258,28368],{"class":473},[151,29260,28371],{"class":503},[151,29262,28415],{"class":481},[151,29264,28418],{"class":503},[151,29266,29267,29269,29271,29273,29275],{"class":469,"line":517},[151,29268,28423],{"class":503},[151,29270,28385],{"class":473},[151,29272,12386],{"class":503},[151,29274,28390],{"class":481},[151,29276,28432],{"class":503},[11,29278,29279],{},"And the config objects are read from JSON files like this:",[459,29281,29283],{"className":26941,"code":29282,"language":26943,"meta":464,"style":464},"var adHocBaseEnvConfig = JSON.parse(fs.readFileSync(`src/examples/ad-hoc/base/config/${adHocBaseEnvName}.json`, 'utf8'));\nvar adHocAppEnvConfig = JSON.parse(fs.readFileSync(`src/examples/ad-hoc/app/config/${adHocAppEnvName}.json`, 'utf8'));\n",[30,29284,29285,29332],{"__ignoreMap":464},[151,29286,29287,29290,29293,29295,29298,29300,29303,29306,29309,29311,29314,29316,29319,29321,29324,29326,29329],{"class":469,"line":470},[151,29288,29289],{"class":12347},"var",[151,29291,29292],{"class":503}," adHocBaseEnvConfig ",[151,29294,1876],{"class":1869},[151,29296,29297],{"class":12360}," JSON",[151,29299,643],{"class":503},[151,29301,29302],{"class":473},"parse",[151,29304,29305],{"class":503},"(fs.",[151,29307,29308],{"class":473},"readFileSync",[151,29310,12386],{"class":503},[151,29312,29313],{"class":481},"`src/examples/ad-hoc/base/config/",[151,29315,19871],{"class":19870},[151,29317,29318],{"class":503},"adHocBaseEnvName",[151,29320,2001],{"class":19870},[151,29322,29323],{"class":481},".json`",[151,29325,106],{"class":503},[151,29327,29328],{"class":481},"'utf8'",[151,29330,29331],{"class":503},"));\n",[151,29333,29334,29336,29339,29341,29343,29345,29347,29349,29351,29353,29356,29358,29361,29363,29365,29367,29369],{"class":469,"line":488},[151,29335,29289],{"class":12347},[151,29337,29338],{"class":503}," adHocAppEnvConfig ",[151,29340,1876],{"class":1869},[151,29342,29297],{"class":12360},[151,29344,643],{"class":503},[151,29346,29302],{"class":473},[151,29348,29305],{"class":503},[151,29350,29308],{"class":473},[151,29352,12386],{"class":503},[151,29354,29355],{"class":481},"`src/examples/ad-hoc/app/config/",[151,29357,19871],{"class":19870},[151,29359,29360],{"class":503},"adHocAppEnvName",[151,29362,2001],{"class":19870},[151,29364,29323],{"class":481},[151,29366,106],{"class":503},[151,29368,29328],{"class":481},[151,29370,29331],{"class":503},[11,29372,29373],{},"The context can be used in constructs like this:",[459,29375,29377],{"className":26941,"code":29376,"language":26943,"meta":464,"style":464},"    const extraEnvVars = this.node.tryGetContext('config').extraEnvVars;\n",[30,29378,29379],{"__ignoreMap":464},[151,29380,29381,29383,29386,29388,29390,29393,29395,29397,29399],{"class":469,"line":470},[151,29382,19860],{"class":12347},[151,29384,29385],{"class":12360}," extraEnvVars",[151,29387,19865],{"class":1869},[151,29389,2324],{"class":15289},[151,29391,29392],{"class":503},".node.",[151,29394,29201],{"class":473},[151,29396,12386],{"class":503},[151,29398,28390],{"class":481},[151,29400,29401],{"class":503},").extraEnvVars;\n",[11,29403,29404],{},"Pulumi has similar functions for getting context values, here's an example of how I get extra environment variables for app environments using Pulumi's config:",[459,29406,29408],{"className":26941,"code":29407,"language":26943,"meta":464,"style":464},"    interface EnvVar {\n      name: string;\n      value: string;\n    }\n\n    let config = new pulumi.Config();\n    let extraEnvVars = config.getObject\u003CEnvVar[]>(\"extraEnvVars\");\n",[30,29409,29410,29420,29432,29443,29447,29451,29470],{"__ignoreMap":464},[151,29411,29412,29415,29418],{"class":469,"line":470},[151,29413,29414],{"class":12347},"    interface",[151,29416,29417],{"class":15254}," EnvVar",[151,29419,19833],{"class":503},[151,29421,29422,29425,29427,29430],{"class":469,"line":488},[151,29423,29424],{"class":12354},"      name",[151,29426,208],{"class":1869},[151,29428,29429],{"class":6205}," string",[151,29431,20086],{"class":503},[151,29433,29434,29437,29439,29441],{"class":469,"line":500},[151,29435,29436],{"class":12354},"      value",[151,29438,208],{"class":1869},[151,29440,29429],{"class":6205},[151,29442,20086],{"class":503},[151,29444,29445],{"class":469,"line":509},[151,29446,9461],{"class":503},[151,29448,29449],{"class":469,"line":517},[151,29450,1090],{"emptyLinePlaceholder":609},[151,29452,29453,29456,29459,29461,29463,29465,29468],{"class":469,"line":534},[151,29454,29455],{"class":12347},"    let",[151,29457,29458],{"class":503}," config ",[151,29460,1876],{"class":1869},[151,29462,4236],{"class":1869},[151,29464,27660],{"class":503},[151,29466,29467],{"class":473},"Config",[151,29469,20012],{"class":503},[151,29471,29472,29474,29477,29479,29482,29485,29487,29490,29493,29496],{"class":469,"line":1413},[151,29473,29455],{"class":12347},[151,29475,29476],{"class":503}," extraEnvVars ",[151,29478,1876],{"class":1869},[151,29480,29481],{"class":503}," config.",[151,29483,29484],{"class":473},"getObject",[151,29486,3613],{"class":503},[151,29488,29489],{"class":15254},"EnvVar",[151,29491,29492],{"class":503},"[]>(",[151,29494,29495],{"class":481},"\"extraEnvVars\"",[151,29497,20129],{"class":503},[11,29499,29500,29501,29504,29505,29508],{},"In my ",[30,29502,29503],{},"Pulumi.alpha.yaml"," file I have the ",[30,29506,29507],{},"extraEnvVars"," set like this:",[459,29510,29512],{"className":14359,"code":29511,"language":14361,"meta":464,"style":464},"config:\n  aws:region: us-east-1\n  extraEnvVars:\n    - name: FOO\n      value: BAR\n    - name: BIZ\n      value: BUZ\n",[30,29513,29514,29520,29530,29537,29549,29558,29569],{"__ignoreMap":464},[151,29515,29516,29518],{"class":469,"line":470},[151,29517,15233],{"class":14368},[151,29519,14372],{"class":503},[151,29521,29522,29525,29527],{"class":469,"line":488},[151,29523,29524],{"class":14368},"  aws:region",[151,29526,6208],{"class":503},[151,29528,29529],{"class":481},"us-east-1\n",[151,29531,29532,29535],{"class":469,"line":500},[151,29533,29534],{"class":14368},"  extraEnvVars",[151,29536,14372],{"class":503},[151,29538,29539,29542,29544,29546],{"class":469,"line":509},[151,29540,29541],{"class":503},"    - ",[151,29543,20415],{"class":14368},[151,29545,6208],{"class":503},[151,29547,29548],{"class":481},"FOO\n",[151,29550,29551,29553,29555],{"class":469,"line":517},[151,29552,29436],{"class":14368},[151,29554,6208],{"class":503},[151,29556,29557],{"class":481},"BAR\n",[151,29559,29560,29562,29564,29566],{"class":469,"line":534},[151,29561,29541],{"class":503},[151,29563,20415],{"class":14368},[151,29565,6208],{"class":503},[151,29567,29568],{"class":481},"BIZ\n",[151,29570,29571,29573,29575],{"class":469,"line":1413},[151,29572,29436],{"class":14368},[151,29574,6208],{"class":503},[151,29576,29577],{"class":481},"BUZ\n",[11,29579,29580],{},"I haven't done too much with configuration, but it seems like the right place to build out all of the dials and switches for optional settings in stack resources that you want people to be able to change in their ad hoc environments, or that you want to set per \"production\" environment (QA, stage, prod, etc.)",[56,29582,29584],{"id":29583},"local-development","Local development",[11,29586,29587,29588,29590],{},"Using the Makefile targets in each library repo, my process for developing ",[30,29589,25986],{},"s involves making code changes followed by Makefile targets that preview/plan/diff against my AWS account, then running deploy/apply/up and waiting for things to finish deploying. Once I can validate that things are looking correct in my account, I run the destroy command and make sure that all of the resources are removed successfully. RDS instances can take up to 10 minutes to create, which means that the base stack takes some time to test. The app environment is able to be spun up quickly, but it can sometimes get stuck and take some time to delete services.",[11,29592,29593],{},"Here are some sample times for deploying ad hoc stacks with CDK.",[459,29595,29598],{"className":29596,"code":29597,"language":997},[995],"# CDK ad hoc base deployment time\n\n ✅  ExampleAdHocBaseStack (dev)\n\n✨  Deployment time: 629.64s\n\n# CDK ad hoc app deployment time\n\n ✅  ExampleAdHocAppStack (alpha)\n\n✨  Deployment time: 126.62s\n",[30,29599,29597],{"__ignoreMap":464},[11,29601,29602,29603,29605],{},"Here is an example of what the ",[30,29604,10638],{}," commands shows for the ad-hoc base stack:",[459,29607,29610],{"className":29608,"code":29609,"language":997},[995],"# Pulumi preview\n~/git/github/pulumi-aws-django$ pulumi -C examples/ad-hoc/base --stack dev preview\nPreviewing update (dev)\n\nView Live: https://app.pulumi.com/briancaffey/ad-hoc-base/dev/previews/718625b2-48f5-4ef4-8ed4-9b2694fda64a\n\n     Type                                                    Name                        Plan\n +   pulumi:pulumi:Stack                                     ad-hoc-base-dev             create\n +   └─ pulumi-contrib:components:AdHocBaseEnv               myAdHocEnv                  create\n +      ├─ pulumi-contrib:components:AlbResources            AlbResources                create\n +      │  ├─ aws:alb:TargetGroup                            DefaultTg                   create\n +      │  ├─ aws:alb:LoadBalancer                           LoadBalancer                create\n +      │  ├─ aws:alb:Listener                               HttpListener                create\n +      │  └─ aws:alb:Listener                               HttpsListener               create\n +      ├─ pulumi-contrib:components:BastionHostResources    BastionHostResources        create\n +      │  ├─ aws:iam:Role                                   BastionHostRole             create\n +      │  ├─ aws:iam:RolePolicy                             BastionHostPolicy           create\n +      │  ├─ aws:iam:InstanceProfile                        BastionHostInstanceProfile  create\n +      │  └─ aws:ec2:Instance                               BastionHostInstance         create\n +      ├─ pulumi-contrib:components:RdsResources            RdsResources                create\n +      │  ├─ aws:rds:SubnetGroup                            DbSubnetGroup               create\n +      │  ├─ aws:ec2:SecurityGroup                          RdsSecurityGroup            create\n +      │  └─ aws:rds:Instance                               DbInstance                  create\n +      ├─ pulumi-contrib:components:SecurityGroupResources  SecurityGroupResources      create\n +      │  ├─ aws:ec2:SecurityGroup                          AlbSecurityGroup            create\n +      │  └─ aws:ec2:SecurityGroup                          AppSecurityGroup            create\n +      ├─ aws:s3:Bucket                                     assetsBucket                create\n +      ├─ awsx:ec2:Vpc                                      dev                         create\n +      │  └─ aws:ec2:Vpc                                    dev                         create\n +      │     ├─ aws:ec2:InternetGateway                     dev                         create\n +      │     ├─ aws:ec2:Subnet                              dev-private-1               create\n +      │     │  └─ aws:ec2:RouteTable                       dev-private-1               create\n +      │     │     ├─ aws:ec2:RouteTableAssociation         dev-private-1               create\n +      │     │     └─ aws:ec2:Route                         dev-private-1               create\n +      │     ├─ aws:ec2:Subnet                              dev-private-2               create\n +      │     │  └─ aws:ec2:RouteTable                       dev-private-2               create\n +      │     │     ├─ aws:ec2:RouteTableAssociation         dev-private-2               create\n +      │     │     └─ aws:ec2:Route                         dev-private-2               create\n +      │     ├─ aws:ec2:Subnet                              dev-public-1                create\n +      │     │  ├─ aws:ec2:RouteTable                       dev-public-1                create\n +      │     │  │  ├─ aws:ec2:RouteTableAssociation         dev-public-1                create\n +      │     │  │  └─ aws:ec2:Route                         dev-public-1                create\n +      │     │  ├─ aws:ec2:Eip                              dev-1                       create\n +      │     │  └─ aws:ec2:NatGateway                       dev-1                       create\n +      │     └─ aws:ec2:Subnet                              dev-public-2                create\n +      │        ├─ aws:ec2:RouteTable                       dev-public-2                create\n +      │        │  ├─ aws:ec2:RouteTableAssociation         dev-public-2                create\n +      │        │  └─ aws:ec2:Route                         dev-public-2                create\n +      │        ├─ aws:ec2:Eip                              dev-2                       create\n +      │        └─ aws:ec2:NatGateway                       dev-2                       create\n +      └─ aws:servicediscovery:PrivateDnsNamespace          PrivateDnsNamespace         create\n\n\nOutputs:\n    albDnsName                 : output\u003Cstring>\n    albSgId                    : output\u003Cstring>\n    appSgId                    : output\u003Cstring>\n    assetsBucketName           : output\u003Cstring>\n    baseStackName              : \"dev\"\n    bastionHostInstanceId      : output\u003Cstring>\n    domainName                 : \"example.com\"\n    listenerArn                : output\u003Cstring>\n    privateSubnetIds           : output\u003Cstring>\n    rdsAddress                 : output\u003Cstring>\n    serviceDiscoveryNamespaceId: output\u003Cstring>\n    vpcId                      : output\u003Cstring>\n\nResources:\n    + 44 to create\n",[30,29611,29609],{"__ignoreMap":464},[56,29613,29615],{"id":29614},"running-infrastructure-pipelines-in-github-actions","Running infrastructure pipelines in GitHub Actions",[11,29617,29618],{},[51,29619,29620],{},"I don't currently have GitHub Actions working for all tools in all environments, this part is still a WIP but is working at a basic level. Another item for the backlog!",[11,29622,27190,29623,29626,29627,29629,29630,29633],{},[30,29624,29625],{},".github/workflows"," directory of the ",[30,29628,26634],{}," repo, I will have the following ",[30,29631,29632],{},"2 * 2 * 2 * 3 = 24"," pipelines for running infrastructure as code pipelines:",[459,29635,29637],{"className":461,"code":29636,"language":463,"meta":464,"style":464},"{ad_hoc,prod}_{base,app}_{create_update,destroy}_{cdk,terraform,pulumi}.yml\n",[30,29638,29639],{"__ignoreMap":464},[151,29640,29641],{"class":469,"line":470},[151,29642,29636],{"class":503},[76,29644,29645,29648,29651],{},[79,29646,29647],{},"For CDK I'm using CDK CLI commands",[79,29649,29650],{},"For Terraform I'm also using terraform CLI commands",[79,29652,29653],{},"For Pulumi I'm using the official Pulumi GitHub Action",[11,29655,29068,29656,29661],{},[20,29657,29660],{"href":29658,"rel":29659},"https://www.pulumi.com/docs/guides/continuous-delivery/github-actions/",[24],"a great article"," about how to use their official GitHub Action. This action calls the Pulumi CLI under the hood with all of the correct flags.",[11,29663,29664],{},"The general pattern that all of these pipelines use is:",[76,29666,29667,29670,29673],{},[79,29668,29669],{},"Do a synth/plan/preview, and upload the synth/plan/preview file to an artifact",[79,29671,29672],{},"Pause and wait on manual review of the planned changes",[79,29674,29675],{},"download the artifact and run deploy/apply/up against it, or optionally cancel the operation if the changes you see in the GitHub Actions pipeline logs are not what you expected.",[11,29677,29678],{},"I do this by having two jobs in each GitHub Action: one for synth/plan/preview and one for deploy/apply/up.",[11,29680,29681,29682,29684],{},"The job for deploy/apply/up includes an ",[30,29683,28577],{}," that is configured in GitHub to be a protected environment that requires approvals. Even if you are the only approver (which I am on this project), it is the easiest and safest way preview infrastructure changes before they happen. If you see something in the plan and it isn't what you wanted to change, you cancel the job.",[56,29686,29688],{"id":29687},"application-deployments","Application deployments",[76,29690,29691,29694,29700],{},[79,29692,29693],{},"There are two GitHub Actions pipelines for deploying the frontend and the backend. Both of these pipelines run bash scripts that call AWS CLI commands to perform rolling updates on all of the services used in the application (frontend, API, workers, scheduler)",[79,29695,29696,29697,29699],{},"The backend deployment script runs database migrations, the ",[30,29698,27586],{}," command and any other commands needed to run before the rolling update starts (clearing the cache, loading fixtures, etc.)",[79,29701,29702],{},"What is important to note here is that application deployments are not dependent on the IaC tool we use. Since we are tagging things consistently across CDK, Terraform and Pulumi, we can look up resources by tag rather than getting \"outputs\" of the app stacks.",[56,29704,29706],{"id":29705},"interacting-with-aws-via-iac","Interacting with AWS via IaC",[76,29708,29709,29718,29733],{},[79,29710,29711,29712,29717],{},"CDK interacts directly with CloudFormation (and custom resources which allow for running arbitrary SDK calls and lambda functions) and provides ",[20,29713,29716],{"href":29714,"rel":29715},"https://docs.aws.amazon.com/cdk/v2/guide/constructs.html",[24],"L1, L2 and L3 constructs"," which offer different levels of abstraction over CloudFormation.",[79,29719,29720,29721,29198,29726,643],{},"Terraform has the ",[20,29722,29725],{"href":29723,"rel":29724},"https://registry.terraform.io/providers/hashicorp/aws/latest/docs",[24],"AWS Provider",[20,29727,29730],{"href":29728,"rel":29729},"https://registry.terraform.io/namespaces/terraform-aws-modules",[24],[30,29731,29732],{},"terraform-aws-modules",[79,29734,29735,29736,16995,29738,187,29741,129,29744,16995,29749,187,29752,26792,29755,643],{},"Pulumi has AWS Classic (",[30,29737,26008],{},[15,29739,29740],{},"provider",[30,29742,29743],{},"AWSx",[20,29745,29748],{"href":29746,"rel":29747},"https://www.pulumi.com/registry/packages/awsx/",[24],"Crosswalk for Pulumi",[15,29750,29751],{},"library",[30,29753,29754],{},"aws_native",[15,29756,29740],{},[210,29758,29759],{},[11,29760,29761,29763],{},[30,29762,29754],{}," \"manages and provisions resources using the AWS Cloud Control API, which typically supports new AWS features on the day of launch.\"",[11,29765,29766,29768],{},[30,29767,29754],{}," looks like a really interesting option, but it is currently in public preview so I have not decided to use it. I am using the AWSx library only for my VPC and associated resources, everything else uses the AWS Classic provider.",[11,29770,29771],{},"For CDK I use mostly L2 constructs and some L1 constructs.",[11,29773,29774,29775,29777],{},"Fot Terraform I use the VPC from the ",[30,29776,29732],{},", and everything else uses the AWS Terraform Provider.",[56,29779,29781],{"id":29780},"what-i-did-not-put-in-iac","What I did not put in IaC",[76,29783,29784,29787,29790],{},[79,29785,29786],{},"ECR (Elastic Container Registry)",[79,29788,29789],{},"ACM (Amazon Certificate Manager)",[79,29791,29792],{},"(Roles used for deployments)",[11,29794,29795,29796,187,29798,29800,29801,29804],{},"I created the Elastic Container Registry ",[30,29797,26811],{},[30,29799,26814],{}," repos manually in the AWS Console. I also manually requested an ACM certificate for ",[30,29802,29803],{},"*.mydomain.com"," for the domain that I use for testing that I purchased through Route53 domains.",[11,29806,29807],{},"I currently am using another less-than best practice of using Administrative Credentials stored in GitHub secrets. The better approach here is to make roles for different pipelines and use OIDC to authenticate instead of storing credentials. This is another good item for the backlog.",[56,29809,29811],{"id":29810},"tagging","Tagging",[76,29813,29814,29817,29820,29826,29832,29835],{},[79,29815,29816],{},"Terraform and CDK both make it easy to automatically tag all resources in a stack",[79,29818,29819],{},"It is possible to do this in Pulumi, but you need to write a little bit of code.",[79,29821,29822],{},[20,29823,29824],{"href":29824,"rel":29825},"https://www.pulumi.com/blog/automatically-enforcing-aws-resource-tagging-policies/",[24],[79,29827,29828],{},[20,29829,29830],{"href":29830,"rel":29831},"https://github.com/joeduffy/aws-tags-example/tree/master/autotag-ts",[24],[79,29833,29834],{},"Tagging is important since I look up resources by tag in GitHub Actions pipelines (for example, the Bastion Host is looked up by tag)",[79,29836,29837,29838,29843],{},"Automatically tagging resources works through ",[20,29839,29842],{"href":29840,"rel":29841},"https://www.pulumi.com/docs/intro/vs/terraform/",[24],"stack transformations"," are unique to Pulumi",[56,29845,29847],{"id":29846},"smoke-checking-application-environments","Smoke checking application environments",[11,29849,29850],{},"Here's the list of things I check when standing up an application environment:",[76,29852,29855,29864,29873,29882,29891,29900,29909,29918,29924,29930,29936,29942,29948,29954],{"className":29853},[29854],"contains-task-list",[79,29856,29859,29863],{"className":29857},[29858],"task-list-item",[29860,29861],"input",{"checked":609,"disabled":609,"type":29862},"checkbox"," Run the init/tsc, synth/plan/preview and deploy/apply/up commands successfully",[79,29865,29867,29869,29870,748],{"className":29866},[29858],[29860,29868],{"checked":609,"disabled":609,"type":29862}," Access the bastion host (",[30,29871,29872],{},"make aws-ssm-start-session",[79,29874,29876,29878,29879,748],{"className":29875},[29858],[29860,29877],{"checked":609,"disabled":609,"type":29862}," Run ECSExec to access a shell in a backend container (",[30,29880,29881],{},"make aws-ecs-exec",[79,29883,29885,29887,29888,748],{"className":29884},[29858],[29860,29886],{"checked":609,"disabled":609,"type":29862}," Test database connectivity (",[30,29889,29890],{},"python manage.py showmigrations",[79,29892,29894,29896,29897,748],{"className":29893},[29858],[29860,29895],{"checked":609,"disabled":609,"type":29862}," Run the migrations (",[30,29898,29899],{},"python manage.py migrate",[79,29901,29903,29905,29906,748],{"className":29902},[29858],[29860,29904],{"checked":609,"disabled":609,"type":29862}," Run collectstatic (",[30,29907,29908],{},"python manage.py collectstatic",[79,29910,29912,29914,29915,748],{"className":29911},[29858],[29860,29913],{"checked":609,"disabled":609,"type":29862}," Visit the site (",[30,29916,29917],{},"alpha.example.com",[79,29919,29921,29923],{"className":29920},[29858],[29860,29922],{"checked":609,"disabled":609,"type":29862}," Publish a blog post",[79,29925,29927,29929],{"className":29926},[29858],[29860,29928],{"checked":609,"disabled":609,"type":29862}," Publish a blog post with an image",[79,29931,29933,29935],{"className":29932},[29858],[29860,29934],{"checked":609,"disabled":609,"type":29862}," Check celery worker logs for successfully complete scheduled tasks",[79,29937,29939,29941],{"className":29938},[29858],[29860,29940],{"checked":609,"disabled":609,"type":29862}," Trigger an autoscaling event by running k6 load tests against an environment",[79,29943,29945,29947],{"className":29944},[29858],[29860,29946],{"checked":609,"disabled":609,"type":29862}," Optionally deploy another backend or frontend image tag using the GitHub Actions pipelines for backend and frontend updates",[79,29949,29951,29953],{"className":29950},[29858],[29860,29952],{"checked":609,"disabled":609,"type":29862}," Destroy the app stack",[79,29955,29957,29959],{"className":29956},[29858],[29860,29958],{"checked":609,"disabled":609,"type":29862}," Destroy the base stack",[56,29961,29963],{"id":29962},"backlog-and-next-steps","Backlog and next steps",[11,29965,29966],{},"Here are some of the next things I'll be working on in these project, roughly in order of importance:",[76,29968,29970,29976,29979,29982,29988,29991,29998,30001,30004,30007,30010,30013,30016,30019,30022],{"className":29969},[29854],[79,29971,29973,29975],{"className":29972},[29858],[29860,29974],{"checked":609,"disabled":609,"type":29862}," Introduce manual approvals in GitHub Actions for all deployments and allow for the previewing or \"planning\" before proceeding with an live operations in infrastructure pipelines",[79,29977,29978],{},"Switch to using OIDC for AWS authentication from GitHub Actions and remove AWS secrets from GitHub",[79,29980,29981],{},"Show how to do account isolation (different accounts for prod vs pre-prod environments)",[79,29983,29984,29985,29987],{},"GitHub Actions deployment pipeline for publishing ",[30,29986,26119],{}," package",[79,29989,29990],{},"Complete all GitHub Action deployment pipelines for base and app stacks (both ad hoc and prod)",[79,29992,29993,29994,29997],{},"For Pulumi and Terraform, use a Secrets Manager secret for the database instead of hardcoding it. Use the ",[30,29995,29996],{},"random"," functions to do this",[79,29999,30000],{},"Refactor GitHub Actions and make them reusable across different projects",[79,30002,30003],{},"Writing tests for Pulumi and CDK. Figure out how to write tests for Terraform modules",[79,30005,30006],{},"Use graviton instances and have the option to select between different architectures",[79,30008,30009],{},"Standardize all resources names across CDK, Terraform and Pulumi",[79,30011,30012],{},"The Pulumi components that define the resources associated with each ECS service are not very dry",[79,30014,30015],{},"Interfaces could be constructed with inheritance (base set of properties that is extended for different types of services)",[79,30017,30018],{},"Fix the CDK issue with priority rule on ALB listeners. I need to used a custom resource for this which is currently a WIP. Terraform and Pulumi look up the next highest listener rule priority under the hood, so you are not required to provide it, but CDK requires it, which means that you can't do ad hoc environments in CDK without a custom resource that looks up what the next available priority number is.",[79,30020,30021],{},"Make all three of the libraries less opinionated. For example, the celery worker and scheduler should be optional and the frontend component should also be optional",[79,30023,30024],{},"experiment with using a frontend with SSR. This is supported by Quasar, the framework I'm currently using to build my frontend SPA site",[11,30026,30027],{},"If you want to get involved or help with any of the above, please let me know!",[56,30029,14265],{"id":30030},"conclusion",[11,30032,30033,30034,30039],{},"I first started out with IaC following this project ",[20,30035,30038],{"href":30036,"rel":30037},"https://github.com/aws-samples/ecs-refarch-cloudformation",[24],"aws-samples/ecs-refarch-cloudformation"," (which is pretty old at this point) and wrote a lot of CloudFormation by hand. The pain of doing that lead me to explore the CDK with Python. I learned TypeScript by rewriting the Python CDK code I wrote in TypeScript. I later worked with a team that was more experienced in Terraform and learned how to use that. I feel like Pulumi takes the best of the two tools and has a really great developer experience. There is a little bit of a learning curve with Pulumi, and you give up some of the simplicity of Terraform.",[589,30041,30042],{},"html pre.shiki code .sq6CD, html code.shiki .sq6CD{--shiki-default:#D73A49;--shiki-default-font-style:inherit;--shiki-dark:#F97583;--shiki-dark-font-style:inherit;--shiki-sepia:#66D9EF;--shiki-sepia-font-style:italic}html pre.shiki code .sXSQT, html code.shiki .sXSQT{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#F8F8F2}html pre.shiki code .sC2Qs, html code.shiki .sC2Qs{--shiki-default:#D73A49;--shiki-dark:#F97583;--shiki-sepia:#F92672}html pre.shiki code .srTi1, html code.shiki .srTi1{--shiki-default:#6F42C1;--shiki-dark:#B392F0;--shiki-sepia:#A6E22E}html pre.shiki code .sMOD_, html code.shiki .sMOD_{--shiki-default:#24292E;--shiki-dark:#E1E4E8;--shiki-sepia:#F8F8F2}html pre.shiki code .sstjo, html code.shiki .sstjo{--shiki-default:#032F62;--shiki-dark:#9ECBFF;--shiki-sepia:#E6DB74}html pre.shiki code .s7F3e, html code.shiki .s7F3e{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#AE81FF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html .sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html.sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html pre.shiki code .s8-w5, html code.shiki .s8-w5{--shiki-default:#6A737D;--shiki-dark:#6A737D;--shiki-sepia:#88846F}html pre.shiki code .s1EfO, html code.shiki .s1EfO{--shiki-default:#032F62;--shiki-dark:#9ECBFF;--shiki-sepia:#F92672}html pre.shiki code .sinWB, html code.shiki .sinWB{--shiki-default:#032F62;--shiki-dark:#9ECBFF;--shiki-sepia:#F8F8F2}html pre.shiki code .sdpu8, html code.shiki .sdpu8{--shiki-default:#005CC5;--shiki-default-font-style:inherit;--shiki-dark:#79B8FF;--shiki-dark-font-style:inherit;--shiki-sepia:#FD971F;--shiki-sepia-font-style:italic}html pre.shiki code .sP7S_, html code.shiki .sP7S_{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#FD971F}html pre.shiki code .s5clZ, html code.shiki .s5clZ{--shiki-default:#22863A;--shiki-dark:#85E89D;--shiki-sepia:#F92672}html pre.shiki code .sz2Vg, html code.shiki .sz2Vg{--shiki-default:#6F42C1;--shiki-default-text-decoration:inherit;--shiki-dark:#B392F0;--shiki-dark-text-decoration:inherit;--shiki-sepia:#A6E22E;--shiki-sepia-text-decoration:underline}html pre.shiki code .s-m8C, html code.shiki .s-m8C{--shiki-default:#005CC5;--shiki-default-font-style:inherit;--shiki-dark:#79B8FF;--shiki-dark-font-style:inherit;--shiki-sepia:#66D9EF;--shiki-sepia-font-style:italic}html pre.shiki code .sOrwc, html code.shiki .sOrwc{--shiki-default:#E36209;--shiki-dark:#FFAB70;--shiki-sepia:#F8F8F2}",{"title":464,"searchDepth":488,"depth":488,"links":30044},[30045,30046,30047,30048,30053,30061,30062,30065,30066,30076,30086,30087,30088,30089,30090,30091,30092,30093,30094,30095,30096,30097],{"id":16115,"depth":488,"text":16116},{"id":25802,"depth":488,"text":25802},{"id":25822,"depth":488,"text":25823},{"id":25968,"depth":488,"text":25969,"children":30049},[30050,30051,30052],{"id":25972,"depth":500,"text":25973},{"id":25998,"depth":500,"text":25999},{"id":26012,"depth":500,"text":26012},{"id":26086,"depth":488,"text":26087,"children":30054},[30055,30056,30057,30058,30059,30060],{"id":26122,"depth":500,"text":26123},{"id":26162,"depth":500,"text":26163},{"id":26228,"depth":500,"text":26229},{"id":26427,"depth":500,"text":26428},{"id":26456,"depth":500,"text":26457},{"id":26536,"depth":500,"text":26537},{"id":26582,"depth":488,"text":26583},{"id":26612,"depth":488,"text":26612,"children":30063},[30064],{"id":26627,"depth":500,"text":26628},{"id":26740,"depth":488,"text":26741},{"id":26829,"depth":488,"text":26830,"children":30067},[30068,30069,30070,30071,30072,30073,30074,30075],{"id":26862,"depth":500,"text":26863},{"id":26879,"depth":500,"text":26880},{"id":26897,"depth":500,"text":26898},{"id":26935,"depth":500,"text":26850},{"id":27043,"depth":500,"text":27044},{"id":27070,"depth":500,"text":27071},{"id":27232,"depth":500,"text":26859},{"id":27279,"depth":500,"text":27280},{"id":27331,"depth":488,"text":27332,"children":30077},[30078,30079,30080,30081,30082,30083,30084,30085],{"id":27347,"depth":500,"text":27348},{"id":27375,"depth":500,"text":27376},{"id":27482,"depth":500,"text":27483},{"id":27500,"depth":500,"text":27501},{"id":27510,"depth":500,"text":27511},{"id":27556,"depth":500,"text":27557},{"id":27577,"depth":500,"text":27578},{"id":27940,"depth":500,"text":27328},{"id":29059,"depth":488,"text":29060},{"id":29099,"depth":488,"text":29100},{"id":29150,"depth":488,"text":29151},{"id":29583,"depth":488,"text":29584},{"id":29614,"depth":488,"text":29615},{"id":29687,"depth":488,"text":29688},{"id":29705,"depth":488,"text":29706},{"id":29780,"depth":488,"text":29781},{"id":29810,"depth":488,"text":29811},{"id":29846,"depth":488,"text":29847},{"id":29962,"depth":488,"text":29963},{"id":30030,"depth":488,"text":14265},"2023-01-07","Three reusable infrastructure as code libraries for abstracting containerized web app architecture on AWS ECS",[30101,30103,30105,30107,30110,30113],{"link":30102,"site":25725},"https://news.ycombinator.com/item?id=34291336",{"link":30104,"site":23769},"https://www.reddit.com/r/aws/comments/105vo53/my_infrastructure_as_code_rosetta_stone_deploying/",{"link":30106,"site":10715},"https://dev.to/briancaffey/my-infrastructure-as-code-rosetta-stone-deploying-the-same-web-application-on-aws-ecs-fargate-with-cdk-terraform-and-pulumi-oe4",{"link":30108,"site":30109},"https://medium.com/@briancaffey/my-infrastructure-as-code-rosetta-stone-with-cdk-terraform-and-pulumi-44fcb8233e6a","medium",{"link":30111,"site":30112},"https://briancaffey.hashnode.dev/setting-up-ad-hoc-development-environments-for-django-applications-with-aws-ecs-terraform-and-github-actions","hashnode",{"link":30114,"site":30115},"https://briancaffey.substack.com/p/my-infrastructure-as-code-rosetta","substack","/static/iac_rosetta_stone_og_image.png",{},"/2023/01/07/i-deployed-the-same-containerized-serverless-django-app-with-aws-cdk-terraform-and-pulumi",{"title":25742,"description":30099},"2023/01/07/i-deployed-the-same-containerized-serverless-django-app-with-aws-cdk-terraform-and-pulumi",[30122,30123,27951,10648,30124,30125,27264,30126,30127,30128,30129],"django","cdk","cloudformation","github-actions","ecs","fargate","containers","docker","2IlIcpBQ8QRwIfMAy_ouOLNEi3DznmeWO-GbImGUk4Q",{"id":30132,"title":30133,"body":30134,"comments":609,"date":33572,"description":33573,"draft":602,"extension":605,"external":33574,"image":30740,"meta":33592,"navigation":609,"path":33593,"seo":33594,"stem":33595,"tags":33596,"__hash__":33597},"blog/2022/03/27/ad-hoc-developer-environments-for-django-with-aws-ecs-terraform-and-github-actions.md","Setting up ad hoc development environments for Django applications with AWS ECS, Terraform and GitHub Actions",{"type":8,"value":30135,"toc":33501},[30136,30138,30146,30150,30153,30219,30223,30226,30249,30252,30287,30291,30294,30298,30301,30321,30324,30328,30331,30352,30358,30361,30364,30418,30421,30425,30428,30439,30442,30456,30462,30465,30469,30472,30483,30486,30489,30492,30495,30503,30506,30509,30513,30516,30529,30550,30554,30557,30568,30571,30574,30577,30588,30603,30607,30610,30613,30634,30637,30668,30671,30695,30700,30714,30723,30727,30736,30741,30745,30753,30756,30759,30762,30765,30768,30771,30774,30777,30781,30822,30826,30829,30831,30839,30843,30846,30853,30856,30862,30864,30867,30881,30885,30888,30892,30895,30898,30909,30912,30914,30917,30921,30924,30926,30929,30933,30936,30939,30955,30958,30968,30972,30975,30979,30997,31011,31014,31025,31037,31041,31047,31053,31059,31076,31080,31102,31109,31115,31133,31143,31147,31153,31164,31171,31175,31178,31221,31224,31229,31232,31235,32876,32886,32894,32919,32949,32953,32959,32985,32989,32992,32996,32999,33006,33013,33017,33030,33034,33037,33041,33044,33048,33051,33055,33058,33129,33132,33136,33154,33158,33179,33183,33189,33207,33213,33217,33227,33333,33336,33342,33346,33358,33362,33370,33373,33377,33381,33384,33388,33391,33394,33397,33400,33404,33418,33422,33425,33429,33432,33436,33439,33443,33446,33450,33459,33463,33466,33470,33473,33490,33493,33495,33498],[56,30137,16116],{"id":16115},[11,30139,30140,30141,30145],{},"This article will show how software development teams can build on-demand instances of a web application project for dog-food testing, quality review, internal and external demos and other use cases that require short-lived but feature-complete environments. It will focus on the technical implementation of building ad hoc environments using a specific set of tools (including AWS ECS, Terraform and GitHub Actions). I will also be giving context on high-level implementation decisions based on what I think are best practices guided by the ",[20,30142,30144],{"href":25954,"rel":30143},[24],"12-Factor Application methodology",". If any of this interests you, please have a read and let me know what you think in the comments on the outlets where I'll be sharing this article (links at the end).",[56,30147,30149],{"id":30148},"github-links","GitHub Links",[11,30151,30152],{},"This article references three open-source code repositories on GitHub.",[76,30154,30155,30174,30196],{},[79,30156,30157,30160],{},[20,30158,26634],{"href":25797,"rel":30159},[24],[76,30161,30162,30165,30168,30171],{},[79,30163,30164],{},"this repo contains an example microblogging application called μblog built with Django",[79,30166,30167],{},"the same application is implemented as a traditional Model Template View (MTV) site, a decoupled REST API and Javascript web application and a GraphQL API",[79,30169,30170],{},"it is a monorepo that also includes a frontend Vue.js application, CI/CD pipelines, a VuePress documentation site as well as tooling and instructions for settings up a local development environments (both with and without docker)",[79,30172,30173],{},"it includes a complete set of GitHub Action examples for automating the processes of creating, updating and destroying ad hoc environments that will be an important part of what is covered in this article",[79,30175,30176,30179],{},[20,30177,26110],{"href":25769,"rel":30178},[24],[76,30180,30181,30184,30187],{},[79,30182,30183],{},"a collection of modules for running Django applications on AWS using Terraform",[79,30185,30186],{},"one of the submodules can be used for creating ad hoc environments which will be what we use to create ad hoc environments",[79,30188,30189,30190,29626,30193,30195],{},"this module has been published to Terraform Registry and is used in the ",[30,30191,30192],{},"terraform/live/ad-hoc",[30,30194,26634],{}," repo",[79,30197,30198,30203],{},[20,30199,30202],{"href":30200,"rel":30201},"https://github.com/briancaffey/terraform-aws-ad-hoc-environments",[24],"terraform-aws-ad-hoc-environments",[76,30204,30205,30208,30213],{},[79,30206,30207],{},"a Terraform module that provides shared infrastructure used by ad hoc environments (including VPC, RDS instance, bastion host, security groups and IAM roles, etc.)",[79,30209,30210,30211],{},"this module has also been published to Terraform Registry and is also used in ",[30,30212,26634],{},[79,30214,30215,30216,30218],{},"this module is designed to be used with the ",[30,30217,26110],{}," Terraform module",[56,30220,30222],{"id":30221},"assumptions","Assumptions",[11,30224,30225],{},"There are all sorts of applications, and all sort of engineering teams. For some context on what I'm describing in this article, here are some basic assumptions that I'm making about the type of engineering team and software application product that would be a good fit for this type of development workflow.",[76,30227,30228,30231,30234,30237,30240,30243,30246],{},[79,30229,30230],{},"engineering team is composed of a backend team, a frontend team, a devops team and works closely with a product team",[79,30232,30233],{},"backend team primarily develops a REST API",[79,30235,30236],{},"frontend team develops a JavaScript SPA (frontend website)",[79,30238,30239],{},"SPA consumes backend REST API",[79,30241,30242],{},"product team frequently needs to demo applications to prospective clients",[79,30244,30245],{},"development teams don't have deep expertise in infrastructure, containers, CI/CD or automation",[79,30247,30248],{},"devops team has been tasked with building automation that will allow anyone on the team to quickly spin up a complete environment for testing and demoing purposes within minutes",[11,30250,30251],{},"Here are assumptions about specific tools and technologies used at the company:",[76,30253,30254,30257,30260,30263,30266,30269,30272,30275,30278,30281,30284],{},[79,30255,30256],{},"backend is a REST API developed with Django and a Postgres database",[79,30258,30259],{},"backend is packaged into a docker container",[79,30261,30262],{},"frontend is also packaged into a docker container using multi-stage builds and NGINX",[79,30264,30265],{},"frontend does not require any build-time configuration (all configuration needed by frontend is fetched from backend)",[79,30267,30268],{},"backend application's configuration is driven by plain-text environment variables at run-time",[79,30270,30271],{},"engineering team uses AWS",[79,30273,30274],{},"automation pipeline exists for building, tagging and pushing backend and frontend container images to an ECR repository",[79,30276,30277],{},"devops team uses AWS ECS for running containerized workloads",[79,30279,30280],{},"devops team uses Terraform for provisioning infrastructure",[79,30282,30283],{},"devops team uses GitHub Actions for building automation pipelines",[79,30285,30286],{},"team is somewhat cost-conscious",[56,30288,30290],{"id":30289},"what-are-ad-hoc-environments","What are ad hoc environments?",[11,30292,30293],{},"Ad hoc environments are short-lived environments that are designed to be used for testing a specific set of features or for demoing a specific application configuration in an isolated environment. It is intended to be a functional duplicate of the main production environment. An ad hoc environment is the first cloud environment that the application code will be deployed to after a developer has been working on it in a local development environment.",[56,30295,30297],{"id":30296},"trade-offs-to-make-when-designing-ad-hoc-environment-infrastructure-and-automation","Trade-offs to make when designing ad hoc environment infrastructure and automation",[11,30299,30300],{},"Now that we have a sense of what we are building and the team we are working with, let's think about the high-level trade-offs that we will face as we build a solution for providing on-demand ad hoc environments. When building infrastructure and workflows for ad hoc environments, there are a few things to solve for:",[76,30302,30303,30306,30309,30312,30315,30318],{},[79,30304,30305],{},"simplicity of the end-user interface and process for requesting an ad hoc environment",[79,30307,30308],{},"startup speed",[79,30310,30311],{},"total cost of ownership",[79,30313,30314],{},"degree of similarity to production environments",[79,30316,30317],{},"shared vs isolated resources",[79,30319,30320],{},"automation complexity",[11,30322,30323],{},"Let's look at these items by considering how we can set up the main infrastructure components that will be used to run our ad hoc application environments.",[736,30325,30327],{"id":30326},"relational-databases","Relational Databases",[11,30329,30330],{},"Startup speed can be measured by the time between when an environment is requested and when that environment can be used by whoever requested it. In this period of time, an automation pipeline may do some of the following:",[76,30332,30333,30343,30346,30349],{},[79,30334,10029,30335,106,30338,187,30340,30342],{},[30,30336,30337],{},"terraform init",[30,30339,26037],{},[30,30341,26041],{}," to build infrastructure",[79,30344,30345],{},"run scripts to prepare the application such as database migrations",[79,30347,30348],{},"seeding initial sample data with a script or database dump",[79,30350,30351],{},"message the user with information about the environment (URLs, commands for accessing an interactive shell, etc.)",[11,30353,30354,30355,30357],{},"RDS instances can take a long time to create relative to other AWS resources such as S3 buckets and IAM roles. RDS instances are also more costly than other resources. We could use a single, shared RDS instance placed in a private subnet of a shared VPC. Each ad hoc environment could use a different named database in the RDS instance in the form ",[30,30356,27243],{},". Using one RDS instance per ad hoc environment would be slow to startup and tear down and also costly if there are many developers using ad hoc environments simultaneously.",[11,30359,30360],{},"If we choose to isolate the application's relational database at the database level (and not the RDS instance level), then we will need our automation workflow to create a database per ad hoc environment.",[11,30362,30363],{},"Let's spin up a simple example to illustrate how this would would work.",[76,30365,30366,30373,30378,30383,30394,30397,30403,30409],{},[79,30367,30368,30369,30372],{},"A developer is working on ",[30,30370,30371],{},"feature-abc"," that involves a significant refactor of the data model.",[79,30374,30375,30376,643],{},"The developer decides to spin up an ad hoc environment called ",[30,30377,30371],{},[79,30379,30380,30381,643],{},"Our automation will need to create a database in the RDS instance called ",[30,30382,30371],{},[79,30384,30385,30386,30389,30390,30393],{},"We can configure a bastion host with ",[30,30387,30388],{},"psql"," installed that has network and security group access to our RDS instance, and we can give our GitHub Actions an SSH key that can be used to run ",[30,30391,30392],{},"createdb"," over SSH.",[79,30395,30396],{},"The automation also runs database migrations once the application has started, and we can view the logs of the database migration to check for any irregularities or other issues.",[79,30398,30399,30400,30402],{},"This will give the developer and the rest of team confidence that the promoting ",[30,30401,30371],{}," to the next pre-production environments will not have any errors.",[79,30404,30405,30406,30408],{},"The developer may even choose to load a SQL dump of the next pre-production environment into their ",[30,30407,30371],{}," database get even more confidence that there will be no data integrity errors.",[79,30410,30411,30412,30414,30415,30417],{},"When the developer's PR is merged and approved, the ad hoc environment ",[30,30413,30371],{}," can be destroyed, including the ",[30,30416,30371],{}," database in the shared RDS instance.",[11,30419,30420],{},"With this approach we won't incur the costs of multiple RDS instances. Ad hoc environments will start up faster because an RDS instance per environment is not required. We do have slightly less resource isolation, and we need to introduce a bastion host, but I consider this an acceptable trade-off.",[736,30422,30424],{"id":30423},"redis-key-value-database","Redis (key-value database)",[11,30426,30427],{},"Redis is another database used in the application and it plays a few different roles:",[76,30429,30430,30433,30436],{},[79,30431,30432],{},"primarily, it is a caching layer that can cache request responses to reduce load on the database and speed up our application",[79,30434,30435],{},"it is a message broker for our async task workers (celery)",[79,30437,30438],{},"it can be used as a backend for other 3rd party Django apps that our main application may need to use (such as django-constance, cache-ops, django-channels, etc.)",[11,30440,30441],{},"AWS offers a managed Redis service called ElastiCache. Redis running on an ElastiCache instance can do database isolation similar to how Postgres running on RDS can do database isolation as we discussed previously, but there are some key differences:",[76,30443,30444,30447],{},[79,30445,30446],{},"redis databases are numbered, not named",[79,30448,30449,30450,30452,30453,30455],{},"the backend application uses isolated numbered databases for the different 3rd party apps that I just mentioned (for example: celery can use database ",[30,30451,9181],{},",  API caching layer can use database ",[30,30454,6760],{},", etc.)",[11,30457,30458,30459,30461],{},"This makes it difficult to use a single ElastiCache instance for our ad hoc environments since we would need to figure out which numbered database to assign to a specific role for each ad hoc environment (e.g. how do we know which numbered database to use for the API caching for the ",[30,30460,30371],{}," ad hoc environment).",[11,30463,30464],{},"So how can we approach providing isolated redis instances for multiple ad hoc environments? Spoiler: my solution is to run redis as a stateful service in ECS. Before we dig into how to do this, we need to talk about another important part of our application: compute.",[736,30466,30468],{"id":30467},"compute","Compute",[11,30470,30471],{},"Our backend application is composed of a few different services that all share the same code base. In other words, our backend's services uses the same docker image but run different processes for each component:",[76,30473,30474,30477,30480],{},[79,30475,30476],{},"gunicorn for the core API",[79,30478,30479],{},"celery for the task workers",[79,30481,30482],{},"celerybeat for task scheduling",[11,30484,30485],{},"If our application used websockets, we could have another service that runs an asgi server process (like daphne or uvicorn).",[11,30487,30488],{},"Since our backend application is packaged into a container and we are using AWS as our cloud provider, ECS is a great choice for running our backend services. ECS is a container orchestration tool that I usually describe as a nice middle ground between docker swarm and Kubernetes. Simply put, it is a flexible option for running our containerized services that make up our backend application.",[11,30490,30491],{},"With ECS you can choose to run containers directly on EC2 instances that you manage, or you can run containers using Fargate. Fargate is a serverless compute option that takes care of managing both the underlying \"computer\" and operating system that run our application's containers. All of our backend dependencies are defined in our Dockerfile, so we do not to maintain or update the underlying operating system that runs our containers -- AWS handles all of this for us. To use Fargate, we simply tell AWS which containers to run and how much CPU and memory to use in the ECS Task that runs the containers. To scale our app horizontally, the ECS service that managed ECS tasks simply increases the number of tasks that run.",[11,30493,30494],{},"Since we are going to use the Fargate launch type for our ECS Tasks, let's talk about the ergonomics of these serverless compute instances compared to running our services directly on an EC2 instances.",[76,30496,30497,30500],{},[79,30498,30499],{},"We can't SSH into Fargate compute instances. We can instead use AWS Systems Manager and EcsExec to open an interactive shell in a running backend container. This can be useful for developers who might need to run a management command or access an interactive Django shell to verify behavior in their ad hoc environment.",[79,30501,30502],{},"We can't simply change code on the server and restart services. This can sometimes be a useful pattern for debugging something that can only be tested on a cloud environment (e.g. something that can't easily be reproduced on your local machine), so this requires that developers push new images to their backend services for every change they want to see reflected on their ad hoc environment. Later on I'll discuss how we can provide tooling for developers to quickly update the image used in their backend services.",[11,30504,30505],{},"With AWS Fargate, you will pay more than you would for a comparable amount of CPU and memory on EC2 instances. Similar to EC2 spot instances, Fargate offers interruptable instances called Fargate Spot which costs significantly less than regular Fargate instances. Fargate spot is appropriate for our ad hoc environments since ad hoc environments are non-critical workloads. In the event that a Fargate spot instance is interrupted, the ECS service will automatically launch another Fargate task to replace the task that was stopped.",[11,30507,30508],{},"In my opinion, ECS with Fargate is ideal for running the stateless services that make up our backend application. In terms of parity with our production environment, we can keep almost everything the same, except use regular Fargate instances instead of Fargate spot instances.",[736,30510,30512],{"id":30511},"redis-revisited","Redis, revisited",[11,30514,30515],{},"We can run redis as an ECS service instead of using ElastiCache. In order to do this, we will need our backend services (gunicorn, celery and celerybeat) to be able to communicate with a fourth ECS service that will be running redis (using an official redis image from Docker Hub, or a redis image that we define in ECR).",[11,30517,30518,30519,30522,30523,30525,30526,30528],{},"By default, ",[15,30520,30521],{},"there is no way for our backend services to know how to communicate with any other service in our ECS cluster",". If you have used docker-compose, you may know that you can use the service name ",[30,30524,27500],{}," in a backend service to easily communicate with a redis service called ",[30,30527,27500],{},". This networking convenience is not available to use out of the box with ECS. To achieve this in AWS, we need some way to manage a unique ad hoc environment-specific Route 53 DNS record that points to the private IP of the Fargate task that is running redis in an ECS cluster for a given ad hoc environment. Such a service exists in AWS and it is called Cloud Map. Cloud Map offers service discovery so that our backend services can make network calls to a static DNS address that will reliably point to the correct private IP of the ECS task running the redis container.",[11,30530,30531,30532,30534,30535,30538,30539,18952,30542,30545,30546,30549],{},"We can define a service discovery namespace (which will essentially be a top level domain, or TLD) that all of our ad hoc environments can share. Let's assume this namespace is called ",[30,30533,26466],{},". Each ad hoc environment can then define a service discovery service in the shared namespace for redis that is called ",[30,30536,30537],{},"{ad-hoc-env-name}-redis",". This way, we can have a reliable address that we can configure as an environment for our backend that will look like this: ",[30,30540,30541],{},"redis://{ad-hoc-env-name}-redis.ad-hoc:6379/0",[30,30543,30544],{},"{ad-hoc-env-name-redis}.ad-hoc"," will be the hostname of the redis service, and Route 53 will create records that point to ",[30,30547,30548],{},"{ad-hoc-env-name}-redis.ad-hoc"," to the private IP of the redis Fargate task for each ad hoc environment.",[736,30551,30553],{"id":30552},"load-balancing","Load Balancing",[11,30555,30556],{},"We now have our backend services (gunicorn, celery and celerybeat) running on Fargate spot instances, and these services can communicate with the redis service in our ad hoc environment's ECS cluster using service discovery that we configured with Cloud Map. We still need to think about a few things:",[76,30558,30559,30562,30565],{},[79,30560,30561],{},"how will we expose our API service to the public (or private) internet",[79,30563,30564],{},"how will we expose our frontend application to the public (or private) internet",[79,30566,30567],{},"how will we make sure that requests go the correct ECS services",[11,30569,30570],{},"Application load balancers (ALBs) are a great way to expose web app traffic to the internet. We could either have one application load balancer per ad hoc environment, or one application load balancer shared between all ad hoc environments. ALBs are somewhat slow to create and they also incur a significant monthly cost. They are also highly scalable, so using a shared ALB for all ad hoc environments would work.",[11,30572,30573],{},"Individual ad hoc environments can then create target groups and listener rules for a shared ALB for each service that needs to serve requests from the internet (the backend and the frontend). In our case this is the backend API server and the frontend server that serves our static frontend site using NGINX.",[11,30575,30576],{},"ECS services that need to be exposed to the internet can specify the target group, port and container to use for load balancing. A target group is created that defines the health check and other settings, and a load balancer listener rule is created on the shared load balancer that will forward traffic matching certain conditions to the target group for our service.",[11,30578,30579,30580,30583,30584,30587],{},"For a given ad hoc environment, we need to specify that only traffic with certain paths should be sent to the backend service, and all other traffic should be sent to the frontend service. For example, we may only want to send traffic that starts with the path ",[30,30581,30582],{},"/api"," or ",[30,30585,30586],{},"/admin"," to the backend target group, and all other traffic should be sent to the frontend target group. We can do this by setting conditions on the listener rules that forward traffic do the frontend and backend target groups based on the hostname and path.",[11,30589,30590,30591,106,30593,30595,30596,30598,30599,30602],{},"We want our listener rule logic to forward ",[30,30592,30582],{},[30,30594,30586],{}," and any other backend traffic to the backend target group, and forward all other traffic (",[30,30597,27531],{},") to the frontend target group. In order to do this, we need the backend listener rule to have a higher priority than the frontend listener rule for each ad hoc environment. Since we are using the same load balancer for all ad hoc environments, the priority values for each listener rule need to be unique. If we don't set the priority explicitly, then the priority will be set automatically to the next available value in ascending order. In order to make sure that the backend listener rule has a higher priority than the frontend listener rule for each ad hoc environment, we need to tell Terraform that the frontend module ",[30,30600,30601],{},"depends_on"," the backend module. This way the backend listener rule will have a higher priority (e.g. priority of 1) because it will be created first, and the frontend listener rule will have a lower priority (e.g. priority of 2).",[56,30604,30606],{"id":30605},"more-on-shared-resources-vs-per-environment-resources","More on shared resources vs per-environment resources",[11,30608,30609],{},"Up until now we have discussed infrastructure design decisions at a high level, but we have not yet talked about how to organize our infrastructure as code. At a basic level, components of our ad hoc environment either fall into shared infrastructure or infrastructure that is specific to an individual ad hoc environment. Here's a list of the resources that are shared and the resources that are specific to each ad hoc environment.",[11,30611,30612],{},"Shared resources include:",[76,30614,30615,30617,30620,30623,30626,30629,30631],{},[79,30616,26898],{},[79,30618,30619],{},"IAM policies",[79,30621,30622],{},"Security groups",[79,30624,30625],{},"RDS instance",[79,30627,30628],{},"Service Discovery namespace",[79,30630,27052],{},[79,30632,30633],{},"Bastion host",[11,30635,30636],{},"Ad hoc environment resources include:",[76,30638,30639,30641,30644,30647,30650,30653,30656,30662,30665],{},[79,30640,27348],{},[79,30642,30643],{},"ECS Tasks and Services (for backend and frontend applications)",[79,30645,30646],{},"ECS Tasks for running management commands (such as migrate)",[79,30648,30649],{},"CloudWatch logging groups for containers defined in ECS Tasks",[79,30651,30652],{},"ALB Target groups",[79,30654,30655],{},"ALB listener rules",[79,30657,30658,30659,748],{},"Route 53 record that points to the load balancer (e.g. ",[30,30660,30661],{},"ad-hoc-env-name.example.com",[79,30663,30664],{},"S3 bucket for static and media assets",[79,30666,30667],{},"Service Discovery Service for redis service in ECS cluster",[11,30669,30670],{},"Shared resources can be defined in one terraform configuration and deployed once. These resources will be long-lived as long as the application is under active development and the team requires on-demand provisioning of ad hoc environments.",[11,30672,30673,30674,30676,30677,30680,30681,106,30684,106,30687,30690,30691,30694],{},"Ad hoc environment resources can be defined in another terraform configuration that references outputs from the shared resource configuration using ",[30,30675,26752],{},". Each ad hoc environment can be defined by a ",[30,30678,30679],{},"\u003Cname>.tfvars"," file that contains the name of the ad hoc environment (such as ",[30,30682,30683],{},"brian",[30,30685,30686],{},"brian2",[30,30688,30689],{},"demo-feature-abc",", etc.). This ",[30,30692,30693],{},"\u003Cname>"," value will also be the name of the Terraform workspace and will be used to name and tag AWS resources associated with the corresponding ad hoc environment.",[11,30696,19225,30697,30699],{},[30,30698,30679],{}," file will allow developers to use a simple, standard file interface for defining application specific values, such as the version of the backend and frontend. This brings developers into the concepts and practices of \"infrastructure as code\" and \"configuration as code\" and also helps the entire team keep track of how different environments are configured.",[11,30701,30702,30703,30705,30706,30709,30710,643],{},"Ad hoc environment ",[30,30704,30679],{}," files are stored in a directory of a special git repository that also defines the ad hoc environment terraform configuration. Currently, the ",[30,30707,30708],{},"tfvars"," files are stored ",[20,30711,13074],{"href":30712,"rel":30713},"https://github.com/briancaffey/django-step-by-step/tree/main/terraform/live/ad-hoc/envs",[24],[11,30715,30716,30717,187,30720,643],{},"Now let's look at the two terraform configurations used for defining ",[15,30718,30719],{},"shared resources",[15,30721,30722],{},"ad hoc environment resources",[56,30724,30726],{"id":30725},"ad-hoc-environment-diagram","Ad Hoc Environment Diagram",[11,30728,30729,30730,29198,30733,643],{},"Here's an overview of the resources used for the ad hoc environments. The ",[15,30731,30732],{},"letters represent shared resources",[15,30734,30735],{},"numbers represent per-environment resources",[11,30737,30738],{},[2718,30739],{"alt":20386,"src":30740},"/static/adhoc.png",[736,30742,30744],{"id":30743},"shared-architecture","Shared architecture",[11,30746,30747,30748,748],{},"A. VPC (created using the ",[20,30749,30752],{"href":30750,"rel":30751},"https://registry.terraform.io/modules/terraform-aws-modules/vpc/aws/latest",[24],"official AWS VPC Module",[11,30754,30755],{},"B. Public subnets for bastion host, NAT Gateways and Load Balancer",[11,30757,30758],{},"C. Private subnets for application workloads and RDS",[11,30760,30761],{},"D. Application Load Balancer that is shared between all ad hoc environments. A pre-provisioned wildcard ACM certificate is attached to the load balancer that is used to secure traffic for load-balanced ECS services",[11,30763,30764],{},"E. Service discovery namespace that provides a namespace for application workloads to access the redis service running in ECS",[11,30766,30767],{},"F. IAM roles needed for ECS tasks to access AWS services",[11,30769,30770],{},"G. RDS instance using postgres engine that is shared between all ad hoc environments",[11,30772,30773],{},"H. Bastion host used to access RDS from GitHub Actions (needed for creating per-environment databases)",[11,30775,30776],{},"I. NAT Gateway used to give traffic in private subnets a route to the public internet",[736,30778,30780],{"id":30779},"environment-specific-architecture","Environment-specific architecture",[700,30782,30783,30786,30789,30792,30795,30798,30801,30804,30807,30812,30816,30819],{},[79,30784,30785],{},"ECS Cluster that groups all ECS tasks for a single ad hoc environment",[79,30787,30788],{},"Listener rules and target groups that direct traffic from the load balancer to the ECS services for an ad hoc environment.",[79,30790,30791],{},"Redis service running in ECS that provides caching and serves as a task broker for celery",[79,30793,30794],{},"Route53 records that point to the load balancer",[79,30796,30797],{},"Frontend service that serves the Vue.js application over NGINX",[79,30799,30800],{},"API service that serves the backend with Gunicorn",[79,30802,30803],{},"Celery worker that process jobs in the default queue",[79,30805,30806],{},"Celery beat that schedules celery tasks",[79,30808,30809,30811],{},[30,30810,27586],{}," task",[79,30813,30814,30811],{},[30,30815,27589],{},[79,30817,30818],{},"CloudWatch log groups are created for each ECS task in an ad hoc environment",[79,30820,30821],{},"Each ad hoc environment gets a database in the shared RDS instance",[56,30823,30825],{"id":30824},"shared-resources-terraform-configuration","Shared resources terraform configuration",[11,30827,30828],{},"Let's have a detailed look at the terraform configuration for shared resources that will support ad hoc environments.",[736,30830,26898],{"id":26897},[11,30832,30833,30834,30838],{},"We can use the ",[20,30835,30837],{"href":30750,"rel":30836},[24],"AWS VPC module"," for creating the shared VPC with Terraform. This module provides a high level interface that will provision lots of the components that are needed for a VPC following best practices, and it is less code for the DevOps team to manage compared to defining each component of a VPC (route tables, subnets, internet gateways, etc.).",[736,30840,30842],{"id":30841},"cloud-map-service-discovery-namespace","Cloud Map Service Discovery Namespace",[11,30844,30845],{},"Cloud Map is used in order to allow services in our ECS cluster to communicate with each other. The only reason that Cloud Map is needed is so that the backend services (API, celery workers, beat) can communicate with Redis, which will be an important service for our application, providing caching and also serving as a broker for celery. If we were to use Django Channels for websockets, the Redis service would also function as the backend for Django Channels.",[11,30847,30848,30849,30852],{},"We will only need to specify ",[30,30850,30851],{},"service_registries"," on the redis service in our ECS cluster. What this will do is provide an address that our other services can use to communicate with redis. This address is created in the form of a Route 53 record, and it points to the private IP address of the redis service. If the private IP of the redis service is updated, the Route 53 record record for our redis service will be updated as well.",[11,30854,30855],{},"In order for service discovery to work in the VPC that we created, we need to add the following options to the terraform AWS VPC module:",[459,30857,30860],{"className":30858,"code":30859,"language":997},[995],"# DNS settings\nenable_dns_hostnames = true\nenable_dns_support   = true\n",[30,30861,30859],{"__ignoreMap":464},[736,30863,26850],{"id":26935},[11,30865,30866],{},"There are two important security groups that we will set up as part of the shared infrastructure layer to be used by each ad hoc environment: one security group for the load balancer, and one security group where all of our ECS services will run.",[11,30868,30869,30870,187,30872,10480,30874,187,30877,30880],{},"The load balancer security group will allow all traffic on port ",[30,30871,27033],{},[30,30873,27007],{},[30,30875,30876],{},"HTTP",[30,30878,30879],{},"HTTPS"," traffic. The ECS security group will only allow inbound traffic from the application load balancer security group. It will also allow for traffic from port 6379 for redis traffic.",[736,30882,30884],{"id":30883},"iam-roles","IAM Roles",[11,30886,30887],{},"There are two important IAM roles that we will need for our ECS tasks. We need a task execution role that our ECS tasks will use to interact with other AWS services, such as S3, Secrets Manager, etc.",[736,30889,30891],{"id":30890},"rds-instance","RDS Instance",[11,30893,30894],{},"We will create one RDS instance in one of the private subnets in our VPC. This RDS instance will have one Postgres database per ad hoc environment. This RDS instance has a security group that allows all traffic from our ECS security group.",[736,30896,26853],{"id":30897},"load-balancer",[11,30899,30900,30901,30904,30905,30908],{},"We will use one load balancer for all ad hoc environments. This load balancer will have a wildcard ACM certificate attached to it (",[30,30902,30903],{},"*.dev.example.com",", for example). Each ad hoc environment will create a Route 53 record that will point to this load balancer's public DNS name. For example, ",[30,30906,30907],{},"brian.dev.example.com"," will be the address of my ad hoc environment. Requests to this address will then be routed to either the frontend ECS service or the backend ECS service depending on request header values and request path values that will be set on the listener rules.",[11,30910,30911],{},"By default, a load balancer supports up to 50 listener rules, so we can create plenty of ad hoc environments before we need to increase the default quota. There will be a discussion at the end of this article about AWS service quotas.",[736,30913,26859],{"id":27232},[11,30915,30916],{},"The bastion host will be created in one of the VPC's public subnets. This will primarily be used for connecting to RDS to create new databases for new ad hoc environments, or for manually manipulating data in an ad hoc environment for debugging.",[56,30918,30920],{"id":30919},"ad-hoc-environment-resources","Ad hoc environment resources",[11,30922,30923],{},"Now that we have defined a shared set of infrastructure that our ad hoc environments will use, let's have a look at the resources that will be specific to ad hoc environments that will be added on top of the shared resources.",[736,30925,27348],{"id":27347},[11,30927,30928],{},"The ECS Cluster is a simple grouping of ECS tasks and services.",[736,30930,30932],{"id":30931},"ecs-tasks-and-services","ECS Tasks and Services",[11,30934,30935],{},"Each environment will have a set of ECS tasks and services that will be used to run the application.",[11,30937,30938],{},"There are four important ECS services in our application that are used to run \"long-running\" ECS tasks. Long-running tasks are tasks that start processes that run indefinitely, rather than running until completion. The long-running tasks in our application include:",[76,30940,30941,30944,30947,30950,30953],{},[79,30942,30943],{},"backend web application (gunciron web server)",[79,30945,30946],{},"backend celery worker",[79,30948,30949],{},"backend celery beat",[79,30951,30952],{},"frontend web site (nginx web server)",[79,30954,27500],{},[11,30956,30957],{},"The infrastructure code also defines some tasks that are not long-running but rather short lived tasks that run until completion and do not start again. These tasks include:",[76,30959,30960,30962,30965],{},[79,30961,27586],{},[79,30963,30964],{},"database migrations",[79,30966,30967],{},"any other ad-hoc task that we want to run, usually wrapped in a Django management command",[56,30969,30971],{"id":30970},"how-to-setup-an-ad-hoc-environment","How to setup an ad hoc environment",[11,30973,30974],{},"Now that we have been over the resources that will be created to support our ad hoc environments, let's talk about how we can enable individuals on our team to create and update ad hoc environments.",[736,30976,30978],{"id":30977},"design-decisions","Design decisions",[11,30980,30981,30982,30985,30986,187,30988,30990,30991,30993,30994,30996],{},"The devops team will decide on the interface that will be used for creating an ad hoc environment. Since we are using Terraform, this interface will be a Terraform configuration. The minimum amount of information that our ad hoc environment configuration needs is image tags for the frontend and backend images to use. Other configurations will be provided by default values set in ",[30,30983,30984],{},"variables.tf",", and these defaults can easily be overridden by passing values to ",[30,30987,26037],{},[30,30989,26041],{},". I'm choosing to use ",[30,30992,30679],{}," as the way to pass configuration values to our ad hoc environments where ",[30,30995,30693],{}," is the name of the ad hoc environment being created. This will give us the following benefits:",[76,30998,30999,31005],{},[79,31000,31001,31002,31004],{},"all ad hoc environments will be visible to the entire team in git since each ad hoc environment will have a ",[30,31003,30679],{}," file associated with it",[79,31006,31007,31008],{},"adding additional customization to an ad hoc environment does not add additional complexity to our automation pipeline since all customization is added through a single file that will be referenced by ",[30,31009,31010],{},"$WORKSPACE.tfvars",[11,31012,31013],{},"The downsides of this approach are:",[76,31015,31016,31019],{},[79,31017,31018],{},"creating ad hoc environments requires knowledge of git, so non-technical product team members might need help from the engineering team when setting up an ad hoc environment",[79,31020,31021,31022,31024],{},"there is an additional \"manual\" step of creating a ",[30,31023,30679],{}," file that must be done before running a pipeline to create an ad hoc environment",[11,31026,31027,31028,31030,31031,31033,31034,31036],{},"Provided that a ",[30,31029,30679],{}," file has been created and pushed to the repo, creating or updating an ad hoc environment will be as simple as running a pipeline in GitHub Actions that specifies the ",[30,31032,30693],{}," of our ad hoc environment. If no such ",[30,31035,30679],{}," file exists, our pipeline will fail.",[736,31038,31040],{"id":31039},"github-action","GitHub Action",[11,31042,31043,31044,208],{},"Creating ad hoc environments will involve manually triggering a GitHub Action that runs on ",[30,31045,31046],{},"workflow_dispatch",[459,31048,31051],{"className":31049,"code":31050,"language":997},[995],"on:\n  workflow_dispatch:\n    inputs:\n      workspace:\n        description: 'Name of terraform workspace to use'\n        required: true\n        default: 'dev'\n        type: string\n",[30,31052,31050],{"__ignoreMap":464},[11,31054,31055,31056,31058],{},"We only have to enter the name of the ad hoc environment we want to create or update. The ad hoc environment name is used as the Terraform workspace name. This name is also the name of the ",[30,31057,30679],{}," file that must be created per environment.",[11,31060,31061,31062,106,31064,187,31066,31068,31069,31071,31072,31075],{},"This workflow will do ",[30,31063,30337],{},[30,31065,26037],{},[30,31067,26041],{}," using the ",[30,31070,30679],{}," file. When everything has been created, we will use the AWS CLI to prepare the environment so that it can be used. We will use the ",[30,31073,31074],{},"aws ecs run-task"," command to run database migrations needed so that the application code can make database queries.",[56,31077,31079],{"id":31078},"how-to-update-code-in-an-existing-ad-hoc-environment","How to update code in an existing ad hoc environment",[11,31081,31082,31083,31085,31086,31089,31090,31093,31094,31097,31098,31101],{},"Assuming that we have deployed an ad hoc environment called ",[30,31084,30683],{}," with version ",[30,31087,31088],{},"v1.0.0"," of the backend application and ",[30,31091,31092],{},"v2.0.0"," of the frontend application, let's think about the process of updating the application to ",[30,31095,31096],{},"v1.1.0"," of the backend and ",[30,31099,31100],{},"v2.1.0"," of the frontend.",[11,31103,31104,31105,31108],{},"The simplest approach to updating the application would be edit the ",[30,31106,31107],{},"brian.tfvars"," file with the new versions:",[459,31110,31113],{"className":31111,"code":31112,"language":997},[995],"# brian.tfvars\nbe_image_tag = \"v1.1.0\"\nfe_image_tag = \"v2.1.0\"\n",[30,31114,31112],{"__ignoreMap":464},[11,31116,31117,31118,106,31120,187,31122,31124,31125,31127,31128,13576],{},"If we run the same pipeline that we initially used to deploy ad hoc environment (with ",[30,31119,30337],{},[30,31121,26037],{},[30,31123,26041],{},") against the updated ",[30,31126,31107],{}," file, this will result in a rolling update of the frontend and backend services (",[20,31129,31132],{"href":31130,"rel":31131},"https://docs.aws.amazon.com/AmazonECS/latest/developerguide/deployment-type-ecs.html",[24],"more on rolling updates here",[11,31134,31135,31136,31138,31139,31142],{},"If there are database migrations included in the new version of the code that is going out, we need to run database migrations after the ",[30,31137,26041],{}," completes. We use a top level output from the ad hov environment terraform configuration that is a ",[30,31140,31141],{},"run-task"," command with all appropriate arguments that will run database migrations when called from GitHub Actions.",[736,31144,31146],{"id":31145},"order-of-operations","Order of Operations",[11,31148,31149,31150,643],{},"For ad hoc environments, it is probably fine to update the services and then run the database migrations. Ad hoc environments may only have a single \"user\" -- the developer, so ",[15,31151,31152],{},"we don't need to worry about any errors that may occur if requests are made against the new version of code before database migrations have been applied",[11,31154,31155,31156,31159,31160,31163],{},"Let's consider a simple example to illustrate what can go wrong here. If we add a ",[30,31157,31158],{},"total_views"," to our blog post model to track the total number of page views a post has, we would add a field to the model, generate migration file with ",[30,31161,31162],{},"makemigrations",", and then update our views and model serializers to make use of this new field. In the time between updating our service and running the database migrations, any requests to endpoints that access the new database field will fail since the table does not yet exist.",[11,31165,31166,31167,31170],{},"If we first run database migrations ",[15,31168,31169],{},"and then"," update application code (ECS services), then we can avoid errors about fields not existing. In our production application, we want to aim for fewer errors, so we should be using this \"order of operations\": first run new database migrations and then update application code.",[736,31172,31174],{"id":31173},"github-action-for-application-updates","GitHub Action for application updates",[11,31176,31177],{},"We need a GitHub Action that can do the following:",[76,31179,31180,31186,31192,31198,31203,31209,31215],{},[79,31181,31182,31183,748],{},"fetch the current container definition JSON files for each backend tasks (",[30,31184,31185],{},"aws ecs describe-task-definition",[79,31187,31188,31189,748],{},"write new container definitions JSON with the new backend image tag (using ",[30,31190,31191],{},"jq",[79,31193,31194,31195,748],{},"register new task definitions with the new container definition JSON files for each task (",[30,31196,31197],{},"aws ecs register-task-definition",[79,31199,31200,31201,748],{},"call run-task with the newly updated migration ECS task (",[30,31202,31074],{},[79,31204,31205,31206,748],{},"wait for the task to exit and display the logs (",[30,31207,31208],{},"aws ecs wait tasks-stopped",[79,31210,31211,31212,748],{},"update the backend services (gunicorn, celery, celery beat) (",[30,31213,31214],{},"aws ecs update-service",[79,31216,31217,31218,748],{},"wait for the new backend services to be stable (",[30,31219,31220],{},"aws ecs wait services-stable",[11,31222,31223],{},"Here's a visual representation of the backend update process:",[11,31225,31226],{},[2718,31227],{"alt":20386,"src":31228},"/static/adhoc/ad_hoc.backend_update.drawio.png",[11,31230,31231],{},"In order have the correct arguments for all of the AWS CLI calls used in the above workflow, we can use the AWS CLI to fetch resource names by tag.",[11,31233,31234],{},"Here is what I'm using for the script. There lots of comments, so please refer to those comments for an explanation of what the script is doing.",[459,31236,31238],{"className":461,"code":31237,"language":463,"meta":464,"style":464},"#!/bin/bash\n\n# This script will be called to update an ad hoc environment backend\n# with a new image tag. It will first run pre-update tasks (such as migrations)\n# and then do a rolling update of the backend services.\n\n# It is called from the ad_hock_backend_update.yml GitHub Actions file\n\n# Required environment variables that need to be exported before running this script:\n\n# WORKSPACE - ad hoc environment workspace\n# SHARED_RESOURCES_WORKSPACE - shared resources workspace\n# BACKEND_IMAGE_TAG - backend image tag to update services to (e.g. v1.2.3)\n# AWS_ACCOUNT_ID - AWS account ID is used for the ECR repository URL\n\necho \"Updating backend services...\"\n\n# first define a variable containing the new image URI\nNEW_BACKEND_IMAGE_URI=\"$AWS_ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com/backend:$BACKEND_IMAGE_TAG\"\n\n\n# register new task definitions\n# https://docs.aws.amazon.com/cli/latest/reference/ecs/describe-task-definition.html#description\nfor TASK in \"migrate\" \"gunicorn\" \"default\" \"beat\"\ndo\n  echo \"Updating $TASK task definition...\"\n\n  # in Terraform we name our tasks based on the ad hoc environment name\n  # (also the Terraform workspace name) and the name of the task\n  # (e.g. migrate, gunicorn, default, beat)\n  TASK_FAMILY=$WORKSPACE-$TASK\n\n  # save the task definition JSON to a variable\n  TASK_DESCRIPTION=$(aws ecs describe-task-definition \\\n    --task-definition $TASK_FAMILY \\\n  )\n\n  # save container definitions to a file for each task\n  echo $TASK_DESCRIPTION | jq -r \\\n    .taskDefinition.containerDefinitions \\\n    > /tmp/$TASK_FAMILY.json\n\n  # write new container definition JSON with updated image\n  echo \"Writing new $TASK_FAMILY container definitions JSON...\"\n\n  # replace old image URI with new image URI in a new container definitions JSON\n  cat /tmp/$TASK_FAMILY.json \\\n    | jq \\\n    --arg IMAGE \"$NEW_BACKEND_IMAGE_URI\" '.[0].image |= $IMAGE' \\\n    > /tmp/$TASK_FAMILY-new.json\n\n  # Get the existing configuration for the task definition (memory, cpu, etc.)\n  # from the variable that we saved the task definition JSON to earlier\n  echo \"Getting existing configuration for $TASK_FAMILY...\"\n\n  MEMORY=$( echo $TASK_DESCRIPTION | jq -r \\\n    .taskDefinition.memory \\\n  )\n\n  CPU=$( echo $TASK_DESCRIPTION | jq -r \\\n    .taskDefinition.cpu \\\n  )\n\n  ECS_EXECUTION_ROLE_ARN=$( echo $TASK_DESCRIPTION | jq -r \\\n    .taskDefinition.executionRoleArn \\\n  )\n\n  ECS_TASK_ROLE_ARN=$( echo $TASK_DESCRIPTION | jq -r \\\n    .taskDefinition.taskRoleArn \\\n  )\n\n  # check the content of the new container definition JSON\n  cat /tmp/$TASK_FAMILY-new.json\n\n  # register new task definition using the new container definitions\n  # and the values that we read off of the existing task definitions\n  echo \"Registering new $TASK_FAMILY task definition...\"\n\n  aws ecs register-task-definition \\\n    --family $TASK_FAMILY \\\n    --container-definitions file:///tmp/$TASK_FAMILY-new.json \\\n    --memory $MEMORY \\\n    --cpu $CPU \\\n    --network-mode awsvpc \\\n    --execution-role-arn $ECS_EXECUTION_ROLE_ARN \\\n    --task-role-arn $ECS_TASK_ROLE_ARN \\\n    --requires-compatibilities \"FARGATE\"\n\ndone\n\n# Now we need to run migrate, collectstatic and any other commands that need to be run\n# before doing a rolling update of the backend services\n\n# We will use the new task definitions we just created to run these commands\n\n# get the ARN of the most recent revision of the migrate task definition\nTASK_DEFINITION=$( \\\n  aws ecs describe-task-definition \\\n    --task-definition $WORKSPACE-migrate \\\n    | jq -r \\\n    .taskDefinition.taskDefinitionArn \\\n)\n\n# get private subnets as space separated string from shared resources VPC\nSUBNETS=$( \\\n  aws ec2 describe-subnets \\\n    --filters \"Name=tag:env,Values=$SHARED_RESOURCES_WORKSPACE\" \"Name=tag:Name,Values=*private*\" \\\n    --query 'Subnets[*].SubnetId' \\\n    --output text \\\n)\n\n# replace spaces with commas using tr\nSUBNET_IDS=$(echo $SUBNETS | tr ' ' ',')\n\n# https://github.com/aws/aws-cli/issues/5348\n# get ecs_sg_id - just a single value\nECS_SG_ID=$( \\\n  aws ec2 describe-security-groups \\\n    --filters \"Name=tag:Name,Values=$SHARED_RESOURCES_WORKSPACE-ecs-sg\" \\\n    --query 'SecurityGroups[*].GroupId' \\\n    --output text \\\n)\n\necho \"Running database migrations...\"\n\n# timestamp used for log retrieval (milliseconds after Jan 1, 1970 00:00:00 UTC)\nSTART_TIME=$(date +%s000)\n\n# run the migration task and capture the taskArn into a variable called TASK_ID\nTASK_ID=$( \\\n  aws ecs run-task \\\n    --cluster $WORKSPACE-cluster \\\n    --task-definition $TASK_DEFINITION \\\n    --network-configuration \"awsvpcConfiguration={subnets=[$SUBNET_IDS],securityGroups=[$ECS_SG_ID],assignPublicIp=ENABLED}\" \\\n    | jq -r '.tasks[0].taskArn' \\\n  )\n\necho \"Task ID is $TASK_ID\"\n\n# wait for the migrate task to exit\n# https://docs.aws.amazon.com/cli/latest/reference/ecs/wait/tasks-stopped.html#description\n# > It will poll every 6 seconds until a successful state has been reached.\n# > This will exit with a return code of 255 after 100 failed checks.\naws ecs wait tasks-stopped \\\n  --tasks $TASK_ID \\\n  --cluster $WORKSPACE-cluster\n\n# timestamp used for log retrieval (milliseconds after Jan 1, 1970 00:00:00 UTC)\nEND_TIME=$(date +%s000)\n\n# print the CloudWatch log events to STDOUT\naws logs get-log-events \\\n  --log-group-name \"/ecs/$WORKSPACE/migrate\" \\\n  --log-stream-name \"migrate/migrate/${TASK_ID##*/}\" \\\n  --start-time $START_TIME \\\n  --end-time $END_TIME \\\n  | jq -r '.events[].message'\n\necho \"Migrations complete. Starting rolling update for backend services...\"\n\n# update backend services\nfor TASK in \"gunicorn\" \"default\" \"beat\"\ndo\n\n  # get taskDefinitionArn for each service to be used in update-service command\n  # this will get the most recent revision of each task (the one that was just created)\n  # https://docs.aws.amazon.com/cli/latest/reference/ecs/describe-task-definition.html#description\n  TASK_DEFINITION=$( \\\n    aws ecs describe-task-definition \\\n      --task-definition $WORKSPACE-$TASK \\\n      | jq -r \\\n      .taskDefinition.taskDefinitionArn \\\n  )\n\n  # update each service with new task definintion\n  aws ecs update-service \\\n    --cluster $WORKSPACE-cluster \\\n    --service $WORKSPACE-$TASK \\\n    --task-definition $TASK_DEFINITION \\\n    --no-cli-pager\n\ndone\n\necho \"Services updated. Waiting for services to become stable...\"\n\n# wait for all service to be stable (runningCount == desiredCount for each service)\naws ecs wait services-stable \\\n  --cluster $WORKSPACE-cluster \\\n  --services $WORKSPACE-gunicorn $WORKSPACE-default $WORKSPACE-beat\n\necho \"Services are now stable. Backend services are now up to date with $BACKEND_IMAGE_TAG.\"\n\necho \"Backend update is now complete!\"\n",[30,31239,31240,31245,31249,31254,31259,31264,31268,31273,31277,31282,31286,31291,31296,31301,31306,31310,31317,31321,31326,31346,31350,31354,31359,31364,31385,31390,31404,31408,31413,31418,31423,31438,31442,31447,31467,31477,31482,31486,31491,31507,31514,31528,31532,31537,31549,31553,31558,31572,31581,31601,31612,31616,31621,31626,31638,31642,31664,31671,31675,31679,31700,31707,31711,31715,31736,31743,31747,31751,31772,31779,31783,31787,31792,31802,31806,31811,31816,31827,31831,31843,31852,31867,31877,31887,31897,31907,31917,31925,31929,31934,31938,31943,31948,31952,31958,31963,31969,31981,31992,32005,32016,32024,32029,32034,32040,32052,32065,32084,32095,32106,32110,32115,32121,32149,32154,32160,32166,32178,32190,32205,32215,32224,32229,32234,32242,32247,32253,32270,32275,32281,32293,32305,32318,32328,32350,32364,32369,32374,32387,32392,32398,32404,32410,32416,32431,32441,32452,32457,32462,32478,32483,32489,32501,32517,32536,32547,32558,32571,32576,32584,32589,32595,32610,32615,32620,32626,32632,32638,32650,32662,32677,32689,32697,32702,32707,32713,32725,32736,32750,32759,32765,32770,32775,32780,32788,32793,32799,32813,32824,32845,32850,32863,32868],{"__ignoreMap":464},[151,31241,31242],{"class":469,"line":470},[151,31243,31244],{"class":1527},"#!/bin/bash\n",[151,31246,31247],{"class":469,"line":488},[151,31248,1090],{"emptyLinePlaceholder":609},[151,31250,31251],{"class":469,"line":500},[151,31252,31253],{"class":1527},"# This script will be called to update an ad hoc environment backend\n",[151,31255,31256],{"class":469,"line":509},[151,31257,31258],{"class":1527},"# with a new image tag. It will first run pre-update tasks (such as migrations)\n",[151,31260,31261],{"class":469,"line":517},[151,31262,31263],{"class":1527},"# and then do a rolling update of the backend services.\n",[151,31265,31266],{"class":469,"line":534},[151,31267,1090],{"emptyLinePlaceholder":609},[151,31269,31270],{"class":469,"line":1413},[151,31271,31272],{"class":1527},"# It is called from the ad_hock_backend_update.yml GitHub Actions file\n",[151,31274,31275],{"class":469,"line":1418},[151,31276,1090],{"emptyLinePlaceholder":609},[151,31278,31279],{"class":469,"line":2462},[151,31280,31281],{"class":1527},"# Required environment variables that need to be exported before running this script:\n",[151,31283,31284],{"class":469,"line":2471},[151,31285,1090],{"emptyLinePlaceholder":609},[151,31287,31288],{"class":469,"line":2480},[151,31289,31290],{"class":1527},"# WORKSPACE - ad hoc environment workspace\n",[151,31292,31293],{"class":469,"line":2489},[151,31294,31295],{"class":1527},"# SHARED_RESOURCES_WORKSPACE - shared resources workspace\n",[151,31297,31298],{"class":469,"line":2497},[151,31299,31300],{"class":1527},"# BACKEND_IMAGE_TAG - backend image tag to update services to (e.g. v1.2.3)\n",[151,31302,31303],{"class":469,"line":3140},[151,31304,31305],{"class":1527},"# AWS_ACCOUNT_ID - AWS account ID is used for the ECR repository URL\n",[151,31307,31308],{"class":469,"line":3149},[151,31309,1090],{"emptyLinePlaceholder":609},[151,31311,31312,31314],{"class":469,"line":3158},[151,31313,412],{"class":2226},[151,31315,31316],{"class":481}," \"Updating backend services...\"\n",[151,31318,31319],{"class":469,"line":3167},[151,31320,1090],{"emptyLinePlaceholder":609},[151,31322,31323],{"class":469,"line":3175},[151,31324,31325],{"class":1527},"# first define a variable containing the new image URI\n",[151,31327,31328,31331,31333,31335,31338,31341,31344],{"class":469,"line":3184},[151,31329,31330],{"class":503},"NEW_BACKEND_IMAGE_URI",[151,31332,1876],{"class":1869},[151,31334,8592],{"class":481},[151,31336,31337],{"class":503},"$AWS_ACCOUNT_ID",[151,31339,31340],{"class":481},".dkr.ecr.us-east-1.amazonaws.com/backend:",[151,31342,31343],{"class":503},"$BACKEND_IMAGE_TAG",[151,31345,16406],{"class":481},[151,31347,31348],{"class":469,"line":3193},[151,31349,1090],{"emptyLinePlaceholder":609},[151,31351,31352],{"class":469,"line":3720},[151,31353,1090],{"emptyLinePlaceholder":609},[151,31355,31356],{"class":469,"line":3729},[151,31357,31358],{"class":1527},"# register new task definitions\n",[151,31360,31361],{"class":469,"line":3735},[151,31362,31363],{"class":1527},"# https://docs.aws.amazon.com/cli/latest/reference/ecs/describe-task-definition.html#description\n",[151,31365,31366,31368,31371,31373,31376,31379,31382],{"class":469,"line":3745},[151,31367,16732],{"class":1869},[151,31369,31370],{"class":503}," TASK ",[151,31372,16417],{"class":1869},[151,31374,31375],{"class":481}," \"migrate\"",[151,31377,31378],{"class":481}," \"gunicorn\"",[151,31380,31381],{"class":481}," \"default\"",[151,31383,31384],{"class":481}," \"beat\"\n",[151,31386,31387],{"class":469,"line":3754},[151,31388,31389],{"class":1869},"do\n",[151,31391,31392,31395,31398,31401],{"class":469,"line":3760},[151,31393,31394],{"class":2226},"  echo",[151,31396,31397],{"class":481}," \"Updating ",[151,31399,31400],{"class":503},"$TASK",[151,31402,31403],{"class":481}," task definition...\"\n",[151,31405,31406],{"class":469,"line":3773},[151,31407,1090],{"emptyLinePlaceholder":609},[151,31409,31410],{"class":469,"line":3782},[151,31411,31412],{"class":1527},"  # in Terraform we name our tasks based on the ad hoc environment name\n",[151,31414,31415],{"class":469,"line":3791},[151,31416,31417],{"class":1527},"  # (also the Terraform workspace name) and the name of the task\n",[151,31419,31420],{"class":469,"line":3803},[151,31421,31422],{"class":1527},"  # (e.g. migrate, gunicorn, default, beat)\n",[151,31424,31425,31428,31430,31433,31435],{"class":469,"line":3811},[151,31426,31427],{"class":503},"  TASK_FAMILY",[151,31429,1876],{"class":1869},[151,31431,31432],{"class":503},"$WORKSPACE",[151,31434,12445],{"class":481},[151,31436,31437],{"class":503},"$TASK\n",[151,31439,31440],{"class":469,"line":3820},[151,31441,1090],{"emptyLinePlaceholder":609},[151,31443,31444],{"class":469,"line":7084},[151,31445,31446],{"class":1527},"  # save the task definition JSON to a variable\n",[151,31448,31449,31452,31454,31457,31459,31462,31465],{"class":469,"line":7148},[151,31450,31451],{"class":503},"  TASK_DESCRIPTION",[151,31453,1876],{"class":1869},[151,31455,31456],{"class":503},"$(",[151,31458,27264],{"class":473},[151,31460,31461],{"class":481}," ecs",[151,31463,31464],{"class":481}," describe-task-definition",[151,31466,485],{"class":477},[151,31468,31469,31472,31475],{"class":469,"line":7211},[151,31470,31471],{"class":477},"    --task-definition",[151,31473,31474],{"class":503}," $TASK_FAMILY ",[151,31476,497],{"class":477},[151,31478,31479],{"class":469,"line":7273},[151,31480,31481],{"class":503},"  )\n",[151,31483,31484],{"class":469,"line":7335},[151,31485,1090],{"emptyLinePlaceholder":609},[151,31487,31488],{"class":469,"line":7398},[151,31489,31490],{"class":1527},"  # save container definitions to a file for each task\n",[151,31492,31493,31495,31498,31500,31503,31505],{"class":469,"line":7462},[151,31494,31394],{"class":2226},[151,31496,31497],{"class":503}," $TASK_DESCRIPTION ",[151,31499,3947],{"class":1869},[151,31501,31502],{"class":473}," jq",[151,31504,12915],{"class":477},[151,31506,485],{"class":477},[151,31508,31509,31512],{"class":469,"line":7467},[151,31510,31511],{"class":481},"    .taskDefinition.containerDefinitions",[151,31513,485],{"class":477},[151,31515,31516,31519,31522,31525],{"class":469,"line":7532},[151,31517,31518],{"class":1869},"    >",[151,31520,31521],{"class":481}," /tmp/",[151,31523,31524],{"class":503},"$TASK_FAMILY",[151,31526,31527],{"class":481},".json\n",[151,31529,31530],{"class":469,"line":7537},[151,31531,1090],{"emptyLinePlaceholder":609},[151,31533,31534],{"class":469,"line":7603},[151,31535,31536],{"class":1527},"  # write new container definition JSON with updated image\n",[151,31538,31539,31541,31544,31546],{"class":469,"line":7608},[151,31540,31394],{"class":2226},[151,31542,31543],{"class":481}," \"Writing new ",[151,31545,31524],{"class":503},[151,31547,31548],{"class":481}," container definitions JSON...\"\n",[151,31550,31551],{"class":469,"line":7673},[151,31552,1090],{"emptyLinePlaceholder":609},[151,31554,31555],{"class":469,"line":7678},[151,31556,31557],{"class":1527},"  # replace old image URI with new image URI in a new container definitions JSON\n",[151,31559,31560,31563,31565,31567,31570],{"class":469,"line":7708},[151,31561,31562],{"class":473},"  cat",[151,31564,31521],{"class":481},[151,31566,31524],{"class":503},[151,31568,31569],{"class":481},".json",[151,31571,485],{"class":477},[151,31573,31574,31577,31579],{"class":469,"line":7713},[151,31575,31576],{"class":1869},"    |",[151,31578,31502],{"class":473},[151,31580,485],{"class":477},[151,31582,31583,31586,31589,31591,31594,31596,31599],{"class":469,"line":7746},[151,31584,31585],{"class":477},"    --arg",[151,31587,31588],{"class":481}," IMAGE",[151,31590,16722],{"class":481},[151,31592,31593],{"class":503},"$NEW_BACKEND_IMAGE_URI",[151,31595,8592],{"class":481},[151,31597,31598],{"class":481}," '.[0].image |= $IMAGE'",[151,31600,485],{"class":477},[151,31602,31603,31605,31607,31609],{"class":469,"line":7751},[151,31604,31518],{"class":1869},[151,31606,31521],{"class":481},[151,31608,31524],{"class":503},[151,31610,31611],{"class":481},"-new.json\n",[151,31613,31614],{"class":469,"line":7816},[151,31615,1090],{"emptyLinePlaceholder":609},[151,31617,31618],{"class":469,"line":7821},[151,31619,31620],{"class":1527},"  # Get the existing configuration for the task definition (memory, cpu, etc.)\n",[151,31622,31623],{"class":469,"line":7847},[151,31624,31625],{"class":1527},"  # from the variable that we saved the task definition JSON to earlier\n",[151,31627,31628,31630,31633,31635],{"class":469,"line":7852},[151,31629,31394],{"class":2226},[151,31631,31632],{"class":481}," \"Getting existing configuration for ",[151,31634,31524],{"class":503},[151,31636,31637],{"class":481},"...\"\n",[151,31639,31640],{"class":469,"line":7887},[151,31641,1090],{"emptyLinePlaceholder":609},[151,31643,31644,31647,31649,31652,31654,31656,31658,31660,31662],{"class":469,"line":7892},[151,31645,31646],{"class":503},"  MEMORY",[151,31648,1876],{"class":1869},[151,31650,31651],{"class":503},"$( ",[151,31653,412],{"class":2226},[151,31655,31497],{"class":503},[151,31657,3947],{"class":1869},[151,31659,31502],{"class":473},[151,31661,12915],{"class":477},[151,31663,485],{"class":477},[151,31665,31666,31669],{"class":469,"line":7924},[151,31667,31668],{"class":481},"    .taskDefinition.memory",[151,31670,485],{"class":477},[151,31672,31673],{"class":469,"line":7929},[151,31674,31481],{"class":503},[151,31676,31677],{"class":469,"line":7991},[151,31678,1090],{"emptyLinePlaceholder":609},[151,31680,31681,31684,31686,31688,31690,31692,31694,31696,31698],{"class":469,"line":7996},[151,31682,31683],{"class":503},"  CPU",[151,31685,1876],{"class":1869},[151,31687,31651],{"class":503},[151,31689,412],{"class":2226},[151,31691,31497],{"class":503},[151,31693,3947],{"class":1869},[151,31695,31502],{"class":473},[151,31697,12915],{"class":477},[151,31699,485],{"class":477},[151,31701,31702,31705],{"class":469,"line":8078},[151,31703,31704],{"class":481},"    .taskDefinition.cpu",[151,31706,485],{"class":477},[151,31708,31709],{"class":469,"line":8140},[151,31710,31481],{"class":503},[151,31712,31713],{"class":469,"line":8145},[151,31714,1090],{"emptyLinePlaceholder":609},[151,31716,31717,31720,31722,31724,31726,31728,31730,31732,31734],{"class":469,"line":8259},[151,31718,31719],{"class":503},"  ECS_EXECUTION_ROLE_ARN",[151,31721,1876],{"class":1869},[151,31723,31651],{"class":503},[151,31725,412],{"class":2226},[151,31727,31497],{"class":503},[151,31729,3947],{"class":1869},[151,31731,31502],{"class":473},[151,31733,12915],{"class":477},[151,31735,485],{"class":477},[151,31737,31738,31741],{"class":469,"line":8264},[151,31739,31740],{"class":481},"    .taskDefinition.executionRoleArn",[151,31742,485],{"class":477},[151,31744,31745],{"class":469,"line":8613},[151,31746,31481],{"class":503},[151,31748,31749],{"class":469,"line":8678},[151,31750,1090],{"emptyLinePlaceholder":609},[151,31752,31753,31756,31758,31760,31762,31764,31766,31768,31770],{"class":469,"line":8742},[151,31754,31755],{"class":503},"  ECS_TASK_ROLE_ARN",[151,31757,1876],{"class":1869},[151,31759,31651],{"class":503},[151,31761,412],{"class":2226},[151,31763,31497],{"class":503},[151,31765,3947],{"class":1869},[151,31767,31502],{"class":473},[151,31769,12915],{"class":477},[151,31771,485],{"class":477},[151,31773,31774,31777],{"class":469,"line":8806},[151,31775,31776],{"class":481},"    .taskDefinition.taskRoleArn",[151,31778,485],{"class":477},[151,31780,31781],{"class":469,"line":8870},[151,31782,31481],{"class":503},[151,31784,31785],{"class":469,"line":8875},[151,31786,1090],{"emptyLinePlaceholder":609},[151,31788,31789],{"class":469,"line":8881},[151,31790,31791],{"class":1527},"  # check the content of the new container definition JSON\n",[151,31793,31794,31796,31798,31800],{"class":469,"line":8886},[151,31795,31562],{"class":473},[151,31797,31521],{"class":481},[151,31799,31524],{"class":503},[151,31801,31611],{"class":481},[151,31803,31804],{"class":469,"line":8892},[151,31805,1090],{"emptyLinePlaceholder":609},[151,31807,31808],{"class":469,"line":8963},[151,31809,31810],{"class":1527},"  # register new task definition using the new container definitions\n",[151,31812,31813],{"class":469,"line":8969},[151,31814,31815],{"class":1527},"  # and the values that we read off of the existing task definitions\n",[151,31817,31818,31820,31823,31825],{"class":469,"line":15001},[151,31819,31394],{"class":2226},[151,31821,31822],{"class":481}," \"Registering new ",[151,31824,31524],{"class":503},[151,31826,31403],{"class":481},[151,31828,31829],{"class":469,"line":15009},[151,31830,1090],{"emptyLinePlaceholder":609},[151,31832,31833,31836,31838,31841],{"class":469,"line":15019},[151,31834,31835],{"class":473},"  aws",[151,31837,31461],{"class":481},[151,31839,31840],{"class":481}," register-task-definition",[151,31842,485],{"class":477},[151,31844,31845,31848,31850],{"class":469,"line":15027},[151,31846,31847],{"class":477},"    --family",[151,31849,31474],{"class":503},[151,31851,497],{"class":477},[151,31853,31854,31857,31860,31862,31865],{"class":469,"line":15037},[151,31855,31856],{"class":477},"    --container-definitions",[151,31858,31859],{"class":481}," file:///tmp/",[151,31861,31524],{"class":503},[151,31863,31864],{"class":481},"-new.json",[151,31866,485],{"class":477},[151,31868,31869,31872,31875],{"class":469,"line":15045},[151,31870,31871],{"class":477},"    --memory",[151,31873,31874],{"class":503}," $MEMORY ",[151,31876,497],{"class":477},[151,31878,31879,31882,31885],{"class":469,"line":15055},[151,31880,31881],{"class":477},"    --cpu",[151,31883,31884],{"class":503}," $CPU ",[151,31886,497],{"class":477},[151,31888,31889,31892,31895],{"class":469,"line":15060},[151,31890,31891],{"class":477},"    --network-mode",[151,31893,31894],{"class":481}," awsvpc",[151,31896,485],{"class":477},[151,31898,31899,31902,31905],{"class":469,"line":15068},[151,31900,31901],{"class":477},"    --execution-role-arn",[151,31903,31904],{"class":503}," $ECS_EXECUTION_ROLE_ARN ",[151,31906,497],{"class":477},[151,31908,31909,31912,31915],{"class":469,"line":15076},[151,31910,31911],{"class":477},"    --task-role-arn",[151,31913,31914],{"class":503}," $ECS_TASK_ROLE_ARN ",[151,31916,497],{"class":477},[151,31918,31919,31922],{"class":469,"line":15085},[151,31920,31921],{"class":477},"    --requires-compatibilities",[151,31923,31924],{"class":481}," \"FARGATE\"\n",[151,31926,31927],{"class":469,"line":15095},[151,31928,1090],{"emptyLinePlaceholder":609},[151,31930,31931],{"class":469,"line":15105},[151,31932,31933],{"class":1869},"done\n",[151,31935,31936],{"class":469,"line":15110},[151,31937,1090],{"emptyLinePlaceholder":609},[151,31939,31940],{"class":469,"line":15118},[151,31941,31942],{"class":1527},"# Now we need to run migrate, collectstatic and any other commands that need to be run\n",[151,31944,31945],{"class":469,"line":15128},[151,31946,31947],{"class":1527},"# before doing a rolling update of the backend services\n",[151,31949,31950],{"class":469,"line":15139},[151,31951,1090],{"emptyLinePlaceholder":609},[151,31953,31955],{"class":469,"line":31954},94,[151,31956,31957],{"class":1527},"# We will use the new task definitions we just created to run these commands\n",[151,31959,31961],{"class":469,"line":31960},95,[151,31962,1090],{"emptyLinePlaceholder":609},[151,31964,31966],{"class":469,"line":31965},96,[151,31967,31968],{"class":1527},"# get the ARN of the most recent revision of the migrate task definition\n",[151,31970,31972,31975,31977,31979],{"class":469,"line":31971},97,[151,31973,31974],{"class":503},"TASK_DEFINITION",[151,31976,1876],{"class":1869},[151,31978,31651],{"class":503},[151,31980,497],{"class":477},[151,31982,31984,31986,31988,31990],{"class":469,"line":31983},98,[151,31985,31835],{"class":473},[151,31987,31461],{"class":481},[151,31989,31464],{"class":481},[151,31991,485],{"class":477},[151,31993,31995,31997,32000,32003],{"class":469,"line":31994},99,[151,31996,31471],{"class":477},[151,31998,31999],{"class":503}," $WORKSPACE",[151,32001,32002],{"class":481},"-migrate",[151,32004,485],{"class":477},[151,32006,32008,32010,32012,32014],{"class":469,"line":32007},100,[151,32009,31576],{"class":1869},[151,32011,31502],{"class":473},[151,32013,12915],{"class":477},[151,32015,485],{"class":477},[151,32017,32019,32022],{"class":469,"line":32018},101,[151,32020,32021],{"class":481},"    .taskDefinition.taskDefinitionArn",[151,32023,485],{"class":477},[151,32025,32027],{"class":469,"line":32026},102,[151,32028,3640],{"class":503},[151,32030,32032],{"class":469,"line":32031},103,[151,32033,1090],{"emptyLinePlaceholder":609},[151,32035,32037],{"class":469,"line":32036},104,[151,32038,32039],{"class":1527},"# get private subnets as space separated string from shared resources VPC\n",[151,32041,32043,32046,32048,32050],{"class":469,"line":32042},105,[151,32044,32045],{"class":503},"SUBNETS",[151,32047,1876],{"class":1869},[151,32049,31651],{"class":503},[151,32051,497],{"class":477},[151,32053,32055,32057,32060,32063],{"class":469,"line":32054},106,[151,32056,31835],{"class":473},[151,32058,32059],{"class":481}," ec2",[151,32061,32062],{"class":481}," describe-subnets",[151,32064,485],{"class":477},[151,32066,32068,32071,32074,32077,32079,32082],{"class":469,"line":32067},107,[151,32069,32070],{"class":477},"    --filters",[151,32072,32073],{"class":481}," \"Name=tag:env,Values=",[151,32075,32076],{"class":503},"$SHARED_RESOURCES_WORKSPACE",[151,32078,8592],{"class":481},[151,32080,32081],{"class":481}," \"Name=tag:Name,Values=*private*\"",[151,32083,485],{"class":477},[151,32085,32087,32090,32093],{"class":469,"line":32086},108,[151,32088,32089],{"class":477},"    --query",[151,32091,32092],{"class":481}," 'Subnets[*].SubnetId'",[151,32094,485],{"class":477},[151,32096,32098,32101,32104],{"class":469,"line":32097},109,[151,32099,32100],{"class":477},"    --output",[151,32102,32103],{"class":481}," text",[151,32105,485],{"class":477},[151,32107,32108],{"class":469,"line":25585},[151,32109,3640],{"class":503},[151,32111,32113],{"class":469,"line":32112},111,[151,32114,1090],{"emptyLinePlaceholder":609},[151,32116,32118],{"class":469,"line":32117},112,[151,32119,32120],{"class":1527},"# replace spaces with commas using tr\n",[151,32122,32124,32127,32129,32131,32133,32136,32138,32141,32144,32147],{"class":469,"line":32123},113,[151,32125,32126],{"class":503},"SUBNET_IDS",[151,32128,1876],{"class":1869},[151,32130,31456],{"class":503},[151,32132,412],{"class":2226},[151,32134,32135],{"class":503}," $SUBNETS ",[151,32137,3947],{"class":1869},[151,32139,32140],{"class":473}," tr",[151,32142,32143],{"class":481}," ' '",[151,32145,32146],{"class":481}," ','",[151,32148,3640],{"class":503},[151,32150,32152],{"class":469,"line":32151},114,[151,32153,1090],{"emptyLinePlaceholder":609},[151,32155,32157],{"class":469,"line":32156},115,[151,32158,32159],{"class":1527},"# https://github.com/aws/aws-cli/issues/5348\n",[151,32161,32163],{"class":469,"line":32162},116,[151,32164,32165],{"class":1527},"# get ecs_sg_id - just a single value\n",[151,32167,32169,32172,32174,32176],{"class":469,"line":32168},117,[151,32170,32171],{"class":503},"ECS_SG_ID",[151,32173,1876],{"class":1869},[151,32175,31651],{"class":503},[151,32177,497],{"class":477},[151,32179,32181,32183,32185,32188],{"class":469,"line":32180},118,[151,32182,31835],{"class":473},[151,32184,32059],{"class":481},[151,32186,32187],{"class":481}," describe-security-groups",[151,32189,485],{"class":477},[151,32191,32193,32195,32198,32200,32203],{"class":469,"line":32192},119,[151,32194,32070],{"class":477},[151,32196,32197],{"class":481}," \"Name=tag:Name,Values=",[151,32199,32076],{"class":503},[151,32201,32202],{"class":481},"-ecs-sg\"",[151,32204,485],{"class":477},[151,32206,32208,32210,32213],{"class":469,"line":32207},120,[151,32209,32089],{"class":477},[151,32211,32212],{"class":481}," 'SecurityGroups[*].GroupId'",[151,32214,485],{"class":477},[151,32216,32218,32220,32222],{"class":469,"line":32217},121,[151,32219,32100],{"class":477},[151,32221,32103],{"class":481},[151,32223,485],{"class":477},[151,32225,32227],{"class":469,"line":32226},122,[151,32228,3640],{"class":503},[151,32230,32232],{"class":469,"line":32231},123,[151,32233,1090],{"emptyLinePlaceholder":609},[151,32235,32237,32239],{"class":469,"line":32236},124,[151,32238,412],{"class":2226},[151,32240,32241],{"class":481}," \"Running database migrations...\"\n",[151,32243,32245],{"class":469,"line":32244},125,[151,32246,1090],{"emptyLinePlaceholder":609},[151,32248,32250],{"class":469,"line":32249},126,[151,32251,32252],{"class":1527},"# timestamp used for log retrieval (milliseconds after Jan 1, 1970 00:00:00 UTC)\n",[151,32254,32256,32259,32261,32263,32265,32268],{"class":469,"line":32255},127,[151,32257,32258],{"class":503},"START_TIME",[151,32260,1876],{"class":1869},[151,32262,31456],{"class":503},[151,32264,19646],{"class":473},[151,32266,32267],{"class":481}," +%s000",[151,32269,3640],{"class":503},[151,32271,32273],{"class":469,"line":32272},128,[151,32274,1090],{"emptyLinePlaceholder":609},[151,32276,32278],{"class":469,"line":32277},129,[151,32279,32280],{"class":1527},"# run the migration task and capture the taskArn into a variable called TASK_ID\n",[151,32282,32284,32287,32289,32291],{"class":469,"line":32283},130,[151,32285,32286],{"class":503},"TASK_ID",[151,32288,1876],{"class":1869},[151,32290,31651],{"class":503},[151,32292,497],{"class":477},[151,32294,32296,32298,32300,32303],{"class":469,"line":32295},131,[151,32297,31835],{"class":473},[151,32299,31461],{"class":481},[151,32301,32302],{"class":481}," run-task",[151,32304,485],{"class":477},[151,32306,32308,32311,32313,32316],{"class":469,"line":32307},132,[151,32309,32310],{"class":477},"    --cluster",[151,32312,31999],{"class":503},[151,32314,32315],{"class":481},"-cluster",[151,32317,485],{"class":477},[151,32319,32321,32323,32326],{"class":469,"line":32320},133,[151,32322,31471],{"class":477},[151,32324,32325],{"class":503}," $TASK_DEFINITION ",[151,32327,497],{"class":477},[151,32329,32331,32334,32337,32340,32342,32345,32348],{"class":469,"line":32330},134,[151,32332,32333],{"class":477},"    --network-configuration",[151,32335,32336],{"class":481}," \"awsvpcConfiguration={subnets=[",[151,32338,32339],{"class":503},"$SUBNET_IDS",[151,32341,27747],{"class":481},[151,32343,32344],{"class":503},"$ECS_SG_ID",[151,32346,32347],{"class":481},"],assignPublicIp=ENABLED}\"",[151,32349,485],{"class":477},[151,32351,32353,32355,32357,32359,32362],{"class":469,"line":32352},135,[151,32354,31576],{"class":1869},[151,32356,31502],{"class":473},[151,32358,12915],{"class":477},[151,32360,32361],{"class":481}," '.tasks[0].taskArn'",[151,32363,485],{"class":477},[151,32365,32367],{"class":469,"line":32366},136,[151,32368,31481],{"class":503},[151,32370,32372],{"class":469,"line":32371},137,[151,32373,1090],{"emptyLinePlaceholder":609},[151,32375,32377,32379,32382,32385],{"class":469,"line":32376},138,[151,32378,412],{"class":2226},[151,32380,32381],{"class":481}," \"Task ID is ",[151,32383,32384],{"class":503},"$TASK_ID",[151,32386,16406],{"class":481},[151,32388,32390],{"class":469,"line":32389},139,[151,32391,1090],{"emptyLinePlaceholder":609},[151,32393,32395],{"class":469,"line":32394},140,[151,32396,32397],{"class":1527},"# wait for the migrate task to exit\n",[151,32399,32401],{"class":469,"line":32400},141,[151,32402,32403],{"class":1527},"# https://docs.aws.amazon.com/cli/latest/reference/ecs/wait/tasks-stopped.html#description\n",[151,32405,32407],{"class":469,"line":32406},142,[151,32408,32409],{"class":1527},"# > It will poll every 6 seconds until a successful state has been reached.\n",[151,32411,32413],{"class":469,"line":32412},143,[151,32414,32415],{"class":1527},"# > This will exit with a return code of 255 after 100 failed checks.\n",[151,32417,32419,32421,32423,32426,32429],{"class":469,"line":32418},144,[151,32420,27264],{"class":473},[151,32422,31461],{"class":481},[151,32424,32425],{"class":481}," wait",[151,32427,32428],{"class":481}," tasks-stopped",[151,32430,485],{"class":477},[151,32432,32434,32436,32439],{"class":469,"line":32433},145,[151,32435,537],{"class":477},[151,32437,32438],{"class":503}," $TASK_ID ",[151,32440,497],{"class":477},[151,32442,32444,32447,32449],{"class":469,"line":32443},146,[151,32445,32446],{"class":477},"  --cluster",[151,32448,31999],{"class":503},[151,32450,32451],{"class":481},"-cluster\n",[151,32453,32455],{"class":469,"line":32454},147,[151,32456,1090],{"emptyLinePlaceholder":609},[151,32458,32460],{"class":469,"line":32459},148,[151,32461,32252],{"class":1527},[151,32463,32465,32468,32470,32472,32474,32476],{"class":469,"line":32464},149,[151,32466,32467],{"class":503},"END_TIME",[151,32469,1876],{"class":1869},[151,32471,31456],{"class":503},[151,32473,19646],{"class":473},[151,32475,32267],{"class":481},[151,32477,3640],{"class":503},[151,32479,32481],{"class":469,"line":32480},150,[151,32482,1090],{"emptyLinePlaceholder":609},[151,32484,32486],{"class":469,"line":32485},151,[151,32487,32488],{"class":1527},"# print the CloudWatch log events to STDOUT\n",[151,32490,32492,32494,32496,32499],{"class":469,"line":32491},152,[151,32493,27264],{"class":473},[151,32495,6181],{"class":481},[151,32497,32498],{"class":481}," get-log-events",[151,32500,485],{"class":477},[151,32502,32504,32507,32510,32512,32515],{"class":469,"line":32503},153,[151,32505,32506],{"class":477},"  --log-group-name",[151,32508,32509],{"class":481}," \"/ecs/",[151,32511,31432],{"class":503},[151,32513,32514],{"class":481},"/migrate\"",[151,32516,485],{"class":477},[151,32518,32520,32523,32526,32528,32531,32534],{"class":469,"line":32519},154,[151,32521,32522],{"class":477},"  --log-stream-name",[151,32524,32525],{"class":481}," \"migrate/migrate/${",[151,32527,32286],{"class":503},[151,32529,32530],{"class":1869},"##*/",[151,32532,32533],{"class":481},"}\"",[151,32535,485],{"class":477},[151,32537,32539,32542,32545],{"class":469,"line":32538},155,[151,32540,32541],{"class":477},"  --start-time",[151,32543,32544],{"class":503}," $START_TIME ",[151,32546,497],{"class":477},[151,32548,32550,32553,32556],{"class":469,"line":32549},156,[151,32551,32552],{"class":477},"  --end-time",[151,32554,32555],{"class":503}," $END_TIME ",[151,32557,497],{"class":477},[151,32559,32561,32564,32566,32568],{"class":469,"line":32560},157,[151,32562,32563],{"class":1869},"  |",[151,32565,31502],{"class":473},[151,32567,12915],{"class":477},[151,32569,32570],{"class":481}," '.events[].message'\n",[151,32572,32574],{"class":469,"line":32573},158,[151,32575,1090],{"emptyLinePlaceholder":609},[151,32577,32579,32581],{"class":469,"line":32578},159,[151,32580,412],{"class":2226},[151,32582,32583],{"class":481}," \"Migrations complete. Starting rolling update for backend services...\"\n",[151,32585,32587],{"class":469,"line":32586},160,[151,32588,1090],{"emptyLinePlaceholder":609},[151,32590,32592],{"class":469,"line":32591},161,[151,32593,32594],{"class":1527},"# update backend services\n",[151,32596,32598,32600,32602,32604,32606,32608],{"class":469,"line":32597},162,[151,32599,16732],{"class":1869},[151,32601,31370],{"class":503},[151,32603,16417],{"class":1869},[151,32605,31378],{"class":481},[151,32607,31381],{"class":481},[151,32609,31384],{"class":481},[151,32611,32613],{"class":469,"line":32612},163,[151,32614,31389],{"class":1869},[151,32616,32618],{"class":469,"line":32617},164,[151,32619,1090],{"emptyLinePlaceholder":609},[151,32621,32623],{"class":469,"line":32622},165,[151,32624,32625],{"class":1527},"  # get taskDefinitionArn for each service to be used in update-service command\n",[151,32627,32629],{"class":469,"line":32628},166,[151,32630,32631],{"class":1527},"  # this will get the most recent revision of each task (the one that was just created)\n",[151,32633,32635],{"class":469,"line":32634},167,[151,32636,32637],{"class":1527},"  # https://docs.aws.amazon.com/cli/latest/reference/ecs/describe-task-definition.html#description\n",[151,32639,32641,32644,32646,32648],{"class":469,"line":32640},168,[151,32642,32643],{"class":503},"  TASK_DEFINITION",[151,32645,1876],{"class":1869},[151,32647,31651],{"class":503},[151,32649,497],{"class":477},[151,32651,32653,32656,32658,32660],{"class":469,"line":32652},169,[151,32654,32655],{"class":473},"    aws",[151,32657,31461],{"class":481},[151,32659,31464],{"class":481},[151,32661,485],{"class":477},[151,32663,32665,32668,32670,32672,32675],{"class":469,"line":32664},170,[151,32666,32667],{"class":477},"      --task-definition",[151,32669,31999],{"class":503},[151,32671,12445],{"class":481},[151,32673,32674],{"class":503},"$TASK ",[151,32676,497],{"class":477},[151,32678,32680,32683,32685,32687],{"class":469,"line":32679},171,[151,32681,32682],{"class":1869},"      |",[151,32684,31502],{"class":473},[151,32686,12915],{"class":477},[151,32688,485],{"class":477},[151,32690,32692,32695],{"class":469,"line":32691},172,[151,32693,32694],{"class":481},"      .taskDefinition.taskDefinitionArn",[151,32696,485],{"class":477},[151,32698,32700],{"class":469,"line":32699},173,[151,32701,31481],{"class":503},[151,32703,32705],{"class":469,"line":32704},174,[151,32706,1090],{"emptyLinePlaceholder":609},[151,32708,32710],{"class":469,"line":32709},175,[151,32711,32712],{"class":1527},"  # update each service with new task definintion\n",[151,32714,32716,32718,32720,32723],{"class":469,"line":32715},176,[151,32717,31835],{"class":473},[151,32719,31461],{"class":481},[151,32721,32722],{"class":481}," update-service",[151,32724,485],{"class":477},[151,32726,32728,32730,32732,32734],{"class":469,"line":32727},177,[151,32729,32310],{"class":477},[151,32731,31999],{"class":503},[151,32733,32315],{"class":481},[151,32735,485],{"class":477},[151,32737,32739,32742,32744,32746,32748],{"class":469,"line":32738},178,[151,32740,32741],{"class":477},"    --service",[151,32743,31999],{"class":503},[151,32745,12445],{"class":481},[151,32747,32674],{"class":503},[151,32749,497],{"class":477},[151,32751,32753,32755,32757],{"class":469,"line":32752},179,[151,32754,31471],{"class":477},[151,32756,32325],{"class":503},[151,32758,497],{"class":477},[151,32760,32762],{"class":469,"line":32761},180,[151,32763,32764],{"class":477},"    --no-cli-pager\n",[151,32766,32768],{"class":469,"line":32767},181,[151,32769,1090],{"emptyLinePlaceholder":609},[151,32771,32773],{"class":469,"line":32772},182,[151,32774,31933],{"class":1869},[151,32776,32778],{"class":469,"line":32777},183,[151,32779,1090],{"emptyLinePlaceholder":609},[151,32781,32783,32785],{"class":469,"line":32782},184,[151,32784,412],{"class":2226},[151,32786,32787],{"class":481}," \"Services updated. Waiting for services to become stable...\"\n",[151,32789,32791],{"class":469,"line":32790},185,[151,32792,1090],{"emptyLinePlaceholder":609},[151,32794,32796],{"class":469,"line":32795},186,[151,32797,32798],{"class":1527},"# wait for all service to be stable (runningCount == desiredCount for each service)\n",[151,32800,32802,32804,32806,32808,32811],{"class":469,"line":32801},187,[151,32803,27264],{"class":473},[151,32805,31461],{"class":481},[151,32807,32425],{"class":481},[151,32809,32810],{"class":481}," services-stable",[151,32812,485],{"class":477},[151,32814,32816,32818,32820,32822],{"class":469,"line":32815},188,[151,32817,32446],{"class":477},[151,32819,31999],{"class":503},[151,32821,32315],{"class":481},[151,32823,485],{"class":477},[151,32825,32827,32830,32832,32835,32837,32840,32842],{"class":469,"line":32826},189,[151,32828,32829],{"class":477},"  --services",[151,32831,31999],{"class":503},[151,32833,32834],{"class":481},"-gunicorn",[151,32836,31999],{"class":503},[151,32838,32839],{"class":481},"-default",[151,32841,31999],{"class":503},[151,32843,32844],{"class":481},"-beat\n",[151,32846,32848],{"class":469,"line":32847},190,[151,32849,1090],{"emptyLinePlaceholder":609},[151,32851,32853,32855,32858,32860],{"class":469,"line":32852},191,[151,32854,412],{"class":2226},[151,32856,32857],{"class":481}," \"Services are now stable. Backend services are now up to date with ",[151,32859,31343],{"class":503},[151,32861,32862],{"class":481},".\"\n",[151,32864,32866],{"class":469,"line":32865},192,[151,32867,1090],{"emptyLinePlaceholder":609},[151,32869,32871,32873],{"class":469,"line":32870},193,[151,32872,412],{"class":2226},[151,32874,32875],{"class":481}," \"Backend update is now complete!\"\n",[11,32877,32878,32879,32882,32883,32885],{},"With this GitHub Actions workflow, a developer can now easily change the version of backend code that is running in their ad hoc environments without needing to involve Terraform. Using the ",[30,32880,32881],{},"ad_hoc_backend_update.yml"," GitHub Actions workflow, a developer only needs to enter the name of the workspace and the version of the backend code they want to use. The workflow will then run the ",[30,32884,27589],{}," task and update the backend services.",[736,32887,32889,32890,32893],{"id":32888},"using-ignore_changes-in-the-definitions","Using ",[30,32891,32892],{},"ignore_changes"," in the definitions",[11,32895,32896,32897,32899,32900,32903,32904,32907,32908,32913,32914,313,32916,32918],{},"There is one more important point to make about Terraform before we conclude this discussion on updating backend code for existing ad hoc environments. Consider the scenario where a developer has launched an ad hoc environment with backend version ",[30,32898,31088],{},". They make a small change to the backend code and push version ",[30,32901,32902],{},"v1.0.1",". Next, the developer remembers that a backend environment variable needs to be changed. This can be done by updating their ",[30,32905,32906],{},"*.tfvars"," file. If they now re-run the ad hoc environment update pipeline ",[15,32909,32910,32911,4231],{},"without also updating the backend version in their ",[30,32912,32906],{},", then their code will be reverted from ",[30,32915,32902],{},[30,32917,31088],{},". We would need to coordinate version changes between updating the backend with the pipelines that use Terraform commands and the pipelines that use the AWS CLI commands.",[11,32920,32921,32922,32925,32926,32932,32933,32936,32937,32939,32940,32942,32943,32945,32946,32948],{},"There is a setting on the ",[30,32923,32924],{},"aws_ecs_service"," resource in Terraform we can can use to prevent this from happening. This setting is called ",[20,32927,32930],{"href":32928,"rel":32929},"https://www.terraform.io/language/meta-arguments/lifecycle",[24],[30,32931,32892],{}," and is defined under the resource's ",[30,32934,32935],{},"lifecycle"," configuration block. With this setting, when we update the ",[30,32938,32906],{}," file with our new environment variable value, we will create another recent task definition with the same ",[30,32941,31088],{}," image, but the ECS service will not update in response to this change (that's what the ",[30,32944,32892],{}," is for). Once we make the ",[30,32947,32906],{}," file update and redeploy using the Terraform pipeline, nothing on our ad hoc changes, but we did get a new task definitions defined in our AWS account for each backend service. When we go to make the backend update with the pipeline that uses AWS CLI commands, the most recent task revision is used to create the new task definition, so it will include the environment variable change that we added earlier.",[736,32950,32952],{"id":32951},"frontend-updates","Frontend updates",[11,32954,32955,32956,32958],{},"The process described above is needed for updating the backend application. Updating the frontend application involves a similar process to the backend update. The main difference is that no task (such as the ",[30,32957,27589],{}," command on the backend) needs to run before the service is updated. Here's an overview of the frontend update process:",[76,32960,32961,32966,32971,32976,32981],{},[79,32962,32963,32964,748],{},"fetch the container definition JSON for the frontend tasks (",[30,32965,31185],{},[79,32967,32968,32969,748],{},"write new container definition JSON with the new frontend image tag (using ",[30,32970,31191],{},[79,32972,32973,32974,748],{},"register new task definitions with the new container definition JSON file for the frontend task (",[30,32975,31197],{},[79,32977,32978,32979,748],{},"update the frontend service (",[30,32980,31214],{},[79,32982,31217,32983,748],{},[30,32984,31220],{},[56,32986,32988],{"id":32987},"setting-up-everything-from-a-new-aws-account-and-github-actions","Setting up everything from a new AWS account and GitHub Actions",[11,32990,32991],{},"Here's a quick overview of initial setup steps that are needed in order to use the automation defined in the GitHub Actions for ad hoc environments.",[736,32993,32995],{"id":32994},"configure-aws-credentials-locally","Configure AWS credentials locally",[11,32997,32998],{},"There is one Terraform command that we will run on our local machine to setup a remote backend for storing our Terraform state. In order to run this, we need to configure AWS credentials locally.",[736,33000,12872,33002,33005],{"id":33001},"run-the-make-tf-bootstrap-command",[30,33003,33004],{},"make tf-bootstrap"," command",[11,33007,33008,33009,33012],{},"This command will setup an S3 bucket and DynamoDB table for storing Terraform state. Running this command will also require that a ",[30,33010,33011],{},"bootstrap.tfvars"," file has been created from the template. This will define the AWS region and name to be used for creating resources.",[736,33014,33016],{"id":33015},"build-and-push-a-backend-and-frontend-image","Build and push a backend and frontend image",[11,33018,19225,33019,33022,33023,187,33026,33029],{},[30,33020,33021],{},"tf-bootstrap"," command also creates ECR repositories for the backend and frontend images. We can use the ",[30,33024,33025],{},"ecr_backend.yml",[30,33027,33028],{},"ecr_frontend.yml"," GitHub Actions workflows to build and push the backend and frontend images to the ECR repositories. These pipelines accept a single parameter which is a git tag that must exist in the repo. This git tag will then be used as the image tag for the backend and frontend images.",[736,33031,33033],{"id":33032},"purchase-a-domain-name-in-route-53","Purchase a domain name in Route 53",[11,33035,33036],{},"I use Route 53 to manage DNS records, and have purchased a domain name that I use for testing and debugging in this and other projects.",[736,33038,33040],{"id":33039},"create-a-wildcard-acm-certificate","Create a wildcard ACM certificate",[11,33042,33043],{},"I chose to create this certificate outside of Terraform and import it via ARN. We need a wildcard certificate so that multiple ad hoc environments can be hosted on subdomains of the same domain.",[736,33045,33047],{"id":33046},"create-a-new-ec2-key-pair","Create a new EC2 key pair",[11,33049,33050],{},"The key pair should be created manually and it needs to be added to GitHub repository secrets so that it can be used in the ad hoc environment pipelines.",[736,33052,33054],{"id":33053},"add-secrets-to-github","Add secrets to GitHub",[11,33056,33057],{},"The following secrets are needed for GitHub Actions to run. Add these as repository secrets:",[76,33059,33060,33066,33072,33078,33087,33093,33099,33105,33111,33117,33123],{},[79,33061,33062,33065],{},[30,33063,33064],{},"ACM_CERTIFICATE_ARN"," - ARN of the wildcard ACM certificate",[79,33067,33068,33071],{},[30,33069,33070],{},"AWS_ACCESS_KEY_ID"," - AWS access key ID",[79,33073,33074,33077],{},[30,33075,33076],{},"AWS_ACCOUNT_ID"," - AWS account ID",[79,33079,33080,33083,33084,748],{},[30,33081,33082],{},"AWS_DEFAULT_REGION"," - AWS default region (I use ",[30,33085,33086],{},"us-east-1",[79,33088,33089,33092],{},[30,33090,33091],{},"AWS_SECRET_ACCESS_KEY"," - AWS secret access key",[79,33094,33095,33098],{},[30,33096,33097],{},"DOMAIN_NAME"," - domain name for the ad hoc environment (e.g. example.com)",[79,33100,33101,33104],{},[30,33102,33103],{},"KEY_NAME"," - name of the EC2 key pair",[79,33106,33107,33110],{},[30,33108,33109],{},"SSH_PRIVATE_KEY"," - private key for the EC2 key pair",[79,33112,33113,33116],{},[30,33114,33115],{},"TF_BACKEND_BUCKET"," - name of the S3 bucket used for storing Terraform state",[79,33118,33119,33122],{},[30,33120,33121],{},"TF_BACKEND_DYNAMODB_TABLE"," - name of the DynamoDB table used for locking the Terraform state file",[79,33124,33125,33128],{},[30,33126,33127],{},"TF_BACKEND_REGION"," - AWS region for the S3 bucket and DynamoDB table",[11,33130,33131],{},"These secrets are referenced in the GitHub Actions workflows.",[736,33133,33135],{"id":33134},"create-shared-resources","Create shared resources",[11,33137,33138,33139,33142,33143,33145,33146,33149,33150,33153],{},"Now that we have GitHub secrets configured, we can run the ",[30,33140,33141],{},"shared_resources_create_update.yml"," GitHub Actions workflow. This will create  shared resources environment in which we can build our ad hoc environments. This workflow requires a name (e.g. ",[30,33144,10715],{},"). This require that we create a ",[30,33147,33148],{},"dev.tfvars"," file in ",[30,33151,33152],{},"terraform/live/shared-resources"," directory. This usually takes between 5 and 7 minutes to complete since it needs to create an RDS instance which takes a few minutes to provision.",[736,33155,33157],{"id":33156},"create-an-ad-hoc-environment","Create an ad hoc environment",[11,33159,33160,33161,33163,33164,33167,33168,33170,33171,33174,33175,33178],{},"We can now create an ad hoc environment. This requires the name of the shared resources environment (e.g. ",[30,33162,10715],{},") and the name of the ad hoc environment (e.g. ",[30,33165,33166],{},"brian-test","). The only thing we need to do before creating the ",[30,33169,33166],{}," ad hoc environment is to create the ",[30,33172,33173],{},"brian-test.tfvars"," file in the ",[30,33176,33177],{},"terraform/live/ad-hoc/envs"," directory. This will define the versions of the application and any other environment configuration that is needed. This pipeline usually takes about 3 minutes to finish.",[736,33180,33182],{"id":33181},"update-the-backend-version-in-an-ad-hoc-environment","Update the backend version in an ad hoc environment",[11,33184,33185,33186,33188],{},"Now that the backend is up and running and usable, we can update the backend version used in our ad hoc environment. This can be done by running the ",[30,33187,32881],{}," GitHub Actions workflow. To run this workflow you must specify:",[76,33190,33191,33196,33201],{},[79,33192,33193,33194,748],{},"the shared resource workspace (e.g. ",[30,33195,10715],{},[79,33197,33198,33199,748],{},"the ad hoc environment name (e.g. ",[30,33200,33166],{},[79,33202,33203,33204,748],{},"the new version of the backend to be used (e.g. ",[30,33205,33206],{},"v1.0.2",[11,33208,33209,33210,33212],{},"The backend version should already have been built and pushed to the ECR repository using the ",[30,33211,33025],{}," GitHub Actions workflow.",[736,33214,33216],{"id":33215},"use-ecs-exec-to-access-a-django-shell-in-a-running-container","Use ECS Exec to access a Django shell in a running container",[11,33218,33219,33220,33223,33224,33226],{},"Instead of SSHing into the container, we can use the ",[30,33221,33222],{},"ecs-exec"," command to access a shell in the container. This is useful for debugging and testing. One of the outputs of the ad hoc environment terraform configuration contains the script needed to run the ",[30,33225,33222],{}," command:",[459,33228,33230],{"className":461,"code":33229,"language":463,"meta":464,"style":464},"TASK_ARN=$(aws ecs list-tasks \\\n  --cluster alpha-cluster \\\n  --service-name  alpha-gunicorn | jq -r '.taskArns | .[0]' \\\n)\naws ecs execute-command --cluster alpha-cluster \\\n    --task $TASK_ARN \\\n    --container gunicorn \\\n    --interactive \\\n    --command \"/bin/bash\"\n",[30,33231,33232,33250,33259,33278,33282,33298,33308,33318,33325],{"__ignoreMap":464},[151,33233,33234,33237,33239,33241,33243,33245,33248],{"class":469,"line":470},[151,33235,33236],{"class":503},"TASK_ARN",[151,33238,1876],{"class":1869},[151,33240,31456],{"class":503},[151,33242,27264],{"class":473},[151,33244,31461],{"class":481},[151,33246,33247],{"class":481}," list-tasks",[151,33249,485],{"class":477},[151,33251,33252,33254,33257],{"class":469,"line":488},[151,33253,32446],{"class":477},[151,33255,33256],{"class":481}," alpha-cluster",[151,33258,485],{"class":477},[151,33260,33261,33264,33267,33269,33271,33273,33276],{"class":469,"line":500},[151,33262,33263],{"class":477},"  --service-name",[151,33265,33266],{"class":481},"  alpha-gunicorn",[151,33268,3959],{"class":1869},[151,33270,31502],{"class":473},[151,33272,12915],{"class":477},[151,33274,33275],{"class":481}," '.taskArns | .[0]'",[151,33277,485],{"class":477},[151,33279,33280],{"class":469,"line":509},[151,33281,3640],{"class":503},[151,33283,33284,33286,33288,33291,33294,33296],{"class":469,"line":517},[151,33285,27264],{"class":473},[151,33287,31461],{"class":481},[151,33289,33290],{"class":481}," execute-command",[151,33292,33293],{"class":477}," --cluster",[151,33295,33256],{"class":481},[151,33297,485],{"class":477},[151,33299,33300,33303,33306],{"class":469,"line":534},[151,33301,33302],{"class":477},"    --task",[151,33304,33305],{"class":503}," $TASK_ARN ",[151,33307,497],{"class":477},[151,33309,33310,33313,33316],{"class":469,"line":1413},[151,33311,33312],{"class":477},"    --container",[151,33314,33315],{"class":481}," gunicorn",[151,33317,485],{"class":477},[151,33319,33320,33323],{"class":469,"line":1418},[151,33321,33322],{"class":477},"    --interactive",[151,33324,485],{"class":477},[151,33326,33327,33330],{"class":469,"line":2462},[151,33328,33329],{"class":477},"    --command",[151,33331,33332],{"class":481}," \"/bin/bash\"\n",[11,33334,33335],{},"You will then have a shell in the container:",[459,33337,33340],{"className":33338,"code":33339,"language":997},[995],"The Session Manager plugin was installed successfully. Use the AWS CLI to start a session.\n\n\nStarting session with SessionId: ecs-execute-command-073d1947fa71c058c\nroot@ip-10-0-2-167:/code# python manage.py shell\nPython 3.9.9 (main, Dec 21 2021, 10:03:34)\n[GCC 10.2.1 20210110] on linux\nType \"help\", \"copyright\", \"credits\" or \"license\" for more information.\n(InteractiveConsole)\n>>>\n",[30,33341,33339],{"__ignoreMap":464},[736,33343,33345],{"id":33344},"destroy-an-ad-hoc-environment","Destroy an ad hoc environment",[11,33347,33348,33349,33352,33353,33355,33356,13576],{},"Use the ",[30,33350,33351],{},"ad_hoc_env_destroy.yml"," GitHub Actions workflow to destroy an ad hoc environment. To run this workflow you need to specify the shared resource workspace (e.g. ",[30,33354,10715],{},") and the ad hoc environment name (e.g. ",[30,33357,33166],{},[736,33359,33361],{"id":33360},"destroy-the-shared-resources-environment","Destroy the shared resources environment",[11,33363,33348,33364,33367,33368,13576],{},[30,33365,33366],{},"shared_resources_destroy.yml"," GitHub Actions workflow to destroy the shared resources environment. To run this workflow you need to specify the shared resource workspace (e.g. ",[30,33369,10715],{},[11,33371,33372],{},"This defines the full lifecycle of creating and destroying ad hoc environments.",[56,33374,33376],{"id":33375},"future-improvements-open-questions-and-next-steps","Future improvements, open questions and next steps",[736,33378,33380],{"id":33379},"enhanced-security","Enhanced Security",[11,33382,33383],{},"The low hanging fruit here is to use least privilege (for roles used in automation) and to define all roles with IaC. Currently I am using Admin roles for the credentials I store in GitHub which is a shortcut to using IaC and is not a best practice.",[736,33385,33387],{"id":33386},"keeping-track-of-ad-hoc-environments","Keeping track of ad hoc environments",[11,33389,33390],{},"We need to think about how we can keep track of our active ad hoc environments. Active environments will incur additional AWS costs, and we do not want developers or the product team to create lots of environments and then leave them running without actively using them.",[11,33392,33393],{},"We may decide to have some long-lived ad hoc environments, but those would be managed primarily by the DevOps team and respective owners (e.g. QA, product team, etc.).",[11,33395,33396],{},"One way to check the active ad hoc environments would be to use the AWS CLI. We could list the ECS clusters in our development account, and this would show the number of ad hoc environments running. We could go farther and list the ad hoc environments by when they were last updated. We could then request developers or team members to remove ad hoc environments that are not in use.",[11,33398,33399],{},"Or we could have a policy that all ad-hoc environments are deleted automatically at the end of each week.",[736,33401,33403],{"id":33402},"terraform-tooling","Terraform Tooling",[76,33405,33406,33409,33412,33415],{},[79,33407,33408],{},"Testing Terraform Code",[79,33410,33411],{},"Testing GitHub Actions",[79,33413,33414],{},"Using advanced Terraform frameworks like Terragrunt",[79,33416,33417],{},"Using Terraform Cloud for more advanced Terraform features",[736,33419,33421],{"id":33420},"more-secure-way-of-defining-rds-username-and-password","More secure way of defining RDS username and password",[11,33423,33424],{},"Currently the postgres database does not have a secure password. It is both hardcoded in the module as a default value and it will also be saved in plaintext in the Terraform state file.",[736,33426,33428],{"id":33427},"backend-application-update-script","Backend application update script",[11,33430,33431],{},"The script used for updating the backend application could be improved or broken up into multiple scripts to better handle errors and failures that happen during the pipeline. The script runs several different commands and could potentially fail at any step, so it would be nice to improve the error messages so that both developers and DevOps teams can more quickly diagnose pipeline failures.",[736,33433,33435],{"id":33434},"limiting-traffic-to-ad-hoc-environments-to-a-vpn","Limiting traffic to ad hoc environments to a VPN",[11,33437,33438],{},"Another good next step would be to show how we can limit traffic to ad hoc environments to a VPN.",[736,33440,33442],{"id":33441},"figure-out-how-many-ad-hoc-environments-we-can-create-with-the-default-quotas","Figure out how many ad hoc environments we can create with the default quotas",[11,33444,33445],{},"AWS accounts limit the number of resources you can create, but for most of this quota limits you can request an increase per account. I need to figure out how many ad hoc environments I can create with the default quotas.",[736,33447,33449],{"id":33448},"code-repo-organization","Code Repo Organization",[11,33451,33452,33453,33455,33456,33458],{},"One minor improvement would be to move the ",[30,33454,27951],{}," directory out of the ",[30,33457,26634],{}," monorepo into a dedicated repo. We may also want to move GitHub Actions for creating, updating and destroying environments to this new repo. For early stage development, using a single repository that stores both application code and Terraform configuration works, but it would be better to keep these separate at the repository level as the project grows.",[736,33460,33462],{"id":33461},"multiple-aws-accounts","Multiple AWS Accounts",[11,33464,33465],{},"Everything shown here uses a single AWS account: ECR images, Terraform remote state storage, all shared resource environments and all ad hoc environments. Using one account keeps things simple for a demonstration of this workflow, but in practice it would be beneficial to use multiple AWS accounts for different purposes. This would also involve more carefully planned IAM roles for cross-account resource access.",[736,33467,33469],{"id":33468},"modules-for-stable-environments-to-be-used-for-long-lived-pre-production-and-production-environments","Modules for stable environments to be used for long-lived pre-production and production environments",[11,33471,33472],{},"This article looked at how to make tradeoffs between costs, speed of deployment and production parity in ad hoc environments. I'm interested in building a new set of modules that can be used to set up environments that:",[76,33474,33475,33478,33481,33484,33487],{},[79,33476,33477],{},"are more stable and more long-lived",[79,33479,33480],{},"have less resource sharing (dedicated RDS and ElastiCache resources)",[79,33482,33483],{},"implement autoscaling for load-testing (or maybe implement autoscaling for ad hoc environments)",[79,33485,33486],{},"can be used to perform load testing",[79,33488,33489],{},"have enhanced observability tooling",[11,33491,33492],{},"These environments might be used as part of a QA process that does a final sign-off on a new set of features scheduled for deployment to production environments, for example.",[56,33494,14265],{"id":30030},[11,33496,33497],{},"This wraps up the tour of my ad hoc environment infrastructure automation. Thank you for having a read through my article. If you have a similar (or different) approach to building ad hoc environments, I would love to hear about it.",[589,33499,33500],{},"html pre.shiki code .s8-w5, html code.shiki .s8-w5{--shiki-default:#6A737D;--shiki-dark:#6A737D;--shiki-sepia:#88846F}html pre.shiki code .sTrkL, html code.shiki .sTrkL{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#66D9EF}html pre.shiki code .sstjo, html code.shiki .sstjo{--shiki-default:#032F62;--shiki-dark:#9ECBFF;--shiki-sepia:#E6DB74}html pre.shiki code .sMOD_, html code.shiki .sMOD_{--shiki-default:#24292E;--shiki-dark:#E1E4E8;--shiki-sepia:#F8F8F2}html pre.shiki code .sC2Qs, html code.shiki .sC2Qs{--shiki-default:#D73A49;--shiki-dark:#F97583;--shiki-sepia:#F92672}html pre.shiki code .srTi1, html code.shiki .srTi1{--shiki-default:#6F42C1;--shiki-dark:#B392F0;--shiki-sepia:#A6E22E}html pre.shiki code .s7F3e, html code.shiki .s7F3e{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#AE81FF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html .sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html.sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}",{"title":464,"searchDepth":488,"depth":488,"links":33502},[33503,33504,33505,33506,33507,33514,33515,33519,33528,33532,33536,33543,33559,33571],{"id":16115,"depth":488,"text":16116},{"id":30148,"depth":488,"text":30149},{"id":30221,"depth":488,"text":30222},{"id":30289,"depth":488,"text":30290},{"id":30296,"depth":488,"text":30297,"children":33508},[33509,33510,33511,33512,33513],{"id":30326,"depth":500,"text":30327},{"id":30423,"depth":500,"text":30424},{"id":30467,"depth":500,"text":30468},{"id":30511,"depth":500,"text":30512},{"id":30552,"depth":500,"text":30553},{"id":30605,"depth":488,"text":30606},{"id":30725,"depth":488,"text":30726,"children":33516},[33517,33518],{"id":30743,"depth":500,"text":30744},{"id":30779,"depth":500,"text":30780},{"id":30824,"depth":488,"text":30825,"children":33520},[33521,33522,33523,33524,33525,33526,33527],{"id":26897,"depth":500,"text":26898},{"id":30841,"depth":500,"text":30842},{"id":26935,"depth":500,"text":26850},{"id":30883,"depth":500,"text":30884},{"id":30890,"depth":500,"text":30891},{"id":30897,"depth":500,"text":26853},{"id":27232,"depth":500,"text":26859},{"id":30919,"depth":488,"text":30920,"children":33529},[33530,33531],{"id":27347,"depth":500,"text":27348},{"id":30931,"depth":500,"text":30932},{"id":30970,"depth":488,"text":30971,"children":33533},[33534,33535],{"id":30977,"depth":500,"text":30978},{"id":31039,"depth":500,"text":31040},{"id":31078,"depth":488,"text":31079,"children":33537},[33538,33539,33540,33542],{"id":31145,"depth":500,"text":31146},{"id":31173,"depth":500,"text":31174},{"id":32888,"depth":500,"text":33541},"Using ignore_changes in the definitions",{"id":32951,"depth":500,"text":32952},{"id":32987,"depth":488,"text":32988,"children":33544},[33545,33546,33548,33549,33550,33551,33552,33553,33554,33555,33556,33557,33558],{"id":32994,"depth":500,"text":32995},{"id":33001,"depth":500,"text":33547},"Run the make tf-bootstrap command",{"id":33015,"depth":500,"text":33016},{"id":33032,"depth":500,"text":33033},{"id":33039,"depth":500,"text":33040},{"id":33046,"depth":500,"text":33047},{"id":33053,"depth":500,"text":33054},{"id":33134,"depth":500,"text":33135},{"id":33156,"depth":500,"text":33157},{"id":33181,"depth":500,"text":33182},{"id":33215,"depth":500,"text":33216},{"id":33344,"depth":500,"text":33345},{"id":33360,"depth":500,"text":33361},{"id":33375,"depth":488,"text":33376,"children":33560},[33561,33562,33563,33564,33565,33566,33567,33568,33569,33570],{"id":33379,"depth":500,"text":33380},{"id":33386,"depth":500,"text":33387},{"id":33402,"depth":500,"text":33403},{"id":33420,"depth":500,"text":33421},{"id":33427,"depth":500,"text":33428},{"id":33434,"depth":500,"text":33435},{"id":33441,"depth":500,"text":33442},{"id":33448,"depth":500,"text":33449},{"id":33461,"depth":500,"text":33462},{"id":33468,"depth":500,"text":33469},{"id":30030,"depth":488,"text":14265},"2022-06-11","This article will show how software development teams can build on-demand environments for dog-food testing, quality review, internal and external demos and other use cases that require short-lived but feature-complete instances of a web application.",[33575,33577,33579,33581,33583,33584,33587,33589],{"link":33576,"site":25725},"https://news.ycombinator.com/item?id=31704417",{"link":33578,"site":23769},"https://www.reddit.com/r/aws/comments/v9ycwh/my_approach_to_building_ad_hoc_developer/",{"link":33580,"site":10715},"https://dev.to/briancaffey/setting-up-ad-hoc-development-environments-for-django-applications-with-aws-ecs-terraform-and-github-actions-4abh",{"link":33582,"site":30109},"https://medium.com/@briancaffey/setting-up-ad-hoc-development-environments-for-django-applications-with-aws-ecs-terraform-and-84d26e710539",{"link":30111,"site":30112},{"link":33585,"site":33586},"https://twitter.com/briancaffey/status/1535683865330831360","twitter",{"link":33588,"site":30115},"https://briancaffey.substack.com/p/setting-up-ad-hoc-development-environments",{"link":33590,"site":33591},"https://hackernoon.com/ad-hoc-environments-for-django-applications-with-ecs-terraform-and-github-actions","hackernoon",{},"/2022/03/27/ad-hoc-developer-environments-for-django-with-aws-ecs-terraform-and-github-actions",{"title":30133,"description":33573},"2022/03/27/ad-hoc-developer-environments-for-django-with-aws-ecs-terraform-and-github-actions",[30122,27951,30125,27264,30126,30128,30129],"sWTAoM77fFzFT8yIILU-YXWGOXMS0N1YMC3BicFhHPA",{"id":33599,"title":33600,"body":33601,"comments":609,"date":34890,"description":34891,"draft":602,"extension":605,"external":34892,"image":34905,"meta":34906,"navigation":609,"path":34907,"seo":34908,"stem":34909,"tags":34910,"__hash__":34913},"blog/2021/10/31/how-and-why-i-added-adsense-and-adblock-detector-to-my-website.md","How and why I added AdSense and an AdBlock detector to my personal website",{"type":8,"value":33602,"toc":34880},[33603,33606,33618,33621,33625,33628,33648,33651,33654,33657,33681,33684,33690,33695,33798,33801,33819,33822,33825,33859,33862,33866,33869,33873,33882,33886,33889,33897,33902,33910,33918,33927,33932,33937,33948,33953,33958,33961,33966,33973,34425,34432,34445,34450,34460,34637,34642,34754,34759,34764,34768,34771,34782,34793,34796,34819,34825,34828,34831,34834,34845,34848,34852,34855,34861,34869,34877],[11,33604,33605],{},"Update: I disabled the ad block blocker on my personal website. It was an interesting experiment and I learned that ad block detection is a game of cat and mouse. It did not work with uBlock Origin, but it did work with AdBlock. I probably won't be revisiting this topic.",[11,33607,33608,33609,33614,33615,33617],{},"If you are reading this article on ",[20,33610,33613],{"href":33611,"rel":33612},"https://briancaffey.github.io/2021/10/31/how-and-why-i-added-adsense-and-adblock-detector-to-my-website",[24],"briancaffey.github.io/2021/10/31/how-and-why-i-added-adsense-and-adblock-detector-to-my-website",", then you will be prompted to pause your ad blocker if you are using one. If you are ",[51,33616,241],{}," using an ad blocker, I recommend that you consider installing one. Reading this article on a browser with AdBlock enabled will allow you to see how I detect AdBlock and ask people to pause it when they are on my site.",[11,33619,33620],{},"This article is a deep dive on how I added ads to my site with Google AdSense and how I request that visitors to my site pause AdBlock so that I can make more money from Google AdSense.",[56,33622,33624],{"id":33623},"how-i-added-adsense-to-my-site","How I added AdSense to my site",[11,33626,33627],{},"I have been enjoying using my GitHub Pages website to learn more about static sites, JAMStack and Nuxt.js, an awesome Vue.js framework with support for building statically generated sites. I have been able to learn and implement several different features which I have written about on my blog. Some examples include:",[76,33629,33630,33633,33636,33639,33642,33645],{},[79,33631,33632],{},"Adding a Drift chat window so users can message me directly",[79,33634,33635],{},"Implementing a contact form with formsubmit.io",[79,33637,33638],{},"Using Vue.js components in Markdown files to add interactive elements to my articles (such as graphs)",[79,33640,33641],{},"Adding a custom MailChimp newsletter sign-up form that is included in the footer of each page of my blog",[79,33643,33644],{},"Adding an RSS feed for my blog",[79,33646,33647],{},"Adding a site index and submitting it to Google",[11,33649,33650],{},"I have also been learning the suite of Google tools for monitoring and measuring traffic to my site, including Google Analytics and Google Search Console. Google Search Console is helpful for understanding the search terms that people are using when searching Google that result in organic traffic to my site.",[11,33652,33653],{},"At one point I found out that another website was using the same Google Tracking code that I had previously hard-coded into an old version of my website, and my Google Analytics started measuring traffic to URLs that I didn't recognize as belonging to my site. I was able to fix this by adding a Hostname filter rule in Google Analytics.",[11,33655,33656],{},"One area that I have not had any experience with until recently is Google AdSense. Google AdSense allows you to place ads on your website. Here's an overview of what I did to get started:",[76,33658,33659,33662,33665,33668,33675,33678],{},[79,33660,33661],{},"Add a site in Google AdSense",[79,33663,33664],{},"Submit my site for approval (this takes a few days)",[79,33666,33667],{},"Install and configure the Google AdSense plugin for NuxtJS",[79,33669,33670,33671,33674],{},"Add the ",[30,33672,33673],{},"ads.txt"," file generated by Google AdSense to my site",[79,33676,33677],{},"Confirm my address by entering a code that was mailed to me",[79,33679,33680],{},"Connect a bank account to my Google AdSense account",[11,33682,33683],{},"Here's the address confirmation code that I received from Google:",[11,33685,33686],{},[2718,33687],{"alt":33688,"src":33689},"Address confirmation","/static/google_letter.jpeg",[11,33691,33692,33693,208],{},"Here's the config code for AdSense from ",[30,33694,19456],{},[459,33696,33698],{"className":12338,"code":33697,"language":12340,"meta":464,"style":464},"  /*\n   ** Nuxt.js modules\n   */\n  modules: [\n    // Doc: https://axios.nuxtjs.org/usage\n    '@nuxtjs/axios',\n    // Doc: https://github.com/nuxt/content\n    '@nuxt/content',\n    // Doc: https://www.npmjs.com/package/@nuxtjs/sitemap\n    '@nuxtjs/sitemap',\n    '@nuxtjs/feed',\n    'nuxt-i18n',\n    ['@nuxtjs/google-adsense', {     \u003C-- AdSense config\n      id: 'ca-pub-4924597640144289'\n    }]\n  ],\n",[30,33699,33700,33705,33710,33715,33722,33727,33734,33739,33745,33750,33756,33763,33770,33781,33789,33794],{"__ignoreMap":464},[151,33701,33702],{"class":469,"line":470},[151,33703,33704],{"class":1527},"  /*\n",[151,33706,33707],{"class":469,"line":488},[151,33708,33709],{"class":1527},"   ** Nuxt.js modules\n",[151,33711,33712],{"class":469,"line":500},[151,33713,33714],{"class":1527},"   */\n",[151,33716,33717,33720],{"class":469,"line":509},[151,33718,33719],{"class":19627},"  modules",[151,33721,9399],{"class":503},[151,33723,33724],{"class":469,"line":517},[151,33725,33726],{"class":1527},"    // Doc: https://axios.nuxtjs.org/usage\n",[151,33728,33729,33732],{"class":469,"line":534},[151,33730,33731],{"class":481},"    '@nuxtjs/axios'",[151,33733,9417],{"class":503},[151,33735,33736],{"class":469,"line":1413},[151,33737,33738],{"class":1527},"    // Doc: https://github.com/nuxt/content\n",[151,33740,33741,33743],{"class":469,"line":1418},[151,33742,19486],{"class":481},[151,33744,9417],{"class":503},[151,33746,33747],{"class":469,"line":2462},[151,33748,33749],{"class":1527},"    // Doc: https://www.npmjs.com/package/@nuxtjs/sitemap\n",[151,33751,33752,33754],{"class":469,"line":2471},[151,33753,19528],{"class":481},[151,33755,9417],{"class":503},[151,33757,33758,33761],{"class":469,"line":2480},[151,33759,33760],{"class":481},"    '@nuxtjs/feed'",[151,33762,9417],{"class":503},[151,33764,33765,33768],{"class":469,"line":2489},[151,33766,33767],{"class":481},"    'nuxt-i18n'",[151,33769,9417],{"class":503},[151,33771,33772,33775,33778],{"class":469,"line":2497},[151,33773,33774],{"class":503},"    [",[151,33776,33777],{"class":481},"'@nuxtjs/google-adsense'",[151,33779,33780],{"class":503},", {     \u003C-- AdSense config\n",[151,33782,33783,33786],{"class":469,"line":3140},[151,33784,33785],{"class":503},"      id: ",[151,33787,33788],{"class":481},"'ca-pub-4924597640144289'\n",[151,33790,33791],{"class":469,"line":3149},[151,33792,33793],{"class":503},"    }]\n",[151,33795,33796],{"class":469,"line":3158},[151,33797,9466],{"class":503},[11,33799,33800],{},"The process was pretty simple. Google now automatically places ads on my site in a few different formats:",[76,33802,33803,33806,33809,33812],{},[79,33804,33805],{},"ads displayed on the top and bottom of the page",[79,33807,33808],{},"popup ads displayed between route navigation",[79,33810,33811],{},"ads automatically inserted into the body of the page between paragraphs in my articles",[79,33813,33814,33815,33818],{},"ads that I place on articles explicitly using the ",[30,33816,33817],{},"\u003Cadsbygoogle />"," Vue component",[11,33820,33821],{},"When everything was set up properly I started seeing ads on my site, and I see a non-zero value in my estimated earnings in the AdSense console. Google has a payment threshold of $100, so I need make this amount before I can start receiving money from Google.",[11,33823,33824],{},"My estimated earning report shows that I make between $0 and $4.32 in ad sales per day. I'm interested to see how much I can make with an article that I post across the many different channels that I can publish to. I explored this in a previous article, but the main channels I can use for sharing content are:",[76,33826,33827,33830,33833,33836,33838,33841,33844,33847,33850,33853,33856],{},[79,33828,33829],{},"DEV.to",[79,33831,33832],{},"Facebook",[79,33834,33835],{},"Hashnode",[79,33837,1177],{},[79,33839,33840],{},"Reddit",[79,33842,33843],{},"Discord",[79,33845,33846],{},"Hacker Noon",[79,33848,33849],{},"Twitter",[79,33851,33852],{},"My MailChimp mailing list",[79,33854,33855],{},"Substack",[79,33857,33858],{},"Hacker News",[11,33860,33861],{},"This article should be a good place to start exploring how effective the different channels are in driving content to my site, and I'll update this article later with more details and numbers from my AdSense reports.",[56,33863,33865],{"id":33864},"how-i-built-an-adblock-detector-for-my-site","How I built an AdBlock detector for my site",[11,33867,33868],{},"I assume that most people reading my blog have an AdBlock extension installed in their browser like I do, such as AdBlock or ABP (AdBlock Pro). This got me thinking about how I could implement a simple AdBlock detector for my site that would hide the contents of the page if AdBlock is enabled.",[736,33870,33872],{"id":33871},"how-do-you-check-to-see-if-adblock-is-enabled","How do you check to see if AdBlock is enabled?",[11,33874,33875,33876,33881],{},"I started with this question, and I came across ",[20,33877,33880],{"href":33878,"rel":33879},"https://stackoverflow.com/questions/4869154/how-to-detect-adblock-on-my-website",[24],"this StackOverflow question"," which inspired the code that I am now using on this site to detect AdBlock.",[736,33883,33885],{"id":33884},"the-components-of-my-adblock-detector","The components of my AdBlock detector",[11,33887,33888],{},"There are a few different parts of my Nuxt Application that work together to detect if AdBlock is active and request that the user pause AdBlock for the site. The main components are:",[700,33890,33891],{},[79,33892,33893,33894],{},"A component called ",[30,33895,33896],{},"AdBlockBlocker",[76,33898,33899],{},[79,33900,33901],{},"This component is used in the default layout, so it is included in all pages on briancaffey.github.io",[700,33903,33904],{"start":488},[79,33905,33906,33907],{},"Vuex store module called ",[30,33908,33909],{},"adblock",[76,33911,33912,33915],{},[79,33913,33914],{},"this module is used to keep track of a boolean value that indicates if AdBlock is enabled",[79,33916,33917],{},"the module also has a simple getter and a mutation for turning adBlock to true or false",[700,33919,33920],{"start":500},[79,33921,33922,33923,33926],{},"Some logic in the ",[30,33924,33925],{},"default.vue"," layout that is used for almost all of the pages on my site",[76,33928,33929],{},[79,33930,33931],{},"the getter from the Vuex store is used here to either show regular content or a message that the user needs to pause AdBlock",[700,33933,33934],{"start":509},[79,33935,33936],{},"A component/page to display when AdBlock is enabled",[76,33938,33939,33942],{},[79,33940,33941],{},"this component asks the user to please pause AdBlock",[79,33943,33944,33945],{},"I named this component ",[30,33946,33947],{},"PleaseDisableAdblock.vue",[700,33949,33950],{"start":517},[79,33951,33952],{},"localStorage",[76,33954,33955],{},[79,33956,33957],{},"this is used to keep track of the presence of an AdBlocker that is blocking ads",[11,33959,33960],{},"Here's an overview of each part:",[11,33962,33963],{},[15,33964,33965],{},"AdBlockBlocker.vue",[11,33967,33968,33969,33972],{},"This is the key part of how the AdBlock detection works. If the client is unable to download the ",[30,33970,33971],{},"adsbygoogle.js"," file, then that indicates that the user is using AdBlock.",[459,33974,33976],{"className":19811,"code":33975,"language":19813,"meta":464,"style":464},"\u003Ctemplate>\n  \u003Cdiv />\n\u003C/template>\n\n\u003Cscript>\nexport default {\n  mounted () {\n    // if adblock is detected through a value that is set local storage, then show the AdBlock message\n    // the `adblockEnabled` value is set in the catch block of the `detectAdBlock` method\n    // (see adblock/setAdblockEnabled mutation)\n    if (JSON.parse(localStorage.getItem('adblockEnabled')) === true) {\n      this.$store.commit('adblock/setAdblockEnabled', true)\n    }\n    // check to see if the URL can be accessed on a 5 second interval\n    setInterval(() => {\n      this.detectAdBlock()\n    }, 5000)\n  },\n  methods: {\n    async  detectAdBlock () {\n      // this is a URL that should be blocked by AdBlock\n      const googleAdUrl = 'https://pagead2.googlesyndication.com/pagead/js/adsbygoogle.js'\n      // make a request to the above URL\n      await fetch(new Request(googleAdUrl)).then((_) => {\n        // isAdblockEnabled is false by default\n        // Check to see if isAblockEnabled was set to true by a previous request\n        if (this.$store.getters['adblock/isAdblockEnabled'] === true) {\n          // if the request was successful, then the user does not have AdBlock enabled,\n          // so we can set isAdblockEnabled to false using the setAdblockEnabled mutation\n          // this mutation will also set the `adblockEnabled` value in local storage to \"false\"\n          // `adblockEnabled` be `JSON.parse`d since it is saved in localStorage as a string\n          this.$store.commit('adblock/setAdblockEnabled', false)\n          window.localStorage.setItem('adblockEnabled', 'false')\n          // do a full reload of the page\n          window.location.reload()\n        }\n      }).catch((_) => {\n        // if the request was unsuccessful, then the user has AdBlock enabled.\n        // we can set isAdblockEnabled to true using the setAdblockEnabled mutation\n        // this will also set the `adblockEnabled` value in local storage to \"true\"\n        this.$store.commit('adblock/setAdblockEnabled', true)\n        window.localStorage.setItem('adblockEnabled', 'true')\n      })\n    }\n  }\n}\n\u003C/script>\n",[30,33977,33978,33987,33998,34006,34010,34018,34026,34034,34039,34044,34049,34082,34104,34108,34113,34125,34136,34145,34150,34155,34165,34170,34183,34188,34221,34226,34231,34253,34258,34263,34268,34273,34292,34311,34316,34326,34330,34348,34353,34358,34363,34382,34400,34405,34409,34413,34417],{"__ignoreMap":464},[151,33979,33980,33982,33985],{"class":469,"line":470},[151,33981,3613],{"class":503},[151,33983,33984],{"class":14368},"template",[151,33986,3742],{"class":503},[151,33988,33989,33992,33994,33996],{"class":469,"line":488},[151,33990,33991],{"class":503},"  \u003C",[151,33993,23950],{"class":14368},[151,33995,24721],{"class":6607},[151,33997,3742],{"class":503},[151,33999,34000,34002,34004],{"class":469,"line":500},[151,34001,19966],{"class":503},[151,34003,33984],{"class":14368},[151,34005,3742],{"class":503},[151,34007,34008],{"class":469,"line":509},[151,34009,1090],{"emptyLinePlaceholder":609},[151,34011,34012,34014,34016],{"class":469,"line":517},[151,34013,3613],{"class":503},[151,34015,19822],{"class":14368},[151,34017,3742],{"class":503},[151,34019,34020,34022,34024],{"class":469,"line":534},[151,34021,1870],{"class":1869},[151,34023,19470],{"class":1869},[151,34025,19833],{"class":503},[151,34027,34028,34031],{"class":469,"line":1413},[151,34029,34030],{"class":473},"  mounted",[151,34032,34033],{"class":503}," () {\n",[151,34035,34036],{"class":469,"line":1418},[151,34037,34038],{"class":1527},"    // if adblock is detected through a value that is set local storage, then show the AdBlock message\n",[151,34040,34041],{"class":469,"line":2462},[151,34042,34043],{"class":1527},"    // the `adblockEnabled` value is set in the catch block of the `detectAdBlock` method\n",[151,34045,34046],{"class":469,"line":2471},[151,34047,34048],{"class":1527},"    // (see adblock/setAdblockEnabled mutation)\n",[151,34050,34051,34053,34055,34057,34059,34061,34064,34067,34069,34072,34075,34078,34080],{"class":469,"line":2480},[151,34052,23327],{"class":1869},[151,34054,129],{"class":503},[151,34056,27136],{"class":12360},[151,34058,643],{"class":503},[151,34060,29302],{"class":473},[151,34062,34063],{"class":503},"(localStorage.",[151,34065,34066],{"class":473},"getItem",[151,34068,12386],{"class":503},[151,34070,34071],{"class":481},"'adblockEnabled'",[151,34073,34074],{"class":503},")) ",[151,34076,34077],{"class":1869},"===",[151,34079,529],{"class":477},[151,34081,23288],{"class":503},[151,34083,34084,34087,34090,34093,34095,34098,34100,34102],{"class":469,"line":2489},[151,34085,34086],{"class":15289},"      this",[151,34088,34089],{"class":503},".$store.",[151,34091,34092],{"class":473},"commit",[151,34094,12386],{"class":503},[151,34096,34097],{"class":481},"'adblock/setAdblockEnabled'",[151,34099,106],{"class":503},[151,34101,19726],{"class":477},[151,34103,3640],{"class":503},[151,34105,34106],{"class":469,"line":2497},[151,34107,9461],{"class":503},[151,34109,34110],{"class":469,"line":3140},[151,34111,34112],{"class":1527},"    // check to see if the URL can be accessed on a 5 second interval\n",[151,34114,34115,34118,34121,34123],{"class":469,"line":3149},[151,34116,34117],{"class":473},"    setInterval",[151,34119,34120],{"class":503},"(() ",[151,34122,17166],{"class":12347},[151,34124,19833],{"class":503},[151,34126,34127,34129,34131,34134],{"class":469,"line":3158},[151,34128,34086],{"class":15289},[151,34130,643],{"class":503},[151,34132,34133],{"class":473},"detectAdBlock",[151,34135,12461],{"class":503},[151,34137,34138,34141,34143],{"class":469,"line":3167},[151,34139,34140],{"class":503},"    }, ",[151,34142,23145],{"class":477},[151,34144,3640],{"class":503},[151,34146,34147],{"class":469,"line":3175},[151,34148,34149],{"class":503},"  },\n",[151,34151,34152],{"class":469,"line":3184},[151,34153,34154],{"class":503},"  methods: {\n",[151,34156,34157,34160,34163],{"class":469,"line":3193},[151,34158,34159],{"class":1869},"    async",[151,34161,34162],{"class":473},"  detectAdBlock",[151,34164,34033],{"class":503},[151,34166,34167],{"class":469,"line":3720},[151,34168,34169],{"class":1527},"      // this is a URL that should be blocked by AdBlock\n",[151,34171,34172,34175,34178,34180],{"class":469,"line":3729},[151,34173,34174],{"class":12347},"      const",[151,34176,34177],{"class":12360}," googleAdUrl",[151,34179,19865],{"class":1869},[151,34181,34182],{"class":481}," 'https://pagead2.googlesyndication.com/pagead/js/adsbygoogle.js'\n",[151,34184,34185],{"class":469,"line":3735},[151,34186,34187],{"class":1527},"      // make a request to the above URL\n",[151,34189,34190,34193,34195,34197,34200,34203,34206,34209,34212,34215,34217,34219],{"class":469,"line":3745},[151,34191,34192],{"class":1869},"      await",[151,34194,4572],{"class":473},[151,34196,12386],{"class":503},[151,34198,34199],{"class":1869},"new",[151,34201,34202],{"class":473}," Request",[151,34204,34205],{"class":503},"(googleAdUrl)).",[151,34207,34208],{"class":473},"then",[151,34210,34211],{"class":503},"((",[151,34213,34214],{"class":15210},"_",[151,34216,16995],{"class":503},[151,34218,17166],{"class":12347},[151,34220,19833],{"class":503},[151,34222,34223],{"class":469,"line":3754},[151,34224,34225],{"class":1527},"        // isAdblockEnabled is false by default\n",[151,34227,34228],{"class":469,"line":3760},[151,34229,34230],{"class":1527},"        // Check to see if isAblockEnabled was set to true by a previous request\n",[151,34232,34233,34235,34237,34239,34242,34245,34247,34249,34251],{"class":469,"line":3773},[151,34234,23357],{"class":1869},[151,34236,129],{"class":503},[151,34238,23252],{"class":15289},[151,34240,34241],{"class":503},".$store.getters[",[151,34243,34244],{"class":481},"'adblock/isAdblockEnabled'",[151,34246,16654],{"class":503},[151,34248,34077],{"class":1869},[151,34250,529],{"class":477},[151,34252,23288],{"class":503},[151,34254,34255],{"class":469,"line":3782},[151,34256,34257],{"class":1527},"          // if the request was successful, then the user does not have AdBlock enabled,\n",[151,34259,34260],{"class":469,"line":3791},[151,34261,34262],{"class":1527},"          // so we can set isAdblockEnabled to false using the setAdblockEnabled mutation\n",[151,34264,34265],{"class":469,"line":3803},[151,34266,34267],{"class":1527},"          // this mutation will also set the `adblockEnabled` value in local storage to \"false\"\n",[151,34269,34270],{"class":469,"line":3811},[151,34271,34272],{"class":1527},"          // `adblockEnabled` be `JSON.parse`d since it is saved in localStorage as a string\n",[151,34274,34275,34278,34280,34282,34284,34286,34288,34290],{"class":469,"line":3820},[151,34276,34277],{"class":15289},"          this",[151,34279,34089],{"class":503},[151,34281,34092],{"class":473},[151,34283,12386],{"class":503},[151,34285,34097],{"class":481},[151,34287,106],{"class":503},[151,34289,9522],{"class":477},[151,34291,3640],{"class":503},[151,34293,34294,34297,34300,34302,34304,34306,34309],{"class":469,"line":7084},[151,34295,34296],{"class":503},"          window.localStorage.",[151,34298,34299],{"class":473},"setItem",[151,34301,12386],{"class":503},[151,34303,34071],{"class":481},[151,34305,106],{"class":503},[151,34307,34308],{"class":481},"'false'",[151,34310,3640],{"class":503},[151,34312,34313],{"class":469,"line":7148},[151,34314,34315],{"class":1527},"          // do a full reload of the page\n",[151,34317,34318,34321,34324],{"class":469,"line":7211},[151,34319,34320],{"class":503},"          window.location.",[151,34322,34323],{"class":473},"reload",[151,34325,12461],{"class":503},[151,34327,34328],{"class":469,"line":7273},[151,34329,23390],{"class":503},[151,34331,34332,34335,34338,34340,34342,34344,34346],{"class":469,"line":7335},[151,34333,34334],{"class":503},"      }).",[151,34336,34337],{"class":473},"catch",[151,34339,34211],{"class":503},[151,34341,34214],{"class":15210},[151,34343,16995],{"class":503},[151,34345,17166],{"class":12347},[151,34347,19833],{"class":503},[151,34349,34350],{"class":469,"line":7398},[151,34351,34352],{"class":1527},"        // if the request was unsuccessful, then the user has AdBlock enabled.\n",[151,34354,34355],{"class":469,"line":7462},[151,34356,34357],{"class":1527},"        // we can set isAdblockEnabled to true using the setAdblockEnabled mutation\n",[151,34359,34360],{"class":469,"line":7467},[151,34361,34362],{"class":1527},"        // this will also set the `adblockEnabled` value in local storage to \"true\"\n",[151,34364,34365,34368,34370,34372,34374,34376,34378,34380],{"class":469,"line":7532},[151,34366,34367],{"class":15289},"        this",[151,34369,34089],{"class":503},[151,34371,34092],{"class":473},[151,34373,12386],{"class":503},[151,34375,34097],{"class":481},[151,34377,106],{"class":503},[151,34379,19726],{"class":477},[151,34381,3640],{"class":503},[151,34383,34384,34387,34389,34391,34393,34395,34398],{"class":469,"line":7537},[151,34385,34386],{"class":503},"        window.localStorage.",[151,34388,34299],{"class":473},[151,34390,12386],{"class":503},[151,34392,34071],{"class":481},[151,34394,106],{"class":503},[151,34396,34397],{"class":481},"'true'",[151,34399,3640],{"class":503},[151,34401,34402],{"class":469,"line":7603},[151,34403,34404],{"class":503},"      })\n",[151,34406,34407],{"class":469,"line":7608},[151,34408,9461],{"class":503},[151,34410,34411],{"class":469,"line":7673},[151,34412,19957],{"class":503},[151,34414,34415],{"class":469,"line":7678},[151,34416,6274],{"class":503},[151,34418,34419,34421,34423],{"class":469,"line":7708},[151,34420,19966],{"class":503},[151,34422,19822],{"class":14368},[151,34424,3742],{"class":503},[11,34426,34427,34428,34431],{},"One other important point about this component is that it runs AdBlock detection using ",[30,34429,34430],{},"setInterval",", meaning that it will check if AdBlock is enabled every few seconds while a user is on my site.",[76,34433,34434,34440],{},[79,34435,34436,34437,34439],{},"If AdBlock is enabled, the request will fail and the Vuex store value will be updated, which will cause the site to display the ",[30,34438,33947],{}," component.",[79,34441,34442,34443,643],{},"Id AdBlock is not enabled, then the file will be read from disk cache and the Vuex store value will remain ",[30,34444,9522],{},[11,34446,34447],{},[15,34448,34449],{},"Vuex Store",[11,34451,34452,34453,34456,34457,34459],{},"This is a very simple Vuex store module. The ",[30,34454,34455],{},"isAdblockEnabled"," getter will be used in the ",[30,34458,33925],{}," layout to component.",[459,34461,34463],{"className":12338,"code":34462,"language":12340,"meta":464,"style":464},"let initialValue = false\n\nif (process.window) {\n  initialValue = JSON.parse(window.localStorage.getItem('adblockEnabled')) || false\n}\nexport const state = () => ({\n  adblockEnabled: initialValue\n})\n\nexport const getters = {\n  isAdblockEnabled: state => state.adblockEnabled\n}\n\nexport const mutations = {\n  setAdblockEnabled (state, payload) {\n    state.adblockEnabled = payload\n  }\n}\n",[30,34464,34465,34478,34482,34489,34518,34522,34541,34546,34550,34554,34567,34582,34586,34590,34603,34619,34629,34633],{"__ignoreMap":464},[151,34466,34467,34470,34473,34475],{"class":469,"line":470},[151,34468,34469],{"class":12347},"let",[151,34471,34472],{"class":503}," initialValue ",[151,34474,1876],{"class":1869},[151,34476,34477],{"class":477}," false\n",[151,34479,34480],{"class":469,"line":488},[151,34481,1090],{"emptyLinePlaceholder":609},[151,34483,34484,34486],{"class":469,"line":500},[151,34485,17218],{"class":1869},[151,34487,34488],{"class":503}," (process.window) {\n",[151,34490,34491,34494,34496,34498,34500,34502,34505,34507,34509,34511,34513,34516],{"class":469,"line":509},[151,34492,34493],{"class":503},"  initialValue ",[151,34495,1876],{"class":1869},[151,34497,29297],{"class":12360},[151,34499,643],{"class":503},[151,34501,29302],{"class":473},[151,34503,34504],{"class":503},"(window.localStorage.",[151,34506,34066],{"class":473},[151,34508,12386],{"class":503},[151,34510,34071],{"class":481},[151,34512,34074],{"class":503},[151,34514,34515],{"class":1869},"||",[151,34517,34477],{"class":477},[151,34519,34520],{"class":469,"line":517},[151,34521,6274],{"class":503},[151,34523,34524,34526,34529,34532,34534,34536,34538],{"class":469,"line":534},[151,34525,1870],{"class":1869},[151,34527,34528],{"class":12347}," const",[151,34530,34531],{"class":473}," state",[151,34533,19865],{"class":1869},[151,34535,20223],{"class":503},[151,34537,17166],{"class":12347},[151,34539,34540],{"class":503}," ({\n",[151,34542,34543],{"class":469,"line":1413},[151,34544,34545],{"class":503},"  adblockEnabled: initialValue\n",[151,34547,34548],{"class":469,"line":1418},[151,34549,19610],{"class":503},[151,34551,34552],{"class":469,"line":2462},[151,34553,1090],{"emptyLinePlaceholder":609},[151,34555,34556,34558,34560,34563,34565],{"class":469,"line":2471},[151,34557,1870],{"class":1869},[151,34559,34528],{"class":12347},[151,34561,34562],{"class":12360}," getters",[151,34564,19865],{"class":1869},[151,34566,19833],{"class":503},[151,34568,34569,34572,34574,34577,34579],{"class":469,"line":2480},[151,34570,34571],{"class":473},"  isAdblockEnabled",[151,34573,6208],{"class":503},[151,34575,34576],{"class":15210},"state",[151,34578,20832],{"class":12347},[151,34580,34581],{"class":503}," state.adblockEnabled\n",[151,34583,34584],{"class":469,"line":2489},[151,34585,6274],{"class":503},[151,34587,34588],{"class":469,"line":2497},[151,34589,1090],{"emptyLinePlaceholder":609},[151,34591,34592,34594,34596,34599,34601],{"class":469,"line":3140},[151,34593,1870],{"class":1869},[151,34595,34528],{"class":12347},[151,34597,34598],{"class":12360}," mutations",[151,34600,19865],{"class":1869},[151,34602,19833],{"class":503},[151,34604,34605,34608,34610,34612,34614,34617],{"class":469,"line":3149},[151,34606,34607],{"class":473},"  setAdblockEnabled",[151,34609,129],{"class":503},[151,34611,34576],{"class":15210},[151,34613,106],{"class":503},[151,34615,34616],{"class":15210},"payload",[151,34618,23288],{"class":503},[151,34620,34621,34624,34626],{"class":469,"line":3158},[151,34622,34623],{"class":503},"    state.adblockEnabled ",[151,34625,1876],{"class":1869},[151,34627,34628],{"class":503}," payload\n",[151,34630,34631],{"class":469,"line":3167},[151,34632,19957],{"class":503},[151,34634,34635],{"class":469,"line":3175},[151,34636,6274],{"class":503},[11,34638,34639],{},[15,34640,34641],{},"default.vue layout logic",[459,34643,34647],{"className":34644,"code":34645,"language":34646,"meta":464,"style":464},"language-vue-html shiki shiki-themes github-light github-dark monokai","\u003Ctemplate>\n  \u003Cdiv>\n    \u003CNavigation />\n    \u003CPleaseDisableAdblock v-if=\"$store.getters['adblock/isAdblockEnabled']\" />\n    \u003CNuxt v-else />\n    \u003CAdBlockerBlocker />\n\n    \u003CFooter />\n  \u003C/div>\n\u003C/template>\n","vue-html",[30,34648,34649,34657,34665,34676,34701,34713,34722,34726,34737,34746],{"__ignoreMap":464},[151,34650,34651,34653,34655],{"class":469,"line":470},[151,34652,3613],{"class":503},[151,34654,33984],{"class":14368},[151,34656,3742],{"class":503},[151,34658,34659,34661,34663],{"class":469,"line":488},[151,34660,33991],{"class":503},[151,34662,23950],{"class":14368},[151,34664,3742],{"class":503},[151,34666,34667,34670,34673],{"class":469,"line":500},[151,34668,34669],{"class":503},"    \u003C",[151,34671,34672],{"class":14368},"Navigation",[151,34674,34675],{"class":503}," />\n",[151,34677,34678,34680,34683,34686,34688,34690,34693,34695,34697,34699],{"class":469,"line":509},[151,34679,34669],{"class":503},[151,34681,34682],{"class":14368},"PleaseDisableAdblock",[151,34684,34685],{"class":1869}," v-if",[151,34687,1876],{"class":503},[151,34689,8592],{"class":4828},[151,34691,34692],{"class":503},"$store.getters[",[151,34694,34244],{"class":481},[151,34696,8582],{"class":503},[151,34698,8592],{"class":4828},[151,34700,34675],{"class":503},[151,34702,34703,34705,34708,34711],{"class":469,"line":517},[151,34704,34669],{"class":503},[151,34706,34707],{"class":14368},"Nuxt",[151,34709,34710],{"class":1869}," v-else",[151,34712,34675],{"class":503},[151,34714,34715,34717,34720],{"class":469,"line":534},[151,34716,34669],{"class":503},[151,34718,34719],{"class":14368},"AdBlockerBlocker",[151,34721,34675],{"class":503},[151,34723,34724],{"class":469,"line":1413},[151,34725,1090],{"emptyLinePlaceholder":609},[151,34727,34728,34730,34733,34735],{"class":469,"line":1418},[151,34729,34669],{"class":503},[151,34731,34732],{"class":14368},"Footer",[151,34734,24721],{"class":6607},[151,34736,3742],{"class":503},[151,34738,34739,34742,34744],{"class":469,"line":2462},[151,34740,34741],{"class":503},"  \u003C/",[151,34743,23950],{"class":14368},[151,34745,3742],{"class":503},[151,34747,34748,34750,34752],{"class":469,"line":2471},[151,34749,19966],{"class":503},[151,34751,33984],{"class":14368},[151,34753,3742],{"class":503},[11,34755,34756],{},[15,34757,34758],{},"Content to show to request that a user pauses AdBlock",[11,34760,34761,34762,643],{},"When users has AdBlock enabled, I show a simple message that asks them to please disable AdBlock. I also want to invite people who do not wish to disabled AdBlock to read my blog directly on GitHub, or to read it without ads by cloning or forking my repo, building it and running it in development mode with ",[30,34763,12175],{},[56,34765,34767],{"id":34766},"why-am-i-adding-ads-and-an-adblock-detector-to-my-site","Why am I adding ads and an AdBlock detector to my site?",[11,34769,34770],{},"Being asked to pause AdBlock is increasingly common. I have noticed that the experience is not the same on each site that requests AdBlock be paused. For example:",[76,34772,34773,34776,34779],{},[79,34774,34775],{},"a site might ask you to pause AdBlock, but you have the option to continue without pausing AdBlock",[79,34777,34778],{},"a site shows you a preview of an article and asks you to unpause AdBlock to see the full article",[79,34780,34781],{},"all site content is hidden if you are using AdBlock, and a pop-up message ask you to \"Continue to site\" once you have paused AdBlock.",[11,34783,34784,34785,34788,34789,34792],{},"I remember being able to delete AdBlock detection modals and backgrounds on some sites as a way of getting around ad block detectors. This is as easy as ",[30,34786,34787],{},"Cmd + Shift + C",", click on the element that is blocking the page content and ",[30,34790,34791],{},"delete"," the selected element.",[11,34794,34795],{},"One of the reasons I wanted to implement AdBlock detection is to see what is possible from the perspective of user experience (UX). Ideally, here's how I want to the \"please disable AdBlock\" experience to work on my site:",[700,34797,34798,34801,34804,34807,34810,34813,34816],{},[79,34799,34800],{},"A user visits my site with AdBlock enabled",[79,34802,34803],{},"For a few seconds, the user can start reading the content of the article",[79,34805,34806],{},"The page content is replaced with a message that says \"Please disable AdBlock\" and some other links that people may find interesting or helpful.",[79,34808,34809],{},"The user goes into the AdBlock extension and pauses AdBlock on my site",[79,34811,34812],{},"The original page content is then displayed with ads",[79,34814,34815],{},"If the user re-enables AdBlock shortly after pausing it, then the \"Please disable AdBlock\" message should be displayed again",[79,34817,34818],{},"If a u",[11,34820,34821,34822,34824],{},"I don't want to ask the user to press a button or make the user think that they need to refresh the page. This is why I'm using ",[30,34823,34430],{},", I continuously make requests to the Google Ads JavaScript file that will be used to detect if AdBlock enabled.",[11,34826,34827],{},"I'm happy to pause my AdBlock for smaller sites that ask me to, or for newspaper sites that are supported by advertising, and I'm assuming that people visiting my site will also be OK with pausing AdBlock as a way of thanking me for the work that goes into what I share on my blog.",[11,34829,34830],{},"I'm mostly curious to see what happens to my site's traffic, and to see what impact it could have on the earnings I make from AdSense.",[11,34832,34833],{},"Some of my questions are:",[76,34835,34836,34839,34842],{},[79,34837,34838],{},"What will happen to the bounce rate if I request that AdBlock users pause AdBlock for my site?",[79,34840,34841],{},"What is the most effective amount of time to wait before requesting that a new user pause AdBlock for my site",[79,34843,34844],{},"What else can I include on my \"Please Disable AdBlock\" page to encourage new users to pause AdBlock on my site?",[11,34846,34847],{},"I'll have a good follow-up article to share with numbers from my AdSense and Google analytics.",[736,34849,34851],{"id":34850},"one-small-issue","One small issue",[11,34853,34854],{},"While ads are working on my site, I did notice that a console error related to AdSense:",[459,34856,34859],{"className":34857,"code":34858,"language":997},[995],"K {message: \"adsbygoogle.push() error: Only one 'enable_page_level_ads' allowed per page.\", name: 'TagError', pbr: true, stack: \"TagError: adsbygoogle.push() error: Only one 'enab…agead/js/adsbygoogle.js?client=ca-google:77:1130)\"}\nmessage: \"adsbygoogle.push() error: Only one 'enable_page_level_ads' allowed per page.\"\nname: \"TagError\"\npbr: true\nstack: \"TagError: adsbygoogle.push() error: Only one 'enable_page_level_ads' allowed per page.\n    at go (http://pagead2.googlesyndication.com/pagead/js/adsbygoogle.js?client=ca-google:219:326)\n    at fo (http://pagead2.googlesyndication.com/pagead/js/adsbygoogle.js?client=ca-google:218:788)\n    at mo (http://pagead2.googlesyndication.com/pagead/js/adsbygoogle.js?client=ca-google:225:365)\n    at c (http://pagead2.googlesyndication.com/pagead/js/adsbygoogle.js?client=ca-google:226:38)\n    at no (http://pagead2.googlesyndication.com/pagead/js/adsbygoogle.js?client=ca-google:226:156)\n    at yo (http://pagead2.googlesyndication.com/pagead/js/adsbygoogle.js?client=ca-google:235:248)\n    at oo (http://pagead2.googlesyndication.com/pagead/js/adsbygoogle.js?client=ca-google:232:89)\n    at http://pagead2.googlesyndication.com/pagead/js/adsbygoogle.js?client=ca-google:227:47\n    at Od.aa.ma (http://pagead2.googlesyndication.com/pagead/js/adsbygoogle.js?client=ca-google:64:802)\n    at Jf (http://pagead2.googlesyndication.com/pagead/js/adsbygoogle.js?client=ca-google:77:1130)\"\n",[30,34860,34858],{"__ignoreMap":464},[11,34862,34863,34864,34868],{},"Here's a related issue on GitHub: ",[20,34865,34866],{"href":34866,"rel":34867},"https://github.com/nuxt-community/google-adsense-module/issues/141",[24],". I'm still not sure how to fix this issue. If anyone has any ideas, please let me know!",[11,34870,34871,34872,643],{},"If you are interested in following my progress, feel free to subscribe to my MailChimp newsletter by filling out the form in the footer of my website, or by following me on any of the accounts listed on ",[20,34873,34876],{"href":34874,"rel":34875},"https://briancaffey.github.io/contact",[24],"briancaffey.github.io/contact",[589,34878,34879],{},"html pre.shiki code .s8-w5, html code.shiki .s8-w5{--shiki-default:#6A737D;--shiki-dark:#6A737D;--shiki-sepia:#88846F}html pre.shiki code .s_OQ2, html code.shiki .s_OQ2{--shiki-default:#6F42C1;--shiki-dark:#B392F0;--shiki-sepia:#F8F8F2}html pre.shiki code .sMOD_, html code.shiki .sMOD_{--shiki-default:#24292E;--shiki-dark:#E1E4E8;--shiki-sepia:#F8F8F2}html pre.shiki code .sstjo, html code.shiki .sstjo{--shiki-default:#032F62;--shiki-dark:#9ECBFF;--shiki-sepia:#E6DB74}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html .sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html.sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html pre.shiki code .s5clZ, html code.shiki .s5clZ{--shiki-default:#22863A;--shiki-dark:#85E89D;--shiki-sepia:#F92672}html pre.shiki code .st05x, html code.shiki .st05x{--shiki-default:#B31D28;--shiki-default-font-style:italic;--shiki-dark:#FDAEB7;--shiki-dark-font-style:italic;--shiki-sepia:#F44747;--shiki-sepia-font-style:inherit}html pre.shiki code .sC2Qs, html code.shiki .sC2Qs{--shiki-default:#D73A49;--shiki-dark:#F97583;--shiki-sepia:#F92672}html pre.shiki code .srTi1, html code.shiki .srTi1{--shiki-default:#6F42C1;--shiki-dark:#B392F0;--shiki-sepia:#A6E22E}html pre.shiki code .sXSQT, html code.shiki .sXSQT{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#F8F8F2}html pre.shiki code .s7F3e, html code.shiki .s7F3e{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#AE81FF}html pre.shiki code .sP7S_, html code.shiki .sP7S_{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#FD971F}html pre.shiki code .sq6CD, html code.shiki .sq6CD{--shiki-default:#D73A49;--shiki-default-font-style:inherit;--shiki-dark:#F97583;--shiki-dark-font-style:inherit;--shiki-sepia:#66D9EF;--shiki-sepia-font-style:italic}html pre.shiki code .sTHNf, html code.shiki .sTHNf{--shiki-default:#E36209;--shiki-default-font-style:inherit;--shiki-dark:#FFAB70;--shiki-dark-font-style:inherit;--shiki-sepia:#FD971F;--shiki-sepia-font-style:italic}html pre.shiki code .sinWB, html code.shiki .sinWB{--shiki-default:#032F62;--shiki-dark:#9ECBFF;--shiki-sepia:#F8F8F2}",{"title":464,"searchDepth":488,"depth":488,"links":34881},[34882,34883,34887],{"id":33623,"depth":488,"text":33624},{"id":33864,"depth":488,"text":33865,"children":34884},[34885,34886],{"id":33871,"depth":500,"text":33872},{"id":33884,"depth":500,"text":33885},{"id":34766,"depth":488,"text":34767,"children":34888},[34889],{"id":34850,"depth":500,"text":34851},"2021-10-31","This article describes how I added AdSense to my personal website and how I request that site visitors pause AdBlock while reading my blog",[34893,34895,34897,34899,34901,34903],{"link":34894,"site":25725},"https://news.ycombinator.com/item?id=29143003",{"link":34896,"site":23769},"https://www.reddit.com/r/Nuxt/comments/qowjbu/how_and_why_i_added_adsense_and_a_custom_adblock/",{"link":34898,"site":10715},"https://dev.to/briancaffey/how-and-why-i-added-adsense-and-an-adblock-detector-to-my-personal-website-5ag3",{"link":34900,"site":30109},"https://medium.com/@briancaffey/how-and-why-i-added-adsense-and-an-adblock-detector-to-my-personal-website-b210d7f45e22",{"link":34902,"site":30112},"https://briancaffey.hashnode.dev/how-and-why-i-added-adsense-and-an-adblock-detector-to-my-personal-website",{"link":34904,"site":30115},"https://briancaffey.substack.com/p/how-and-why-i-added-adsense-and-an","https://briancaffey.github.io/static/ab_cover.png",{},"/2021/10/31/how-and-why-i-added-adsense-and-adblock-detector-to-my-website",{"title":33600,"description":34891},"2021/10/31/how-and-why-i-added-adsense-and-adblock-detector-to-my-website",[11803,34911,34912,33909],"ads","adsense","IzXWdofqPP_XiHYSa9Bbhmm1HA4hJ-cafl6Xd8qKX40",{"id":34915,"title":34916,"body":34917,"comments":602,"date":35948,"description":35949,"draft":602,"extension":605,"external":35950,"image":35412,"meta":35958,"navigation":609,"path":35959,"seo":35960,"stem":35961,"tags":35962,"__hash__":35965},"blog/2021/10/02/how-i-write-and-share-technical-software-development-articles-in-2021.md","How I write and share technical software development articles in 2021",{"type":8,"value":34918,"toc":35914},[34919,34922,34925,34940,34947,34956,34960,34977,34980,34983,35246,35264,35268,35271,35275,35283,35294,35298,35301,35307,35310,35316,35319,35336,35342,35348,35358,35362,35365,35371,35381,35385,35388,35391,35398,35404,35407,35413,35417,35420,35428,35508,35511,35525,35529,35539,35543,35550,35554,35557,35561,35564,35567,35570,35580,35584,35591,35594,35596,35602,35606,35617,35619,35624,35630,35643,35645,35648,35652,35671,35675,35678,35684,35687,35690,35696,35703,35706,35712,35723,35725,35731,35734,35740,35742,35748,35755,35761,35765,35772,35775,35781,35784,35787,35793,35795,35801,35803,35813,35819,35822,35829,35833,35836,35839,35842,35853,35864,35889,35899,35905,35908,35911],[11,34920,34921],{},"This article describes how I write and share technical articles on my personal website and other developer websites and technical article aggregation sites.",[11,34923,34924],{},"This article is broken into three sections:",[700,34926,34927,34934,34937],{},[79,34928,34929,34930],{},"How I use Nuxt.js to build ",[20,34931,34933],{"href":19426,"rel":34932},[24],"my personal website",[79,34935,34936],{},"How I share my articles on other development sites and article aggregators",[79,34938,34939],{},"Bonus content, project plug and conclusion",[56,34941,34943,34944,34946],{"id":34942},"building-briancaffeygithubio-with-nuxtjs","Building ",[30,34945,662],{}," with Nuxt.js",[11,34948,34949,34950,34955],{},"I first started writing my personal website on GitHub pages using a static website builder called ",[20,34951,34954],{"href":34952,"rel":34953},"https://jekyllrb.com/docs/github-pages/",[24],"Jekyll",". Jekyll is a great tool for getting started with building a personal portfolio or technical blog, and it served me well for several years. I eventually changed my static site generation tool from Jekyll to Nuxt since I wanted to learn more about Vue.js. Also, I don't know Ruby very well, and it was difficult for me to use the Jekyll template language.",[736,34957,34959],{"id":34958},"github-pages","GitHub Pages",[11,34961,34962,34963,34966,34967,34970,34971,34974,34975,643],{},"I host my website on GitHub pages at the following domain: ",[20,34964,662],{"href":19426,"rel":34965},[24],". GitHub pages is a great way to host a public site. The subdomain, ",[30,34968,34969],{},"briancaffey"," in my case, is your GitHub username. GitHub pages will serve content from a specified branch and nested folder. My website uses the branch ",[30,34972,34973],{},"gh-pages"," and the root folder ",[30,34976,19883],{},[736,34978,34979],{"id":30125},"GitHub Actions",[11,34981,34982],{},"I use GitHub actions to deploy my website to GitHub pages. Here's the file that sets up the GitHub action that builds and deploys my site:",[459,34984,34986],{"className":21928,"code":34985,"language":21930,"meta":464,"style":464},"name: github pages\n\non:\n  push:\n    branches:\n      - master\n  pull_request:\n\njobs:\n  deploy:\n    runs-on: ubuntu-latest\n    steps:\n      - uses: actions/checkout@v2\n\n      - name: Setup Node\n        uses: actions/setup-node@v2\n        with:\n          node-version: \"14\"\n\n      - name: Cache dependencies\n        uses: actions/cache@v2\n        with:\n          path: ~/.npm\n          key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}\n          restore-keys: |\n            ${{ runner.os }}-node-\n\n      - run: yarn\n      - run: yarn lint\n      - run: yarn generate\n\n      - name: deploy\n        uses: peaceiris/actions-gh-pages@v3\n        with:\n          github_token: ${{ secrets.GITHUB_TOKEN }}\n          publish_dir: ./docs\n",[30,34987,34988,34996,35000,35006,35012,35018,35024,35030,35034,35040,35046,35054,35060,35071,35075,35085,35094,35100,35109,35113,35123,35132,35138,35146,35154,35162,35166,35170,35180,35190,35200,35204,35214,35223,35229,35237],{"__ignoreMap":464},[151,34989,34990,34992,34994],{"class":469,"line":470},[151,34991,20415],{"class":14368},[151,34993,6208],{"class":503},[151,34995,20420],{"class":481},[151,34997,34998],{"class":469,"line":488},[151,34999,1090],{"emptyLinePlaceholder":609},[151,35001,35002,35004],{"class":469,"line":500},[151,35003,20429],{"class":477},[151,35005,14372],{"class":503},[151,35007,35008,35010],{"class":469,"line":509},[151,35009,20436],{"class":14368},[151,35011,14372],{"class":503},[151,35013,35014,35016],{"class":469,"line":517},[151,35015,20443],{"class":14368},[151,35017,14372],{"class":503},[151,35019,35020,35022],{"class":469,"line":534},[151,35021,14459],{"class":503},[151,35023,20452],{"class":481},[151,35025,35026,35028],{"class":469,"line":1413},[151,35027,20457],{"class":14368},[151,35029,14372],{"class":503},[151,35031,35032],{"class":469,"line":1418},[151,35033,1090],{"emptyLinePlaceholder":609},[151,35035,35036,35038],{"class":469,"line":2462},[151,35037,20468],{"class":14368},[151,35039,14372],{"class":503},[151,35041,35042,35044],{"class":469,"line":2471},[151,35043,20475],{"class":14368},[151,35045,14372],{"class":503},[151,35047,35048,35050,35052],{"class":469,"line":2480},[151,35049,20482],{"class":14368},[151,35051,6208],{"class":503},[151,35053,20487],{"class":481},[151,35055,35056,35058],{"class":469,"line":2489},[151,35057,20492],{"class":14368},[151,35059,14372],{"class":503},[151,35061,35062,35064,35066,35068],{"class":469,"line":2497},[151,35063,14459],{"class":503},[151,35065,20501],{"class":14368},[151,35067,6208],{"class":503},[151,35069,35070],{"class":481},"actions/checkout@v2\n",[151,35072,35073],{"class":469,"line":3140},[151,35074,1090],{"emptyLinePlaceholder":609},[151,35076,35077,35079,35081,35083],{"class":469,"line":3149},[151,35078,14459],{"class":503},[151,35080,20415],{"class":14368},[151,35082,6208],{"class":503},[151,35084,20521],{"class":481},[151,35086,35087,35089,35091],{"class":469,"line":3158},[151,35088,20526],{"class":14368},[151,35090,6208],{"class":503},[151,35092,35093],{"class":481},"actions/setup-node@v2\n",[151,35095,35096,35098],{"class":469,"line":3167},[151,35097,16967],{"class":14368},[151,35099,14372],{"class":503},[151,35101,35102,35104,35106],{"class":469,"line":3175},[151,35103,20542],{"class":14368},[151,35105,6208],{"class":503},[151,35107,35108],{"class":481},"\"14\"\n",[151,35110,35111],{"class":469,"line":3184},[151,35112,1090],{"emptyLinePlaceholder":609},[151,35114,35115,35117,35119,35121],{"class":469,"line":3193},[151,35116,14459],{"class":503},[151,35118,20415],{"class":14368},[151,35120,6208],{"class":503},[151,35122,20562],{"class":481},[151,35124,35125,35127,35129],{"class":469,"line":3720},[151,35126,20526],{"class":14368},[151,35128,6208],{"class":503},[151,35130,35131],{"class":481},"actions/cache@v2\n",[151,35133,35134,35136],{"class":469,"line":3729},[151,35135,16967],{"class":14368},[151,35137,14372],{"class":503},[151,35139,35140,35142,35144],{"class":469,"line":3735},[151,35141,20582],{"class":14368},[151,35143,6208],{"class":503},[151,35145,20587],{"class":481},[151,35147,35148,35150,35152],{"class":469,"line":3745},[151,35149,20592],{"class":14368},[151,35151,6208],{"class":503},[151,35153,20597],{"class":481},[151,35155,35156,35158,35160],{"class":469,"line":3754},[151,35157,20602],{"class":14368},[151,35159,6208],{"class":503},[151,35161,20607],{"class":1869},[151,35163,35164],{"class":469,"line":3760},[151,35165,20612],{"class":481},[151,35167,35168],{"class":469,"line":3773},[151,35169,1090],{"emptyLinePlaceholder":609},[151,35171,35172,35174,35176,35178],{"class":469,"line":3782},[151,35173,14459],{"class":503},[151,35175,20623],{"class":14368},[151,35177,6208],{"class":503},[151,35179,20628],{"class":481},[151,35181,35182,35184,35186,35188],{"class":469,"line":3791},[151,35183,14459],{"class":503},[151,35185,20623],{"class":14368},[151,35187,6208],{"class":503},[151,35189,20650],{"class":481},[151,35191,35192,35194,35196,35198],{"class":469,"line":3803},[151,35193,14459],{"class":503},[151,35195,20623],{"class":14368},[151,35197,6208],{"class":503},[151,35199,20661],{"class":481},[151,35201,35202],{"class":469,"line":3811},[151,35203,1090],{"emptyLinePlaceholder":609},[151,35205,35206,35208,35210,35212],{"class":469,"line":3820},[151,35207,14459],{"class":503},[151,35209,20415],{"class":14368},[151,35211,6208],{"class":503},[151,35213,20676],{"class":481},[151,35215,35216,35218,35220],{"class":469,"line":7084},[151,35217,20526],{"class":14368},[151,35219,6208],{"class":503},[151,35221,35222],{"class":481},"peaceiris/actions-gh-pages@v3\n",[151,35224,35225,35227],{"class":469,"line":7148},[151,35226,16967],{"class":14368},[151,35228,14372],{"class":503},[151,35230,35231,35233,35235],{"class":469,"line":7211},[151,35232,20696],{"class":14368},[151,35234,6208],{"class":503},[151,35236,20701],{"class":481},[151,35238,35239,35241,35243],{"class":469,"line":7273},[151,35240,20706],{"class":14368},[151,35242,6208],{"class":503},[151,35244,35245],{"class":481},"./docs\n",[11,35247,35248,35249,35251,35252,35254,35255,35258,35259,35261,35262,643],{},"When changes are pushed to the ",[30,35250,12512],{}," branch, this GitHub Action runs. It lints the code, builds the site with ",[30,35253,12471],{}," and then the ",[30,35256,35257],{},"peaceiris/actions-gh-pages@v3"," GitHub Action commits only the build artifacts to the ",[30,35260,34973],{}," branch where the content is served on ",[30,35263,662],{},[736,35265,35267],{"id":35266},"nuxtjs-framework","Nuxt.js Framework",[11,35269,35270],{},"Nuxt.js is a versatile Vue.js framework. It can be used to build several different types of websites, including Server Side Rendered (SSR) websites, single page applications (SPA) and static websites. I use the static website mode, also called Server Side Generation (SSG).",[736,35272,35274],{"id":35273},"content-api","Content API",[11,35276,35277,35278,35282],{},"Nuxt has a module called ",[20,35279,35274],{"href":35280,"rel":35281},"https://nuxtjs.org/api/content-api",[24]," that allows you to do the following:",[76,35284,35285,35288,35291],{},[79,35286,35287],{},"write articles in Markdown",[79,35289,35290],{},"Use Vue components in Markdown",[79,35292,35293],{},"write custom front-matter that can be used to query Markdown articles using an API",[736,35295,35297],{"id":35296},"articles-and-folder-structure","Articles and Folder Structure",[11,35299,35300],{},"Nuxt uses a folder structure that automatically generates routes and helps you organize components, pages and layouts. Here is the folder structure for my personal website's repository:",[459,35302,35305],{"className":35303,"code":35304,"language":997},[995],"$ tree -L 1 .\n.\n├── README.md\n├── assets  \u003C-- compiled assets\n├── components \u003C-- Vue components used in the site\n├── content  \u003C-- contains Markdown files for articles\n├── i18n  \u003C-- contains translations\n├── jsconfig.json\n├── layouts  \u003C-- layouts used for each page\n├── middleware  \u003C-- I'm not using this folder\n├── node_modules\n├── nuxt.config.js  \u003C-- config file\n├── package.json\n├── pages  \u003C-- directory structure defines URLs for site pages\n├── plugins  \u003C-- plugins\n├── static  \u003C-- files in this folder are served as-is\n├── store  \u003C-- Vuex store\n├── tailwind.config.js\n└── yarn.lock\n",[30,35306,35304],{"__ignoreMap":464},[11,35308,35309],{},"The content folder has the following structure:",[459,35311,35314],{"className":35312,"code":35313,"language":997},[995],"$ tree -L 3 content\ncontent/\n├── 2016\n│   └── 04\n│       └── 07\n│           └── my-article.md\n├── 2017\n│   └── 01\n│      └── 01\n│           └── my-other-article.md\n...\n└── projects\n│   └── my-project.md\n",[30,35315,35313],{"__ignoreMap":464},[11,35317,35318],{},"This will produce the following routes:",[76,35320,35321,35326,35331],{},[79,35322,35323],{},[30,35324,35325],{},"/2016/04/07/my-article.html",[79,35327,35328],{},[30,35329,35330],{},"/2017/01/01/my-other-article.html",[79,35332,35333],{},[30,35334,35335],{},"/projects/my-project.html",[11,35337,35338,35339,12163],{},"Here's the folder structure for the ",[30,35340,35341],{},"pages",[459,35343,35346],{"className":35344,"code":35345,"language":997},[995],"$ tree -L 4 pages\npages\n├── README.md\n├── _year\n│   └── _month\n│       └── _day\n│           └── _slug.vue\n├── blog\n│   ├── index.vue\n│   └── tags\n│       ├── _tag.vue\n│       └── index.vue\n├── confirm-subscription.vue\n├── contact\n│   └── index.vue\n├── drafts\n│   └── index.vue\n├── index.vue\n├── projects\n│   ├── _slug.vue\n│   └── index.vue\n└── thank-you.vue\n",[30,35347,35345],{"__ignoreMap":464},[11,35349,35350,35351,35353,35354,35357],{},"Directories in the ",[30,35352,35341],{}," directory starting with an underscore (like ",[30,35355,35356],{},"_year",") can be used as URL parameters.",[736,35359,35361],{"id":35360},"markdown-and-vue-components","Markdown and Vue Components",[11,35363,35364],{},"One nice feature of Nuxt and the Nuxt Content module is the ability to use Vue components directly in Markdown files. Here's an example of a Vue component in a Markdown file:",[459,35366,35369],{"className":35367,"code":35368,"language":997},[995],"# Markdown with Vue components\n\nThis is some content\n\n\u003Cmy-component />\n\nThis is more content.\n",[30,35370,35368],{"__ignoreMap":464},[11,35372,35373,35374,35377,35378,643],{},"Vue components used in Markdown files must be included in the ",[30,35375,35376],{},"components/global"," directory. Here's an article on my blog where I used Vue components to show interactive graphs: ",[20,35379,20785],{"href":20785,"rel":35380},[24],[736,35382,35384],{"id":35383},"images","Images",[11,35386,35387],{},"Images are an important part of the articles that I write. Each article has an optional cover image. I try to include a cover image for each of the articles I write. I mostly use Gimp to create the cover images.",[11,35389,35390],{},"The cover image for this article was made with Inkscape. I have also used Gimp and Blender to generate images for my blog.",[11,35392,35393,35394,35397],{},"Including images in the body of an article is pretty simple on my personal site. First I need to add the image to the ",[30,35395,35396],{},"static"," directory, and then I can reference that image using the following syntax:",[459,35399,35402],{"className":35400,"code":35401,"language":997},[995],"![alt text](/static/path/to/my-image.png)\n",[30,35403,35401],{"__ignoreMap":464},[11,35405,35406],{},"Here's the image for this article:",[11,35408,35409],{},[2718,35410],{"alt":35411,"src":35412},"Writing and publishing dev articles","/static/dev-sites.png",[736,35414,35416],{"id":35415},"markdown-front-matter","Markdown front-matter",[11,35418,35419],{},"Front-matter is a way to define metadata for a Markdown file. It is used to define the title, description, and other metadata that can be used to query articles.",[11,35421,35422,35423,208],{},"Here's whe front-matter for this article on my personal site ",[20,35424,35427],{"href":35425,"rel":35426},"https://briancaffey.github.io/2021/10/02/sharing-a-dev-article-everywhere",[24],"briancaffey.github.io/2021/10/02/sharing-a-dev-article-everywhere",[459,35429,35431],{"className":21928,"code":35430,"language":21930,"meta":464,"style":464},"---\ntitle: How I write and share technical development articles in 2021\ndate: '2021-10-02'\ndescription: This article describes how to write and share technical articles in 2021\ntags:\n  - nuxt\n  - vue\n  - publishing\n  - writing\ndraft: true\n---\n",[30,35432,35433,35437,35446,35455,35464,35470,35476,35482,35489,35496,35504],{"__ignoreMap":464},[151,35434,35435],{"class":469,"line":470},[151,35436,19628],{"class":19627},[151,35438,35439,35441,35443],{"class":469,"line":488},[151,35440,19633],{"class":14368},[151,35442,6208],{"class":503},[151,35444,35445],{"class":481},"How I write and share technical development articles in 2021\n",[151,35447,35448,35450,35452],{"class":469,"line":500},[151,35449,19646],{"class":14368},[151,35451,6208],{"class":503},[151,35453,35454],{"class":481},"'2021-10-02'\n",[151,35456,35457,35459,35461],{"class":469,"line":509},[151,35458,19656],{"class":14368},[151,35460,6208],{"class":503},[151,35462,35463],{"class":481},"This article describes how to write and share technical articles in 2021\n",[151,35465,35466,35468],{"class":469,"line":517},[151,35467,19678],{"class":14368},[151,35469,14372],{"class":503},[151,35471,35472,35474],{"class":469,"line":534},[151,35473,19688],{"class":503},[151,35475,19698],{"class":481},[151,35477,35478,35480],{"class":469,"line":1413},[151,35479,19688],{"class":503},[151,35481,19691],{"class":481},[151,35483,35484,35486],{"class":469,"line":1418},[151,35485,19688],{"class":503},[151,35487,35488],{"class":481},"publishing\n",[151,35490,35491,35493],{"class":469,"line":2462},[151,35492,19688],{"class":503},[151,35494,35495],{"class":481},"writing\n",[151,35497,35498,35500,35502],{"class":469,"line":2471},[151,35499,19721],{"class":14368},[151,35501,6208],{"class":503},[151,35503,14382],{"class":477},[151,35505,35506],{"class":469,"line":2480},[151,35507,19628],{"class":19627},[11,35509,35510],{},"I also use front-matter for the following:",[76,35512,35513,35516,35519,35522],{},[79,35514,35515],{},"OpenGraph meta tags & social sharing",[79,35517,35518],{},"links to other outlets",[79,35520,35521],{},"meta tags",[79,35523,35524],{},"article tags",[736,35526,35528],{"id":35527},"nuxt-sitemap","Nuxt Sitemap",[11,35530,17338,35531,35534,35535,35538],{},[30,35532,35533],{},"@nuxt/sitemap"," module, it is easy to generate a sitemap.xml file. This generates a static file that is available on ",[30,35536,35537],{},"/sitemap.xml",". This file is used in the Google Search Console to tell Google which pages on my site should be indexed. This helps with SEO.",[736,35540,35542],{"id":35541},"rss-feed","RSS Feed",[11,35544,35545,35546,35549],{},"An RSS feed is configured using another official Nuxt module called ",[30,35547,35548],{},"@nuxtjs/feed",". This plugin generates an RSS feed for the site in XML. An RSS feed can be used to automatically publish articles to other sites, I'll show this in the next section of this article.",[736,35551,35553],{"id":35552},"google-analytics","Google Analytics",[11,35555,35556],{},"Google Analytics is used to track site traffic and gives insights into what content is popular, where my users are visiting from, how long they spend browsing my site and other helpful metrics. It is likely that many readers of my site may have disabled Google Analytics in their browsers.",[736,35558,35560],{"id":35559},"google-search-console","Google Search Console",[11,35562,35563],{},"Google Search Console is another tool that is helpful from an SEO perspective.",[11,35565,35566],{},"Here is a report from Google Data Studio showing some some of the metrics that I use to analyze my site's traffic:",[589,35568,35569],{},"\n  .google-data-studio {\nposition: relative;\npadding-bottom: 56.25%;\npadding-top: 30px; height: 0; overflow: hidden;\n}\n\n.google-data-studio iframe,\n.google-data-studio object,\n.google-data-studio embed {\nposition: absolute;\ntop: 0;\nleft: 0;\nwidth: 100%;\nheight: 100%;\n}\n",[23950,35571,35574],{"className":35572},[35573],"google-data-studio",[23881,35575],{"width":35576,"height":35577,"src":35578,"frameBorder":9181,"style":35579,"allowFullScreen":609},600,450,"https://datastudio.google.com/embed/reporting/1da052a4-6d61-4dac-b85d-5e9efed870af/page/6zXD","border:0",[736,35581,35583],{"id":35582},"mailchimp","MailChimp",[11,35585,35586,35587,643],{},"I use MailChimp to build a newsletter audience. I'll be sending out a newsletter to my current audience with an update about this article. I wrote an article about how to set up MailChimp on Nuxt. I wrote an article on my blog about how I set up a form for guests to sign up for a newsletter using a MailChimp form: ",[20,35588,35589],{"href":35589,"rel":35590},"https://briancaffey.github.io/zh/2020/10/10/how-to-add-email-signup-form-to-nuxt-site-with-mailchimp.html",[24],[11,35592,35593],{},"Here's what the form looks like:",[1205,35595],{},[142,35597,35598,35599],{},"\n  ",[35600,35601],"newsletter",{},[736,35603,35605],{"id":35604},"formsubmitco","formsubmit.co",[11,35607,35608,35609,35613,35614,21027],{},"Site visitors can send me messages through an online form called ",[20,35610,35605],{"href":35611,"rel":35612},"https://formsubmit.co",[24],". I include this form on my site's ",[20,35615,21026],{"href":34874,"rel":35616},[24],[1205,35618],{},[142,35620,35598,35621],{},[35622,35623],"contact-form",{},[11,35625,35626,35628],{},[1205,35627],{},[1205,35629],{},[11,35631,35632,35633,187,35636,35639,35640,35642],{},"This two forms are examples of using Vue components in Markdown files that I mentioned earlier. Both the ",[30,35634,35635],{},"Nesletter",[30,35637,35638],{},"ContactForm"," components must be in the ",[30,35641,35376],{}," directory.",[736,35644,20987],{"id":20986},[11,35646,35647],{},"Drift is a freemium service that allows site visitors to send me messages in real time. It's a great way to get in touch with site visitors, and it can be configured so that messages go to the Drift mobile app.",[736,35649,35651],{"id":35650},"drafts","Drafts",[11,35653,35654,35655,35657,35658,35661,35662,35666,35667,35670],{},"One option in the front-matter for my blog articles is ",[30,35656,19721],{},". If an article has ",[30,35659,35660],{},"draft: true"," set in the front-matter, then the article will not be listed on the main list of blog articles on my site, and the page will not be indexed by Google. Here's where you can find the draft articles for my site: ",[20,35663,35664],{"href":35664,"rel":35665},"https://briancaffey.github.io/drafts",[24],". The articles here can be accessed publicly, but I only show them on the ",[30,35668,35669],{},"/drafts"," page which is not listed anywhere else on my site.",[56,35672,35674],{"id":35673},"publishing-on-other-outlets","Publishing on other outlets",[11,35676,35677],{},"When publishing articles from my personal website on other sites, I make sure that custom content is either removed or replaced with a link or static image that I can upload to the other site while editing the article.",[11,35679,35680,35681,35683],{},"For example, this article includes an embedded Google Data Studio report in the version that is published on ",[30,35682,662],{},". This embedded iframe will not work when posted to other platforms, so I can instead link to an anchor tag that corresponds to the location of the custom element on my site.",[11,35685,35686],{},"For this article, I have mostly tried to keep the custom content to a minimum so that it will be easy to cross publish on other sites without having to make lots of edits to the markdown. Most of the tweaking will likely have to do with preview images and other custom front-matter properties that some site (like DEV.to) support.",[736,35688,33829],{"id":35689},"devto",[11,35691,35692],{},[20,35693,35694],{"href":35694,"rel":35695},"https://dev.to/briancaffey/how-i-write-and-share-technical-software-development-articles-in-2021-27n2",[24],[11,35697,35698,35702],{},[20,35699,33829],{"href":35700,"rel":35701},"https://dev.to",[24]," is a popular site for sharing technical articles. They allow you to automatically draft articles to publish on their site by adding your site's RSS feed. This article will be published on DEV.to through the RSS feed connection that my account has with DEV.to.",[11,35704,35705],{},"DEV.to articles support their own custom front-matter properties. Here's what the front-matter for the DEV.to article looks like:",[459,35707,35710],{"className":35708,"code":35709,"language":997},[995],"---\ntitle: How I write and share technical software development articles in 2021\npublished: false\ndate: '2021-10-02'\ntags:  nuxt, vue, publishing, blogging\nimage: 'https://briancaffey.github.io/static/dev-sites.png'\ncanonical_url: https://briancaffey.github.io/2021/10/02/how-i-write-and-share-technical-software-development-articles-in-2021\n---\n",[30,35711,35709],{"__ignoreMap":464},[11,35713,35714,35715,35718,35719,35722],{},"If you don't see your article in your list of article drafts on your DEV.to dashboard, you can go into ",[30,35716,35717],{},"Settings > Extensions > Publishing to DEV Community from RSS"," and click on ",[30,35720,35721],{},"Save Feed Settings",". I think this refreshes your RSS feed in your dashboard.",[736,35724,1177],{"id":30109},[11,35726,35727],{},[20,35728,35729],{"href":35729,"rel":35730},"https://medium.com/@briancaffey/how-i-write-and-share-technical-software-development-articles-in-2021-8168d3871bf9",[24],[11,35732,35733],{},"I haven't published anything on Medium, so one of my goals for this article is to cross publish it on Medium in my first article on that platform.",[11,35735,35736],{},[20,35737,35738],{"href":35738,"rel":35739},"https://medium.com/new-story",[24],[736,35741,33835],{"id":30112},[11,35743,35744],{},[20,35745,35746],{"href":35746,"rel":35747},"https://briancaffey.hashnode.dev/how-i-write-and-share-technical-software-development-articles-in-2021",[24],[11,35749,35750,35751,643],{},"Hashnode seems very similar to DEV.to. Here's a comparison that shows some of the advantages of using Hashnode as a blogging platform over DEV.to: ",[20,35752,35753],{"href":35753,"rel":35754},"https://hashnode.com/vs/devto",[24],[11,35756,35757],{},[20,35758,35759],{"href":35759,"rel":35760},"https://hashnode.com/create/story",[24],[736,35762,35764],{"id":35763},"eggheadio","egghead.io",[11,35766,35767,35768,35771],{},"Egghead is another blogging platform that allows you to helps you ",[30,35769,35770],{},"Own Your Online Presence"," and also lets you create free and paid courses and content.",[736,35773,33846],{"id":35774},"hacker-noon",[11,35776,35777],{},[20,35778,35779],{"href":35779,"rel":35780},"https://hackernoon.com/how-to-get-your-dev-blog-noticed-in-2021",[24],[11,35782,35783],{},"Hacker Noon is another platform that I haven't used before as a writer. In order to publish an article on Hacker Noon, the article must go through an editing process. The Hacker Noon editors change the title and cover image of my article, and also added some minor editor notes.",[736,35785,33858],{"id":35786},"hacker-news",[11,35788,35789],{},[20,35790,35791],{"href":35791,"rel":35792},"https://news.ycombinator.com/item?id=28740415",[24],[736,35794,33855],{"id":30115},[11,35796,35797],{},[20,35798,35799],{"href":35799,"rel":35800},"https://briancaffey.substack.com/p/how-i-write-and-share-technical-software",[24],[736,35802,33840],{"id":23769},[11,35804,35805,35806,187,35809,35812],{},"I have shared a lot of content on different programming subreddits specific to some of the tools and frameworks I use, such as ",[30,35807,35808],{},"r/aws",[30,35810,35811],{},"r/django",". When sharing on reddit, I like to share links to my personal website with at least on comment that provides a detailed summary of the article. When sharing on",[11,35814,35815],{},[20,35816,35817],{"href":35817,"rel":35818},"https://www.reddit.com/r/webdev/comments/q0qc3l/how_i_write_and_share_technical_software/",[24],[736,35820,33832],{"id":35821},"facebook",[11,35823,35824,35825,35828],{},"Facebook has very large and active developer communities. Sometimes the communities are more fragmented than the communities on reddit. For example, there are several Nuxt communities on Facebook, but there is just one ",[30,35826,35827],{},"r/nuxt",". Similar to sharing content on reddit, I like to share links to my personal websites with detailed comments on the content of my article.",[736,35830,35832],{"id":35831},"discord-servers","Discord Servers",[11,35834,35835],{},"Discord also has some dedicated servers for software frameworks, such as Nuxt.js. Discord seems to be the official place where Vue.js community members chat in real-time. There are dedicated channels on the server for sharing articles.",[56,35837,34939],{"id":35838},"bonus-content-project-plug-and-conclusion",[11,35840,35841],{},"One more great thing about GitHub pages is that you can publish a site on any of your GitHub repositories that will be hosted on a subpath of your GitHub pages blog.",[11,35843,35844,35845,35848,35849,643],{},"I have been working updating and rewriting my Django + Vue.js + AWS reference project. It contains a documentation site that I am making with VuePress. The repo for this project is here: ",[20,35846,25799],{"href":25797,"rel":35847},[24],". This repository has it's own GitHub Pages configuration, as well as a GitHub Action to help automate the deployment of this project documentation site to GitHub Pages. The project site is currently hosted on ",[20,35850,35852],{"href":25791,"rel":35851},[24],"briancaffey.github.io/django-step-by-step/",[11,35854,35855,35856,35863],{},"The project uses a CDK construct library that I have been developing alongside of this reference project called ",[20,35857,35860],{"href":35858,"rel":35859},"https://github.com/briancaffey/django-cdk",[24],[30,35861,35862],{},"django-cdk",". It provides high-level constructs that allow you to deploy a complete stack of resources on AWS that will support your Django + Vue.js application including:",[76,35865,35866,35869,35872,35875,35878,35880,35883,35886],{},[79,35867,35868],{},"VPC networking",[79,35870,35871],{},"CloudFront & S3 for Frontend",[79,35873,35874],{},"RDS & Elasticache",[79,35876,35877],{},"ECS & EKS options",[79,35879,27052],{},[79,35881,35882],{},"Automatic migrations",[79,35884,35885],{},"Asset and Media file storage with S3",[79,35887,35888],{},"Shell access (for debugging)",[11,35890,35891,35892,35894,35895],{},"The documentation for ",[30,35893,35862],{}," can be found here: ",[20,35896,35897],{"href":35897,"rel":35898},"https://briancaffey.github.io/django-step-by-step/deploy/aws/#about-django-cdk",[24],[11,35900,35901,35902,35904],{},"You may want to split a large project's documentation site into its own site, rather than having it live on the nested path of a personal blog. Following this pattern, your GitHub pages blog can become a site that is much larger than one single Nuxt static site. ",[30,35903,662],{}," is now a hybrid Nuxt.js and VuePress site, with a subset of routes (starting with /django-step-by-step/) being served by VuePress.",[11,35906,35907],{},"As I'm writing this article, Nuxt 3 is almost one week away from a public beta. I'm excited to try upgrading this site to Nuxt 3 and trying out some of the new features that it includes.",[11,35909,35910],{},"Thanks for reading this article, wherever you may have found it on the internet!",[589,35912,35913],{},"html pre.shiki code .s5clZ, html code.shiki .s5clZ{--shiki-default:#22863A;--shiki-dark:#85E89D;--shiki-sepia:#F92672}html pre.shiki code .sMOD_, html code.shiki .sMOD_{--shiki-default:#24292E;--shiki-dark:#E1E4E8;--shiki-sepia:#F8F8F2}html pre.shiki code .sstjo, html code.shiki .sstjo{--shiki-default:#032F62;--shiki-dark:#9ECBFF;--shiki-sepia:#E6DB74}html pre.shiki code .s7F3e, html code.shiki .s7F3e{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#AE81FF}html pre.shiki code .sC2Qs, html code.shiki .sC2Qs{--shiki-default:#D73A49;--shiki-dark:#F97583;--shiki-sepia:#F92672}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html .sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html.sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html pre.shiki code .s_OQ2, html code.shiki .s_OQ2{--shiki-default:#6F42C1;--shiki-dark:#B392F0;--shiki-sepia:#F8F8F2}",{"title":464,"searchDepth":488,"depth":488,"links":35915},[35916,35935,35947],{"id":34942,"depth":488,"text":35917,"children":35918},"Building briancaffey.github.io with Nuxt.js",[35919,35920,35921,35922,35923,35924,35925,35926,35927,35928,35929,35930,35931,35932,35933,35934],{"id":34958,"depth":500,"text":34959},{"id":30125,"depth":500,"text":34979},{"id":35266,"depth":500,"text":35267},{"id":35273,"depth":500,"text":35274},{"id":35296,"depth":500,"text":35297},{"id":35360,"depth":500,"text":35361},{"id":35383,"depth":500,"text":35384},{"id":35415,"depth":500,"text":35416},{"id":35527,"depth":500,"text":35528},{"id":35541,"depth":500,"text":35542},{"id":35552,"depth":500,"text":35553},{"id":35559,"depth":500,"text":35560},{"id":35582,"depth":500,"text":35583},{"id":35604,"depth":500,"text":35605},{"id":20986,"depth":500,"text":20987},{"id":35650,"depth":500,"text":35651},{"id":35673,"depth":488,"text":35674,"children":35936},[35937,35938,35939,35940,35941,35942,35943,35944,35945,35946],{"id":35689,"depth":500,"text":33829},{"id":30109,"depth":500,"text":1177},{"id":30112,"depth":500,"text":33835},{"id":35763,"depth":500,"text":35764},{"id":35774,"depth":500,"text":33846},{"id":35786,"depth":500,"text":33858},{"id":30115,"depth":500,"text":33855},{"id":23769,"depth":500,"text":33840},{"id":35821,"depth":500,"text":33832},{"id":35831,"depth":500,"text":35832},{"id":35838,"depth":488,"text":34939},"2021-10-02","This article describes how to write and share technical articles in 2021",[35951,35952,35953,35954,35955,35956,35957],{"link":35791,"site":25725},{"link":35817,"site":23769},{"link":35694,"site":10715},{"link":35729,"site":30109},{"link":35746,"site":30112},{"link":35799,"site":30115},{"link":35779,"site":33591},{},"/2021/10/02/how-i-write-and-share-technical-software-development-articles-in-2021",{"title":34916,"description":35949},"2021/10/02/how-i-write-and-share-technical-software-development-articles-in-2021",[11803,12646,35963,35964,12647],"publishing","writing","exZf89TZZhjsXjx8SUbHOQp9XnTgmQRcb9d0EvQSbrY",{"id":35967,"title":35968,"body":35969,"comments":602,"date":36032,"description":36033,"draft":602,"extension":605,"external":606,"image":36034,"meta":36035,"navigation":609,"path":36036,"seo":36037,"stem":36038,"tags":36039,"__hash__":36042},"blog/2021/08/07/authenticating-requests-with-jwt-tokens-stored-in-httponly-cookies-in-django.md","Authenticating requests with JWT tokens stored in HTTPOnly cookies in Django",{"type":8,"value":35970,"toc":36025},[35971,35973,35976,35980,35983,35987,35990,35996,35999,36003,36006,36012,36017,36020],[56,35972,16116],{"id":16115},[11,35974,35975],{},"If you want to use JWTs to securely authenticate requests to Django REST Framework applications in a decoupled frontend JavaScript application, you can do the following: store the access token in memory and store the refresh token  in an HttpOnly cookie. The refresh token is used to request new access tokens on an regular interval.",[56,35977,35979],{"id":35978},"some-context","Some context",[11,35981,35982],{},"Django is a web framework based on the Model, Template, View (MTV) paradigm. Django is increasingly used as an API server that is coupled with a Javascript or native frontend application.",[56,35984,35986],{"id":35985},"jwt-auth-with-httponly-cookies","JWT Auth with HttpOnly Cookies",[11,35988,35989],{},"Following this guide:",[11,35991,35992],{},[20,35993,35994],{"href":35994,"rel":35995},"https://hasura.io/blog/best-practices-of-using-jwt-with-graphql/#jwt_security",[24],[11,35997,35998],{},"We can reimplement our JWT authentication setup to be more secure.",[736,36000,36002],{"id":36001},"drf-simple-jwt-modifications","DRF Simple JWT modifications",[11,36004,36005],{},"We need to change the default behavior of the views from DRF simple JWT as described in this issue:",[11,36007,36008],{},[20,36009,36010],{"href":36010,"rel":36011},"https://github.com/jazzband/djangorestframework-simplejwt/issues/71",[24],[11,36013,36014],{},[2718,36015],{"alt":20386,"src":36016},"/static/jwt-authentication.png",[11,36018,36019],{},"This diagram shows how authentication data moves between the Django backend and the Vue.js frontend running in the browser.",[11,36021,36022],{},[51,36023,36024],{},"This article should now be complete complete",{"title":464,"searchDepth":488,"depth":488,"links":36026},[36027,36028,36029],{"id":16115,"depth":488,"text":16116},{"id":35978,"depth":488,"text":35979},{"id":35985,"depth":488,"text":35986,"children":36030},[36031],{"id":36001,"depth":500,"text":36002},"2021-08-01","This article describes how you can use JWT tokens in Django applications with decoupled frontend JavaScript applications running the browser in secure way using HttpOnly cookies.","/static/djjwt/dj_jwt.png",{},"/2021/08/07/authenticating-requests-with-jwt-tokens-stored-in-httponly-cookies-in-django",{"title":35968,"description":36033},"2021/08/07/authenticating-requests-with-jwt-tokens-stored-in-httponly-cookies-in-django",[30122,12646,36040,36041],"jwt","authentication","aNmcAKycPah63ijAQpTQ4t2EZSEo2RrVYllChlVd6qk",{"id":36044,"title":36045,"body":36046,"comments":602,"date":36478,"description":36479,"draft":602,"extension":605,"external":606,"image":36480,"meta":36481,"navigation":609,"path":36482,"seo":36483,"stem":36484,"tags":36485,"__hash__":36486},"blog/2021/07/05/github-actions-for-my-nuxt-blog-hosted-on-github-pages.md","GitHub Action for my Nuxt.js blog hosted on GitHub Pages",{"type":8,"value":36047,"toc":36472},[36048,36052,36063,36066,36078,36334,36341,36343,36413,36417,36469],[56,36049,36051],{"id":36050},"github-actions-for-a-nuxtjs-blog-hosted-on-github-pages","GitHub Actions for a Nuxt.js blog hosted on GitHub Pages",[11,36053,36054,36056,36057,106,36060,36062],{},[15,36055,11825],{}," This article was published in July 2021 and references Nuxt 2, Node.js 14, and older GitHub Action versions. While the core workflow concepts remain valid, specific action versions (e.g., ",[30,36058,36059],{},"actions/checkout@v2",[30,36061,35257],{},") and Node.js versions have been updated since publication. Check official documentation for current best practices when implementing similar workflows today.",[11,36064,36065],{},"To update my blog, I usually build locally and then push changes to GitHub. This makes my git log unreadable because every deployment commit is mixed in with feature commits. This article shows how to use a GitHub Action to automate the deployment of my Nuxt.js blog to GitHub Pages, keeping the git history clean while ensuring continuous integration.",[11,36067,36068,36069,36071,36072,36074,36075,208],{},"In this GitHub repo, the following workflow is triggered on pushes to the ",[30,36070,12512],{}," branch (or ",[30,36073,26180],{}," for newer repos) which will build and deploy changes to ",[20,36076,662],{"href":19426,"rel":36077},[24],[459,36079,36080],{"className":14359,"code":34985,"language":14361,"meta":464,"style":464},[30,36081,36082,36090,36094,36100,36106,36112,36118,36124,36128,36134,36140,36148,36154,36164,36168,36178,36186,36192,36200,36204,36214,36222,36228,36236,36244,36252,36256,36260,36270,36280,36290,36294,36304,36312,36318,36326],{"__ignoreMap":464},[151,36083,36084,36086,36088],{"class":469,"line":470},[151,36085,20415],{"class":14368},[151,36087,6208],{"class":503},[151,36089,20420],{"class":481},[151,36091,36092],{"class":469,"line":488},[151,36093,1090],{"emptyLinePlaceholder":609},[151,36095,36096,36098],{"class":469,"line":500},[151,36097,20429],{"class":477},[151,36099,14372],{"class":503},[151,36101,36102,36104],{"class":469,"line":509},[151,36103,20436],{"class":14368},[151,36105,14372],{"class":503},[151,36107,36108,36110],{"class":469,"line":517},[151,36109,20443],{"class":14368},[151,36111,14372],{"class":503},[151,36113,36114,36116],{"class":469,"line":534},[151,36115,14459],{"class":503},[151,36117,20452],{"class":481},[151,36119,36120,36122],{"class":469,"line":1413},[151,36121,20457],{"class":14368},[151,36123,14372],{"class":503},[151,36125,36126],{"class":469,"line":1418},[151,36127,1090],{"emptyLinePlaceholder":609},[151,36129,36130,36132],{"class":469,"line":2462},[151,36131,20468],{"class":14368},[151,36133,14372],{"class":503},[151,36135,36136,36138],{"class":469,"line":2471},[151,36137,20475],{"class":14368},[151,36139,14372],{"class":503},[151,36141,36142,36144,36146],{"class":469,"line":2480},[151,36143,20482],{"class":14368},[151,36145,6208],{"class":503},[151,36147,20487],{"class":481},[151,36149,36150,36152],{"class":469,"line":2489},[151,36151,20492],{"class":14368},[151,36153,14372],{"class":503},[151,36155,36156,36158,36160,36162],{"class":469,"line":2497},[151,36157,14459],{"class":503},[151,36159,20501],{"class":14368},[151,36161,6208],{"class":503},[151,36163,35070],{"class":481},[151,36165,36166],{"class":469,"line":3140},[151,36167,1090],{"emptyLinePlaceholder":609},[151,36169,36170,36172,36174,36176],{"class":469,"line":3149},[151,36171,14459],{"class":503},[151,36173,20415],{"class":14368},[151,36175,6208],{"class":503},[151,36177,20521],{"class":481},[151,36179,36180,36182,36184],{"class":469,"line":3158},[151,36181,20526],{"class":14368},[151,36183,6208],{"class":503},[151,36185,35093],{"class":481},[151,36187,36188,36190],{"class":469,"line":3167},[151,36189,16967],{"class":14368},[151,36191,14372],{"class":503},[151,36193,36194,36196,36198],{"class":469,"line":3175},[151,36195,20542],{"class":14368},[151,36197,6208],{"class":503},[151,36199,35108],{"class":481},[151,36201,36202],{"class":469,"line":3184},[151,36203,1090],{"emptyLinePlaceholder":609},[151,36205,36206,36208,36210,36212],{"class":469,"line":3193},[151,36207,14459],{"class":503},[151,36209,20415],{"class":14368},[151,36211,6208],{"class":503},[151,36213,20562],{"class":481},[151,36215,36216,36218,36220],{"class":469,"line":3720},[151,36217,20526],{"class":14368},[151,36219,6208],{"class":503},[151,36221,35131],{"class":481},[151,36223,36224,36226],{"class":469,"line":3729},[151,36225,16967],{"class":14368},[151,36227,14372],{"class":503},[151,36229,36230,36232,36234],{"class":469,"line":3735},[151,36231,20582],{"class":14368},[151,36233,6208],{"class":503},[151,36235,20587],{"class":481},[151,36237,36238,36240,36242],{"class":469,"line":3745},[151,36239,20592],{"class":14368},[151,36241,6208],{"class":503},[151,36243,20597],{"class":481},[151,36245,36246,36248,36250],{"class":469,"line":3754},[151,36247,20602],{"class":14368},[151,36249,6208],{"class":503},[151,36251,20607],{"class":1869},[151,36253,36254],{"class":469,"line":3760},[151,36255,20612],{"class":481},[151,36257,36258],{"class":469,"line":3773},[151,36259,1090],{"emptyLinePlaceholder":609},[151,36261,36262,36264,36266,36268],{"class":469,"line":3782},[151,36263,14459],{"class":503},[151,36265,20623],{"class":14368},[151,36267,6208],{"class":503},[151,36269,20628],{"class":481},[151,36271,36272,36274,36276,36278],{"class":469,"line":3791},[151,36273,14459],{"class":503},[151,36275,20623],{"class":14368},[151,36277,6208],{"class":503},[151,36279,20650],{"class":481},[151,36281,36282,36284,36286,36288],{"class":469,"line":3803},[151,36283,14459],{"class":503},[151,36285,20623],{"class":14368},[151,36287,6208],{"class":503},[151,36289,20661],{"class":481},[151,36291,36292],{"class":469,"line":3811},[151,36293,1090],{"emptyLinePlaceholder":609},[151,36295,36296,36298,36300,36302],{"class":469,"line":3820},[151,36297,14459],{"class":503},[151,36299,20415],{"class":14368},[151,36301,6208],{"class":503},[151,36303,20676],{"class":481},[151,36305,36306,36308,36310],{"class":469,"line":7084},[151,36307,20526],{"class":14368},[151,36309,6208],{"class":503},[151,36311,35222],{"class":481},[151,36313,36314,36316],{"class":469,"line":7148},[151,36315,16967],{"class":14368},[151,36317,14372],{"class":503},[151,36319,36320,36322,36324],{"class":469,"line":7211},[151,36321,20696],{"class":14368},[151,36323,6208],{"class":503},[151,36325,20701],{"class":481},[151,36327,36328,36330,36332],{"class":469,"line":7273},[151,36329,20706],{"class":14368},[151,36331,6208],{"class":503},[151,36333,35245],{"class":481},[11,36335,36336,36337],{},"Reference: ",[20,36338,36339],{"href":36339,"rel":36340},"https://github.com/marketplace/actions/github-pages-action#%EF%B8%8F-vue-and-nuxt",[24],[736,36342,13563],{"id":13562},[700,36344,36345,36354,36360,36366,36376,36403],{},[79,36346,36347,36350,36351,36353],{},[15,36348,36349],{},"Trigger",": The workflow runs on every push to ",[30,36352,12512],{}," (or PRs for testing)",[79,36355,36356,36359],{},[15,36357,36358],{},"Checkout",": Downloads the repository code",[79,36361,36362,36365],{},[15,36363,36364],{},"Node Setup",": Installs Node.js 14 (as configured in 2021; newer projects might use v18 or v20)",[79,36367,36368,36371,36372,36375],{},[15,36369,36370],{},"Caching",": Speeds up builds by caching ",[30,36373,36374],{},"node_modules"," and npm cache",[79,36377,36378,14372,36381],{},[15,36379,36380],{},"Build Steps",[76,36382,36383,36388,36393],{},[79,36384,36385,36387],{},[30,36386,12007],{},": Installs dependencies",[79,36389,36390,36392],{},[30,36391,12556],{},": Checks code quality (optional but recommended)",[79,36394,36395,36397,36398,413,36400,748],{},[30,36396,12471],{},": Builds the static site (for Nuxt 2; Nuxt 3 uses ",[30,36399,20720],{},[30,36401,36402],{},"nuxt generate",[79,36404,36405,36408,36409,36412],{},[15,36406,36407],{},"Deploy",": Uses ",[30,36410,36411],{},"peaceiris/actions-gh-pages"," to push the built files to GitHub Pages",[736,36414,36416],{"id":36415},"important-notes-for-modern-projects","Important Notes for Modern Projects",[76,36418,36419,36440,36446,36458],{},[79,36420,36421,36424,36425,36428,36429,36431,36432,36435,36436,30583,36438,643],{},[15,36422,36423],{},"Nuxt 3 Migration",": If you're using Nuxt 3, the ",[30,36426,36427],{},"publish_dir"," is typically ",[30,36430,20755],{}," instead of ",[30,36433,36434],{},"docs",", and the build command might be ",[30,36437,36402],{},[30,36439,20717],{},[79,36441,36442,36445],{},[15,36443,36444],{},"Node.js Versions",": Node 14 reached end-of-life in April 2023. Modern projects should use LTS versions (e.g., v18, v20).",[79,36447,36448,36451,36452,106,36455,36457],{},[15,36449,36450],{},"Action Updates",": Consider updating to newer action versions (",[30,36453,36454],{},"actions/checkout@v4",[30,36456,20751],{},") for improved security and features.",[79,36459,36460,36463,36464,36431,36466,36468],{},[15,36461,36462],{},"Branch Names",": Newer repositories often default to ",[30,36465,26180],{},[30,36467,12512],{},"; adjust the trigger accordingly.",[589,36470,36471],{},"html pre.shiki code .s5clZ, html code.shiki .s5clZ{--shiki-default:#22863A;--shiki-dark:#85E89D;--shiki-sepia:#F92672}html pre.shiki code .sMOD_, html code.shiki .sMOD_{--shiki-default:#24292E;--shiki-dark:#E1E4E8;--shiki-sepia:#F8F8F2}html pre.shiki code .sstjo, html code.shiki .sstjo{--shiki-default:#032F62;--shiki-dark:#9ECBFF;--shiki-sepia:#E6DB74}html pre.shiki code .s7F3e, html code.shiki .s7F3e{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#AE81FF}html pre.shiki code .sC2Qs, html code.shiki .sC2Qs{--shiki-default:#D73A49;--shiki-dark:#F97583;--shiki-sepia:#F92672}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html .sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html.sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}",{"title":464,"searchDepth":488,"depth":488,"links":36473},[36474],{"id":36050,"depth":488,"text":36051,"children":36475},[36476,36477],{"id":13562,"depth":500,"text":13563},{"id":36415,"depth":500,"text":36416},"2021-07-05","This article shows how to use GitHub Actions to update my Nuxt.js blog hosted on GitHub Pages.","/static/gha_nuxt/gha_nuxt.png",{},"/2021/07/05/github-actions-for-my-nuxt-blog-hosted-on-github-pages",{"title":36045,"description":36479},"2021/07/05/github-actions-for-my-nuxt-blog-hosted-on-github-pages",[21297,11803,30125,34958],"Ld4_VZE1eeCvqCrIGHZtn525GLNhbFjRdrfNw4Etyx0",{"id":36488,"title":36489,"body":36490,"comments":602,"date":40757,"description":40758,"draft":602,"extension":605,"external":606,"image":40759,"meta":40760,"navigation":609,"path":40761,"seo":40762,"stem":40763,"tags":40764,"__hash__":40768},"blog/2021/03/18/on-demand-dedicated-serverless-valheim-server-with-cdk-discrod-interactions.md","On-demand, serverless Valheim server setup with AWS CDK, Discord Interactions and GitLab CI",{"type":8,"value":36491,"toc":40735},[36492,36501,36504,36507,36510,36517,36528,36535,36541,36552,36556,36568,36586,36619,36622,36627,36630,36634,36641,36658,36664,36667,36681,36687,36691,36702,36707,36714,36723,36729,36743,36747,36753,37161,37167,37173,37176,37182,37185,37191,37201,37204,37210,37224,37228,37239,37556,37566,37570,37577,37583,37589,37595,37599,37606,37675,37687,38094,38100,38106,38113,38119,38123,38130,38255,38269,38272,38278,38285,38323,38333,38337,38343,38543,38546,38552,38555,38558,38562,38582,38588,38593,38599,38618,38623,39568,39583,39587,39599,40117,40123,40126,40135,40139,40142,40255,40278,40281,40285,40288,40294,40306,40312,40321,40579,40582,40590,40599,40603,40606,40610,40701,40703,40706,40729,40732],[210,36493,36494],{},[11,36495,36496,36497],{},"Here's a link to the GitLab repo I'll be referencing in this article: ",[20,36498,36499],{"href":36499,"rel":36500},"https://gitlab.com/briancaffey/valheim-cdk-discord-interactions",[24],[11,36502,36503],{},"This is an in-depth technical article about running an on-demand, dedicated server for Valheim using Amazon Web Services controlled with Discord Slash Commands, a new part their Interactions API that is currently in beta. Valheim is an open-world online multiplayer survival game loosely based on Norse mythology that has blown up recently.",[11,36505,36506],{},"My main goal with this project was to find an inexpensive way of running a server given how my friends and I play the game, which is typically a few times a week in the evenings. Some combination of 4 of us will start playing, and we drop in and out between walking dogs, cooking, etc. When playing, we all jump on a dedicated voice channel on our group's Discord.",[11,36508,36509],{},"Before setting up a dedicated server, our game's world state was stored on files that lived on one of our computers, and that computer needed to be running the game server in order for anyone to connect. Sending files around would be possible, but would quickly become tedious. There are lots of services that offer dedicated servers for Valheim, as well as many technical guides and channels on the official Valheim Discord server to support the use of dedicated servers. I wanted to see if I could set up a server myself on AWS using CDK, or Cloud Development Kit. CDK is an Infrastructure as Code (IaC) tool that allows you to define, deploy and update AWS infrastructure with popular programming languages such as Python, Typescript, Java, etc.",[56,36511,36513,36516],{"id":36512},"cdk-valheim-construct-on-github",[30,36514,36515],{},"cdk-valheim"," construct on GitHub",[11,36518,36519,36520,36524,36525,208],{},"The best part of CDK is that it enables the creation high-level, reusable constructs that can be published to software registries like npm and PyPI. Developers can import and use these constructs in their own CDK code. A quick google search for \"cdk valheim\" turned up a few results. ",[20,36521,36515],{"href":36522,"rel":36523},"https://github.com/gotodeploy/cdk-valheim",[24]," seems like the best option for what I was looking for. This project uses ECS, a container orchestration tool from AWS that I have experience using with web applications and EFS for persistent file storage. Although it is written in Typescript, I can still use the construct in my preferred programming language (Python) without any extra effort or configuration. This is thanks to the jsii. From ",[20,36526,26139],{"href":26139,"rel":36527},[24],[210,36529,36530],{},[11,36531,36532,36534],{},[30,36533,26141],{}," allows code in any language to naturally interact with JavaScript classes. It is the technology that enables the AWS Cloud Development Kit to deliver polyglot libraries from a single codebase!",[11,36536,36537,36538,36540],{},"Here's an overview of the ",[30,36539,36515],{}," construct:",[76,36542,36543,36546,36549],{},[79,36544,36545],{},"Scheduled scaling of an ECS service using AWS Fargate (a serverless compute engine for containers)",[79,36547,36548],{},"Elastic File System (EFS) file system mounted into the Fargate Task container of our ECS service",[79,36550,36551],{},"Optional automated backups of the EFS file system using AWS Backup",[56,36553,36555],{"id":36554},"discord-interactions-and-slash-commands","Discord Interactions and Slash Commands",[11,36557,36558,36559,36561,36562,30583,36565,643],{},"Scheduled ECS scaling is nice, but we don't always know when we will be able to play together, so it is not the best way to minimize infrastructure costs. My plan was to set the ECS service to an initial task count of zero and then let any of us set the number of tasks to either zero or one through a Discord Slash Command. A Slash Command allows you to interact with a discord bot by typing ",[30,36560,19883],{}," and then tabbing through to the option we want, such as ",[30,36563,36564],{},"/valheim server start",[30,36566,36567],{},"/valheim server status",[11,36569,36570,36571,36574,36575,30583,36578,36581,36582,36585],{},"Invoking a Slash Command from Discord sends a ",[30,36572,36573],{},"POST"," request from Discord to a webhook URL that we have to provide. To handle the webhook, one simple and inexpensive approach is to use API Gateway and a Lambda function that serves a Flask app. The Flask app can then use boto3 (which is included in the Lambda execution environment) to call ",[30,36576,36577],{},"update_service",[30,36579,36580],{},"describe_services"," to scaled the ECS task's ",[30,36583,36584],{},"desiredCount"," based on the slash command options and sub options.",[11,36587,36588,36589,36591,36592,36595,36596,106,36599,187,36602,36605,36606,30583,36609,36611,36612,36614,36615,30583,36617,643],{},"The function that handles the webhook ",[30,36590,36573],{}," request when the sub-command is ",[30,36593,36594],{},"status"," queries AWS for the number of ECS tasks in our service that are ",[30,36597,36598],{},"desired",[30,36600,36601],{},"running",[30,36603,36604],{},"pending"," and then sends back a message that will be displayed to the user who sent the command. When the sub-command is ",[30,36607,36608],{},"start",[30,36610,8568],{},", the ",[30,36613,36584],{}," is either set to ",[30,36616,6760],{},[30,36618,9181],{},[11,36620,36621],{},"Here's an overview of the server setup and how Discord Slash Commands can be used to control the server:",[11,36623,36624],{},[2718,36625],{"alt":20386,"src":36626},"/static/valheim/diagram.png",[11,36628,36629],{},"I'll cover each of the steps labelled in this diagram at the end of the article. The rest of the article will provide detailed instructions for how to set everything up. I won't be going over AWS account setup or Discord server setup.",[56,36631,36633],{"id":36632},"how-to-set-up-the-discord-developer-application-and-interaction","How to set up the Discord developer application and Interaction",[11,36635,36636,36637,36640],{},"First, you need to be the admin of a Discord server. Once you create the server, go to ",[30,36638,36639],{},"Server Settings > Widget"," and take note of the Server ID. This is also known as the Guild ID.",[11,36642,36643,36644,36648,36649,36652,36653,21420,36656,208],{},"Then go to ",[20,36645,36646],{"href":36646,"rel":36647},"https://discord.com/developers/applications",[24]," and create an application. Under ",[30,36650,36651],{},"General Information",", make note of the public key (the application ID). Put these values in a file that will be ",[30,36654,36655],{},".gitignored",[30,36657,11004],{},[459,36659,36662],{"className":36660,"code":36661,"language":997},[995],"export GUILD_ID=123456789\nexport APPLICATION_ID=abc123\n",[30,36663,36661],{"__ignoreMap":464},[11,36665,36666],{},"We will use this file later when registering the Interaction.",[11,36668,36669,36670,36673,36674,187,36677,36680],{},"Next go to the ",[30,36671,36672],{},"OAuth2"," tab and select the ",[30,36675,36676],{},"bot",[30,36678,36679],{},"applications.commands"," permissions. This will generated an OAuth2 authorization link. Copy the link and open it in a browser. We will see an error:",[459,36682,36685],{"className":36683,"code":36684,"language":997},[995],"OAuth2 application does not have a bot\n",[30,36686,36684],{"__ignoreMap":464},[736,36688,36690],{"id":36689},"configure-a-bot-for-our-discord-application","Configure a bot for our Discord application",[11,36692,36693,36694,36697,36698,36701],{},"Next, got to the ",[30,36695,36696],{},"Bot"," tab and click the ",[30,36699,36700],{},"Add Bot"," button.",[210,36703,36704],{},[11,36705,36706],{},"Adding a bot user gives your app visible life in Discord. However, this action is irrevocable! Choose wisely.",[11,36708,36709,36710,36713],{},"Turn off the ",[30,36711,36712],{},"Public Bot"," option and save changes.",[11,36715,36716,36717,36720,36721,208],{},"Get the bot token by clicking on ",[30,36718,36719],{},"Click to Reveal Token",", and add this to ",[30,36722,11004],{},[459,36724,36727],{"className":36725,"code":36726,"language":997},[995],"export BOT_TOKEN=abc.xyz.123\n",[30,36728,36726],{"__ignoreMap":464},[11,36730,36731,36732,36673,36734,187,36736,36738,36739,36742],{},"Now go back to the ",[30,36733,36672],{},[30,36735,36676],{},[30,36737,36679],{}," permissions again, copy the link and open it. Select the server that you want to add this application to. You should see a captcha, and then a message that says ",[30,36740,36741],{},"Authorized",". You should also see a message from your discord server that the bot has joined the server.",[736,36744,36746],{"id":36745},"create-the-interaction","Create the Interaction",[11,36748,36749,36750,36752],{},"Now we will set up the Interaction. Currently the only way to set up the interaction is through an HTTP ",[30,36751,36573],{}," request. This Python script sets up our Interaction:",[459,36754,36756],{"className":24401,"code":36755,"language":24403,"meta":464,"style":464},"\"\"\"\nhttps://discord.com/developers/docs/interactions/slash-commands#registering-a-command\n\"\"\"\n\nimport os\n\nimport requests\n\nAPPLICATION_ID = os.environ.get(\"APPLICATION_ID\")\nGUILD_ID = os.environ.get(\"GUILD_ID\")\nBOT_TOKEN = os.environ.get(\"BOT_TOKEN\")\n\nurl = f\"https://discord.com/api/v8/applications/{APPLICATION_ID}/guilds/{GUILD_ID}/commands\"\n\njson = {\n    \"name\": \"vh\",\n    \"description\": \"Start, stop or get the status of the Valheim server\",\n    \"options\": [\n        {\n            \"name\": \"valheim_server_controls\",\n            \"description\": \"What do you want to do?\",\n            \"type\": 3,\n            \"required\": True,\n            \"choices\": [\n                {\n                    \"name\": \"status\",\n                    \"value\": \"status\"\n                },\n                {\n                    \"name\": \"start\",\n                    \"value\": \"start\"\n                },\n                {\n                    \"name\": \"stop\",\n                    \"value\": \"stop\"\n                }\n            ]\n        },\n    ]\n}\n\nheaders = {\n    \"Authorization\": f\"Bot {BOT_TOKEN}\"\n}\n\nif __name__ == \"__main__\":\n    r = requests.post(url, headers=headers, json=json)\n    print(r.content)\n",[30,36757,36758,36763,36768,36772,36776,36782,36786,36793,36797,36812,36826,36840,36844,36869,36873,36882,36894,36906,36913,36918,36930,36942,36953,36965,36972,36977,36989,36999,37004,37008,37019,37028,37032,37036,37047,37056,37061,37065,37070,37075,37079,37083,37092,37109,37113,37117,37129,37154],{"__ignoreMap":464},[151,36759,36760],{"class":469,"line":470},[151,36761,36762],{"class":481},"\"\"\"\n",[151,36764,36765],{"class":469,"line":488},[151,36766,36767],{"class":481},"https://discord.com/developers/docs/interactions/slash-commands#registering-a-command\n",[151,36769,36770],{"class":469,"line":500},[151,36771,36762],{"class":481},[151,36773,36774],{"class":469,"line":509},[151,36775,1090],{"emptyLinePlaceholder":609},[151,36777,36778,36780],{"class":469,"line":517},[151,36779,16859],{"class":1869},[151,36781,24070],{"class":503},[151,36783,36784],{"class":469,"line":534},[151,36785,1090],{"emptyLinePlaceholder":609},[151,36787,36788,36790],{"class":469,"line":1413},[151,36789,16859],{"class":1869},[151,36791,36792],{"class":503}," requests\n",[151,36794,36795],{"class":469,"line":1418},[151,36796,1090],{"emptyLinePlaceholder":609},[151,36798,36799,36802,36804,36807,36810],{"class":469,"line":2462},[151,36800,36801],{"class":477},"APPLICATION_ID",[151,36803,19865],{"class":1869},[151,36805,36806],{"class":503}," os.environ.get(",[151,36808,36809],{"class":481},"\"APPLICATION_ID\"",[151,36811,3640],{"class":503},[151,36813,36814,36817,36819,36821,36824],{"class":469,"line":2471},[151,36815,36816],{"class":477},"GUILD_ID",[151,36818,19865],{"class":1869},[151,36820,36806],{"class":503},[151,36822,36823],{"class":481},"\"GUILD_ID\"",[151,36825,3640],{"class":503},[151,36827,36828,36831,36833,36835,36838],{"class":469,"line":2480},[151,36829,36830],{"class":477},"BOT_TOKEN",[151,36832,19865],{"class":1869},[151,36834,36806],{"class":503},[151,36836,36837],{"class":481},"\"BOT_TOKEN\"",[151,36839,3640],{"class":503},[151,36841,36842],{"class":469,"line":2489},[151,36843,1090],{"emptyLinePlaceholder":609},[151,36845,36846,36849,36851,36854,36857,36860,36863,36866],{"class":469,"line":2497},[151,36847,36848],{"class":503},"url ",[151,36850,1876],{"class":1869},[151,36852,36853],{"class":12347}," f",[151,36855,36856],{"class":481},"\"https://discord.com/api/v8/applications/",[151,36858,36859],{"class":477},"{APPLICATION_ID}",[151,36861,36862],{"class":481},"/guilds/",[151,36864,36865],{"class":477},"{GUILD_ID}",[151,36867,36868],{"class":481},"/commands\"\n",[151,36870,36871],{"class":469,"line":3140},[151,36872,1090],{"emptyLinePlaceholder":609},[151,36874,36875,36878,36880],{"class":469,"line":3149},[151,36876,36877],{"class":503},"json ",[151,36879,1876],{"class":1869},[151,36881,19833],{"class":503},[151,36883,36884,36887,36889,36892],{"class":469,"line":3158},[151,36885,36886],{"class":481},"    \"name\"",[151,36888,6208],{"class":503},[151,36890,36891],{"class":481},"\"vh\"",[151,36893,9417],{"class":503},[151,36895,36896,36899,36901,36904],{"class":469,"line":3167},[151,36897,36898],{"class":481},"    \"description\"",[151,36900,6208],{"class":503},[151,36902,36903],{"class":481},"\"Start, stop or get the status of the Valheim server\"",[151,36905,9417],{"class":503},[151,36907,36908,36911],{"class":469,"line":3175},[151,36909,36910],{"class":481},"    \"options\"",[151,36912,9399],{"class":503},[151,36914,36915],{"class":469,"line":3184},[151,36916,36917],{"class":503},"        {\n",[151,36919,36920,36923,36925,36928],{"class":469,"line":3193},[151,36921,36922],{"class":481},"            \"name\"",[151,36924,6208],{"class":503},[151,36926,36927],{"class":481},"\"valheim_server_controls\"",[151,36929,9417],{"class":503},[151,36931,36932,36935,36937,36940],{"class":469,"line":3720},[151,36933,36934],{"class":481},"            \"description\"",[151,36936,6208],{"class":503},[151,36938,36939],{"class":481},"\"What do you want to do?\"",[151,36941,9417],{"class":503},[151,36943,36944,36947,36949,36951],{"class":469,"line":3729},[151,36945,36946],{"class":481},"            \"type\"",[151,36948,6208],{"class":503},[151,36950,6557],{"class":477},[151,36952,9417],{"class":503},[151,36954,36955,36958,36960,36963],{"class":469,"line":3735},[151,36956,36957],{"class":481},"            \"required\"",[151,36959,6208],{"class":503},[151,36961,36962],{"class":477},"True",[151,36964,9417],{"class":503},[151,36966,36967,36970],{"class":469,"line":3745},[151,36968,36969],{"class":481},"            \"choices\"",[151,36971,9399],{"class":503},[151,36973,36974],{"class":469,"line":3754},[151,36975,36976],{"class":503},"                {\n",[151,36978,36979,36982,36984,36987],{"class":469,"line":3760},[151,36980,36981],{"class":481},"                    \"name\"",[151,36983,6208],{"class":503},[151,36985,36986],{"class":481},"\"status\"",[151,36988,9417],{"class":503},[151,36990,36991,36994,36996],{"class":469,"line":3773},[151,36992,36993],{"class":481},"                    \"value\"",[151,36995,6208],{"class":503},[151,36997,36998],{"class":481},"\"status\"\n",[151,37000,37001],{"class":469,"line":3782},[151,37002,37003],{"class":503},"                },\n",[151,37005,37006],{"class":469,"line":3791},[151,37007,36976],{"class":503},[151,37009,37010,37012,37014,37017],{"class":469,"line":3803},[151,37011,36981],{"class":481},[151,37013,6208],{"class":503},[151,37015,37016],{"class":481},"\"start\"",[151,37018,9417],{"class":503},[151,37020,37021,37023,37025],{"class":469,"line":3811},[151,37022,36993],{"class":481},[151,37024,6208],{"class":503},[151,37026,37027],{"class":481},"\"start\"\n",[151,37029,37030],{"class":469,"line":3820},[151,37031,37003],{"class":503},[151,37033,37034],{"class":469,"line":7084},[151,37035,36976],{"class":503},[151,37037,37038,37040,37042,37045],{"class":469,"line":7148},[151,37039,36981],{"class":481},[151,37041,6208],{"class":503},[151,37043,37044],{"class":481},"\"stop\"",[151,37046,9417],{"class":503},[151,37048,37049,37051,37053],{"class":469,"line":7211},[151,37050,36993],{"class":481},[151,37052,6208],{"class":503},[151,37054,37055],{"class":481},"\"stop\"\n",[151,37057,37058],{"class":469,"line":7273},[151,37059,37060],{"class":503},"                }\n",[151,37062,37063],{"class":469,"line":7335},[151,37064,16819],{"class":503},[151,37066,37067],{"class":469,"line":7398},[151,37068,37069],{"class":503},"        },\n",[151,37071,37072],{"class":469,"line":7462},[151,37073,37074],{"class":503},"    ]\n",[151,37076,37077],{"class":469,"line":7467},[151,37078,6274],{"class":503},[151,37080,37081],{"class":469,"line":7532},[151,37082,1090],{"emptyLinePlaceholder":609},[151,37084,37085,37088,37090],{"class":469,"line":7537},[151,37086,37087],{"class":503},"headers ",[151,37089,1876],{"class":1869},[151,37091,19833],{"class":503},[151,37093,37094,37097,37099,37101,37104,37107],{"class":469,"line":7603},[151,37095,37096],{"class":481},"    \"Authorization\"",[151,37098,6208],{"class":503},[151,37100,13214],{"class":12347},[151,37102,37103],{"class":481},"\"Bot ",[151,37105,37106],{"class":477},"{BOT_TOKEN}",[151,37108,16406],{"class":481},[151,37110,37111],{"class":469,"line":7608},[151,37112,6274],{"class":503},[151,37114,37115],{"class":469,"line":7673},[151,37116,1090],{"emptyLinePlaceholder":609},[151,37118,37119,37121,37123,37125,37127],{"class":469,"line":7678},[151,37120,17218],{"class":1869},[151,37122,17285],{"class":12360},[151,37124,17288],{"class":1869},[151,37126,17291],{"class":481},[151,37128,14372],{"class":503},[151,37130,37131,37134,37136,37139,37142,37144,37147,37149,37151],{"class":469,"line":7708},[151,37132,37133],{"class":503},"    r ",[151,37135,1876],{"class":1869},[151,37137,37138],{"class":503}," requests.post(url, ",[151,37140,37141],{"class":15210},"headers",[151,37143,1876],{"class":1869},[151,37145,37146],{"class":503},"headers, ",[151,37148,6196],{"class":15210},[151,37150,1876],{"class":1869},[151,37152,37153],{"class":503},"json)\n",[151,37155,37156,37158],{"class":469,"line":7713},[151,37157,24285],{"class":2226},[151,37159,37160],{"class":503},"(r.content)\n",[11,37162,37163,37164,37166],{},"Before running this command, source the ",[30,37165,11004],{}," file:",[459,37168,37171],{"className":37169,"code":37170,"language":997},[995],"source .env\n",[30,37172,37170],{"__ignoreMap":464},[11,37174,37175],{},"Then run the script:",[459,37177,37180],{"className":37178,"code":37179,"language":997},[995],"python3 register_bot.py\n",[30,37181,37179],{"__ignoreMap":464},[11,37183,37184],{},"You should see this response:",[459,37186,37189],{"className":37187,"code":37188,"language":997},[995],"b'{\"id\": \"XXXXXXXXXXXXXX\", \"application_id\": \"XXXXXXXXXXXXXX\", \"name\": \"vh\", \"description\": \"Start, stop or get the status of the Valheim server\", \"version\": \"XXXXXXXXXXXXXX\", \"default_permission\": true, \"guild_id\": \"XXXXXXXXXXXXXX\", \"options\": [{\"type\": 3, \"name\": \"valheim_server_controls\", \"description\": \"What do you want to do?\", \"required\": true, \"choices\": [{\"name\": \"status\", \"value\": \"status\"}, {\"name\": \"start\", \"value\": \"start\"}, {\"name\": \"stop\", \"value\": \"stop\"}]}]}'\n",[30,37190,37188],{"__ignoreMap":464},[11,37192,37193,37194,37196,37197,37200],{},"Now, when you type ",[30,37195,19883],{}," in any channel on the Discord server that you authenticated the bot, you should see the ",[30,37198,37199],{},"vh"," command at the top of the list of autocomplete options.",[11,37202,37203],{},"If we run any of these commands, we should see a response saying:",[459,37205,37208],{"className":37206,"code":37207,"language":997},[995],"This interaction failed\n",[30,37209,37207],{"__ignoreMap":464},[11,37211,37212,37213,37216,37217,37219,37220,13576],{},"This is because we have not configured an ",[30,37214,37215],{},"Interactions Endpoint URL"," under the ",[30,37218,36651],{}," section of our Discord Application's admin page (",[20,37221,37222],{"href":37222,"rel":37223},"https://discord.com/developers/applications/",[24],[56,37225,37227],{"id":37226},"setting-up-the-interactions-endpoint-url-for-our-slash-command","Setting up the Interactions Endpoint URL for our Slash Command",[11,37229,37230,37231,37233,37234,208],{},"In order for our Slash Command to do anything, we need to set up URL that Discord will ",[30,37232,36573],{}," the Interaction event data to, including information such as who sent the Interaction, what channel it was sent on, what options were used, etc. You can see an example of the event payload ",[20,37235,37238],{"href":37236,"rel":37237},"https://discord.com/developers/docs/interactions/slash-commands#receiving-an-interaction",[24],"here on the Discord developer documentation",[459,37240,37242],{"className":6194,"code":37241,"language":6196,"meta":464,"style":464},"{\n    \"type\": 2,\n    \"token\": \"A_UNIQUE_TOKEN\",\n    \"member\": {\n        \"user\": {\n            \"id\": 53908232506183680,\n            \"username\": \"Mason\",\n            \"avatar\": \"a_d5efa99b3eeaa7dd43acca82f5692432\",\n            \"discriminator\": \"1337\",\n            \"public_flags\": 131141\n        },\n        \"roles\": [\"539082325061836999\"],\n        \"premium_since\": null,\n        \"permissions\": \"2147483647\",\n        \"pending\": false,\n        \"nick\": null,\n        \"mute\": false,\n        \"joined_at\": \"2017-03-13T19:19:14.040000+00:00\",\n        \"is_pending\": false,\n        \"deaf\": false\n    },\n    \"id\": \"786008729715212338\",\n    \"guild_id\": \"290926798626357999\",\n    \"data\": {\n        \"options\": [{\n            \"name\": \"cardname\",\n            \"value\": \"The Gitrog Monster\"\n        }],\n        \"name\": \"cardsearch\",\n        \"id\": \"771825006014889984\"\n    },\n    \"channel_id\": \"645027906669510667\"\n}\n",[30,37243,37244,37248,37259,37271,37278,37285,37297,37309,37321,37333,37343,37347,37359,37370,37382,37393,37404,37415,37427,37438,37447,37451,37463,37475,37482,37490,37501,37511,37516,37528,37538,37542,37552],{"__ignoreMap":464},[151,37245,37246],{"class":469,"line":470},[151,37247,12966],{"class":503},[151,37249,37250,37253,37255,37257],{"class":469,"line":488},[151,37251,37252],{"class":6205},"    \"type\"",[151,37254,6208],{"class":503},[151,37256,6619],{"class":477},[151,37258,9417],{"class":503},[151,37260,37261,37264,37266,37269],{"class":469,"line":500},[151,37262,37263],{"class":6205},"    \"token\"",[151,37265,6208],{"class":503},[151,37267,37268],{"class":6211},"\"A_UNIQUE_TOKEN\"",[151,37270,9417],{"class":503},[151,37272,37273,37276],{"class":469,"line":509},[151,37274,37275],{"class":6205},"    \"member\"",[151,37277,21223],{"class":503},[151,37279,37280,37283],{"class":469,"line":517},[151,37281,37282],{"class":6205},"        \"user\"",[151,37284,21223],{"class":503},[151,37286,37287,37290,37292,37295],{"class":469,"line":534},[151,37288,37289],{"class":6205},"            \"id\"",[151,37291,6208],{"class":503},[151,37293,37294],{"class":477},"53908232506183680",[151,37296,9417],{"class":503},[151,37298,37299,37302,37304,37307],{"class":469,"line":1413},[151,37300,37301],{"class":6205},"            \"username\"",[151,37303,6208],{"class":503},[151,37305,37306],{"class":6211},"\"Mason\"",[151,37308,9417],{"class":503},[151,37310,37311,37314,37316,37319],{"class":469,"line":1418},[151,37312,37313],{"class":6205},"            \"avatar\"",[151,37315,6208],{"class":503},[151,37317,37318],{"class":6211},"\"a_d5efa99b3eeaa7dd43acca82f5692432\"",[151,37320,9417],{"class":503},[151,37322,37323,37326,37328,37331],{"class":469,"line":2462},[151,37324,37325],{"class":6205},"            \"discriminator\"",[151,37327,6208],{"class":503},[151,37329,37330],{"class":6211},"\"1337\"",[151,37332,9417],{"class":503},[151,37334,37335,37338,37340],{"class":469,"line":2471},[151,37336,37337],{"class":6205},"            \"public_flags\"",[151,37339,6208],{"class":503},[151,37341,37342],{"class":477},"131141\n",[151,37344,37345],{"class":469,"line":2480},[151,37346,37069],{"class":503},[151,37348,37349,37352,37354,37357],{"class":469,"line":2489},[151,37350,37351],{"class":6205},"        \"roles\"",[151,37353,8365],{"class":503},[151,37355,37356],{"class":6211},"\"539082325061836999\"",[151,37358,18746],{"class":503},[151,37360,37361,37364,37366,37368],{"class":469,"line":2497},[151,37362,37363],{"class":6205},"        \"premium_since\"",[151,37365,6208],{"class":503},[151,37367,9824],{"class":477},[151,37369,9417],{"class":503},[151,37371,37372,37375,37377,37380],{"class":469,"line":3140},[151,37373,37374],{"class":6205},"        \"permissions\"",[151,37376,6208],{"class":503},[151,37378,37379],{"class":6211},"\"2147483647\"",[151,37381,9417],{"class":503},[151,37383,37384,37387,37389,37391],{"class":469,"line":3149},[151,37385,37386],{"class":6205},"        \"pending\"",[151,37388,6208],{"class":503},[151,37390,9522],{"class":477},[151,37392,9417],{"class":503},[151,37394,37395,37398,37400,37402],{"class":469,"line":3158},[151,37396,37397],{"class":6205},"        \"nick\"",[151,37399,6208],{"class":503},[151,37401,9824],{"class":477},[151,37403,9417],{"class":503},[151,37405,37406,37409,37411,37413],{"class":469,"line":3167},[151,37407,37408],{"class":6205},"        \"mute\"",[151,37410,6208],{"class":503},[151,37412,9522],{"class":477},[151,37414,9417],{"class":503},[151,37416,37417,37420,37422,37425],{"class":469,"line":3175},[151,37418,37419],{"class":6205},"        \"joined_at\"",[151,37421,6208],{"class":503},[151,37423,37424],{"class":6211},"\"2017-03-13T19:19:14.040000+00:00\"",[151,37426,9417],{"class":503},[151,37428,37429,37432,37434,37436],{"class":469,"line":3184},[151,37430,37431],{"class":6205},"        \"is_pending\"",[151,37433,6208],{"class":503},[151,37435,9522],{"class":477},[151,37437,9417],{"class":503},[151,37439,37440,37443,37445],{"class":469,"line":3193},[151,37441,37442],{"class":6205},"        \"deaf\"",[151,37444,6208],{"class":503},[151,37446,14587],{"class":477},[151,37448,37449],{"class":469,"line":3720},[151,37450,9432],{"class":503},[151,37452,37453,37456,37458,37461],{"class":469,"line":3729},[151,37454,37455],{"class":6205},"    \"id\"",[151,37457,6208],{"class":503},[151,37459,37460],{"class":6211},"\"786008729715212338\"",[151,37462,9417],{"class":503},[151,37464,37465,37468,37470,37473],{"class":469,"line":3735},[151,37466,37467],{"class":6205},"    \"guild_id\"",[151,37469,6208],{"class":503},[151,37471,37472],{"class":6211},"\"290926798626357999\"",[151,37474,9417],{"class":503},[151,37476,37477,37480],{"class":469,"line":3745},[151,37478,37479],{"class":6205},"    \"data\"",[151,37481,21223],{"class":503},[151,37483,37484,37487],{"class":469,"line":3754},[151,37485,37486],{"class":6205},"        \"options\"",[151,37488,37489],{"class":503},": [{\n",[151,37491,37492,37494,37496,37499],{"class":469,"line":3760},[151,37493,36922],{"class":6205},[151,37495,6208],{"class":503},[151,37497,37498],{"class":6211},"\"cardname\"",[151,37500,9417],{"class":503},[151,37502,37503,37506,37508],{"class":469,"line":3773},[151,37504,37505],{"class":6205},"            \"value\"",[151,37507,6208],{"class":503},[151,37509,37510],{"class":6211},"\"The Gitrog Monster\"\n",[151,37512,37513],{"class":469,"line":3782},[151,37514,37515],{"class":503},"        }],\n",[151,37517,37518,37521,37523,37526],{"class":469,"line":3791},[151,37519,37520],{"class":6205},"        \"name\"",[151,37522,6208],{"class":503},[151,37524,37525],{"class":6211},"\"cardsearch\"",[151,37527,9417],{"class":503},[151,37529,37530,37533,37535],{"class":469,"line":3803},[151,37531,37532],{"class":6205},"        \"id\"",[151,37534,6208],{"class":503},[151,37536,37537],{"class":6211},"\"771825006014889984\"\n",[151,37539,37540],{"class":469,"line":3811},[151,37541,9432],{"class":503},[151,37543,37544,37547,37549],{"class":469,"line":3820},[151,37545,37546],{"class":6205},"    \"channel_id\"",[151,37548,6208],{"class":503},[151,37550,37551],{"class":6211},"\"645027906669510667\"\n",[151,37553,37554],{"class":469,"line":7084},[151,37555,6274],{"class":503},[11,37557,37558,37559,37561,37562,37565],{},"This ",[30,37560,36573],{}," request also includes some special headers used for security that we will need to do validation with our handling function. This part can be handled with a decorator provided by the ",[30,37563,37564],{},"discord-interactions"," package on PyPI, but we will need to add some additional configuration to our API Gateway endpoint since these headers will not be passed through the lambda by default.",[56,37567,37569],{"id":37568},"setting-up-aws-infrastructure-with-cdk","Setting up AWS infrastructure with CDK",[11,37571,37572,37573,37576],{},"Let's start a CDK project in a blank repository that will define our infrastructure, Lambda functions and CI/CD pipeline with GitLab CI. Make sure that you have the ",[30,37574,37575],{},"aws-cdk"," CLI installed globally:",[459,37578,37581],{"className":37579,"code":37580,"language":997},[995],"npm i -g aws-cdk\n",[30,37582,37580],{"__ignoreMap":464},[11,37584,37585,37586,37588],{},"Then start a CDK project in a subdirectory called ",[30,37587,30123],{}," with:",[459,37590,37593],{"className":37591,"code":37592,"language":997},[995],"mkdir cdk && cd cdk && cdk init app --language=python\n",[30,37594,37592],{"__ignoreMap":464},[736,37596,37598],{"id":37597},"add-cdk-project-dependencies","Add CDK project dependencies",[11,37600,37601,37602,37605],{},"The next step is to add all of the dependencies to our CDK project that we will use in this project. In ",[30,37603,37604],{},"setup.py"," add the following:",[459,37607,37609],{"className":24401,"code":37608,"language":24403,"meta":464,"style":464},"    install_requires=[\n        \"aws-cdk.core==1.92.0\",\n        \"aws-cdk.aws_applicationautoscaling==1.92.0\",\n        \"aws-cdk.aws_datasync==1.92.0\",\n        \"aws-cdk.aws_lambda==1.92.0\",\n        \"aws-cdk.aws_s3==1.92.0\",\n        \"aws-cdk.aws_apigateway==1.92.0\",\n        \"cdk-valheim==0.0.16\",\n    ],\n",[30,37610,37611,37621,37628,37635,37642,37649,37656,37663,37670],{"__ignoreMap":464},[151,37612,37613,37616,37618],{"class":469,"line":470},[151,37614,37615],{"class":503},"    install_requires",[151,37617,1876],{"class":1869},[151,37619,37620],{"class":503},"[\n",[151,37622,37623,37626],{"class":469,"line":488},[151,37624,37625],{"class":481},"        \"aws-cdk.core==1.92.0\"",[151,37627,9417],{"class":503},[151,37629,37630,37633],{"class":469,"line":500},[151,37631,37632],{"class":481},"        \"aws-cdk.aws_applicationautoscaling==1.92.0\"",[151,37634,9417],{"class":503},[151,37636,37637,37640],{"class":469,"line":509},[151,37638,37639],{"class":481},"        \"aws-cdk.aws_datasync==1.92.0\"",[151,37641,9417],{"class":503},[151,37643,37644,37647],{"class":469,"line":517},[151,37645,37646],{"class":481},"        \"aws-cdk.aws_lambda==1.92.0\"",[151,37648,9417],{"class":503},[151,37650,37651,37654],{"class":469,"line":534},[151,37652,37653],{"class":481},"        \"aws-cdk.aws_s3==1.92.0\"",[151,37655,9417],{"class":503},[151,37657,37658,37661],{"class":469,"line":1413},[151,37659,37660],{"class":481},"        \"aws-cdk.aws_apigateway==1.92.0\"",[151,37662,9417],{"class":503},[151,37664,37665,37668],{"class":469,"line":1418},[151,37666,37667],{"class":481},"        \"cdk-valheim==0.0.16\"",[151,37669,9417],{"class":503},[151,37671,37672],{"class":469,"line":2462},[151,37673,37674],{"class":503},"    ],\n",[11,37676,37677,37678,22326,37681,37684,37685,208],{},"Next we can add the CDK construct for ",[30,37679,37680],{},"ValheimWorld",[30,37682,37683],{},"cdk_stack.py"," file that was generated in our project as well as the imports for the packages we included in ",[30,37686,37604],{},[459,37688,37690],{"className":24401,"code":37689,"language":24403,"meta":464,"style":464},"from aws_cdk import core as cdk\n\nfrom aws_cdk import (\n    core,\n    aws_datasync as datasync,\n    aws_iam as iam,\n    aws_lambda as _lambda,\n    aws_apigateway as apigw,\n    aws_applicationautoscaling as appScaling,\n    aws_s3 as s3,\n)\nfrom cdk_valheim import ValheimWorld, ValheimWorldScalingSchedule\n\n\nclass CdkStack(cdk.Stack):\n\n    def __init__(self, scope: cdk.Construct, construct_id: str, **kwargs) -> None:\n        super().__init__(scope, construct_id, **kwargs)\n        # The code that defines your stack goes here\n        self.valheim_world = ValheimWorld(\n            self,\n            'ValheimWorld',\n            cpu=2048,\n            memory_limit_mib=4096,\n            schedules=[ValheimWorldScalingSchedule(\n                start=appScaling.CronOptions(hour='12', week_day='1-5'),\n                stop=appScaling.CronOptions(hour='1', week_day='1-5'),\n            )],\n            environment={\n                \"SERVER_NAME\": os.environ.get(\"SERVER_NAME\", \"CDK Valheim\"),\n                \"WORLD_NAME\": os.environ.get(\"WORLD_NAME\", \"Amazon\"),\n                \"SERVER_PASS\": os.environ.get(\"SERVER_PASS\", \"fargate\"),\n                \"BACKUPS\": 'false',\n            })\n",[30,37691,37692,37709,37713,37724,37729,37739,37749,37759,37769,37779,37789,37793,37805,37809,37813,37831,37835,37873,37892,37897,37910,37916,37923,37934,37945,37955,37986,38012,38017,38026,38044,38061,38078,38089],{"__ignoreMap":464},[151,37693,37694,37696,37699,37701,37704,37706],{"class":469,"line":470},[151,37695,16853],{"class":1869},[151,37697,37698],{"class":503}," aws_cdk ",[151,37700,16859],{"class":1869},[151,37702,37703],{"class":503}," core ",[151,37705,16998],{"class":1869},[151,37707,37708],{"class":503}," cdk\n",[151,37710,37711],{"class":469,"line":488},[151,37712,1090],{"emptyLinePlaceholder":609},[151,37714,37715,37717,37719,37721],{"class":469,"line":500},[151,37716,16853],{"class":1869},[151,37718,37698],{"class":503},[151,37720,16859],{"class":1869},[151,37722,37723],{"class":503}," (\n",[151,37725,37726],{"class":469,"line":509},[151,37727,37728],{"class":503},"    core,\n",[151,37730,37731,37734,37736],{"class":469,"line":517},[151,37732,37733],{"class":503},"    aws_datasync ",[151,37735,16998],{"class":1869},[151,37737,37738],{"class":503}," datasync,\n",[151,37740,37741,37744,37746],{"class":469,"line":534},[151,37742,37743],{"class":503},"    aws_iam ",[151,37745,16998],{"class":1869},[151,37747,37748],{"class":503}," iam,\n",[151,37750,37751,37754,37756],{"class":469,"line":1413},[151,37752,37753],{"class":503},"    aws_lambda ",[151,37755,16998],{"class":1869},[151,37757,37758],{"class":503}," _lambda,\n",[151,37760,37761,37764,37766],{"class":469,"line":1418},[151,37762,37763],{"class":503},"    aws_apigateway ",[151,37765,16998],{"class":1869},[151,37767,37768],{"class":503}," apigw,\n",[151,37770,37771,37774,37776],{"class":469,"line":2462},[151,37772,37773],{"class":503},"    aws_applicationautoscaling ",[151,37775,16998],{"class":1869},[151,37777,37778],{"class":503}," appScaling,\n",[151,37780,37781,37784,37786],{"class":469,"line":2471},[151,37782,37783],{"class":503},"    aws_s3 ",[151,37785,16998],{"class":1869},[151,37787,37788],{"class":503}," s3,\n",[151,37790,37791],{"class":469,"line":2480},[151,37792,3640],{"class":503},[151,37794,37795,37797,37800,37802],{"class":469,"line":2489},[151,37796,16853],{"class":1869},[151,37798,37799],{"class":503}," cdk_valheim ",[151,37801,16859],{"class":1869},[151,37803,37804],{"class":503}," ValheimWorld, ValheimWorldScalingSchedule\n",[151,37806,37807],{"class":469,"line":2497},[151,37808,1090],{"emptyLinePlaceholder":609},[151,37810,37811],{"class":469,"line":3140},[151,37812,1090],{"emptyLinePlaceholder":609},[151,37814,37815,37817,37820,37822,37824,37826,37829],{"class":469,"line":3149},[151,37816,16519],{"class":12347},[151,37818,37819],{"class":15254}," CdkStack",[151,37821,12386],{"class":503},[151,37823,30123],{"class":15260},[151,37825,643],{"class":503},[151,37827,37828],{"class":15260},"Stack",[151,37830,15264],{"class":503},[151,37832,37833],{"class":469,"line":3158},[151,37834,1090],{"emptyLinePlaceholder":609},[151,37836,37837,37839,37841,37843,37845,37847,37850,37853,37856,37858,37860,37862,37864,37867,37869,37871],{"class":469,"line":3167},[151,37838,16566],{"class":12347},[151,37840,15272],{"class":2226},[151,37842,12386],{"class":503},[151,37844,15277],{"class":15232},[151,37846,106],{"class":503},[151,37848,37849],{"class":15232},"scope",[151,37851,37852],{"class":503},": cdk.Construct, ",[151,37854,37855],{"class":15232},"construct_id",[151,37857,6208],{"class":503},[151,37859,15343],{"class":6205},[151,37861,106],{"class":503},[151,37863,24677],{"class":1869},[151,37865,37866],{"class":15232},"kwargs",[151,37868,17374],{"class":503},[151,37870,15437],{"class":477},[151,37872,14372],{"class":503},[151,37874,37875,37878,37881,37884,37887,37889],{"class":469,"line":3175},[151,37876,37877],{"class":6205},"        super",[151,37879,37880],{"class":503},"().",[151,37882,37883],{"class":2226},"__init__",[151,37885,37886],{"class":503},"(scope, construct_id, ",[151,37888,24677],{"class":1869},[151,37890,37891],{"class":503},"kwargs)\n",[151,37893,37894],{"class":469,"line":3184},[151,37895,37896],{"class":1527},"        # The code that defines your stack goes here\n",[151,37898,37899,37902,37905,37907],{"class":469,"line":3193},[151,37900,37901],{"class":15289},"        self",[151,37903,37904],{"class":503},".valheim_world ",[151,37906,1876],{"class":1869},[151,37908,37909],{"class":503}," ValheimWorld(\n",[151,37911,37912,37914],{"class":469,"line":3720},[151,37913,15290],{"class":15289},[151,37915,9417],{"class":503},[151,37917,37918,37921],{"class":469,"line":3729},[151,37919,37920],{"class":481},"            'ValheimWorld'",[151,37922,9417],{"class":503},[151,37924,37925,37928,37930,37932],{"class":469,"line":3735},[151,37926,37927],{"class":15210},"            cpu",[151,37929,1876],{"class":1869},[151,37931,309],{"class":477},[151,37933,9417],{"class":503},[151,37935,37936,37939,37941,37943],{"class":469,"line":3745},[151,37937,37938],{"class":15210},"            memory_limit_mib",[151,37940,1876],{"class":1869},[151,37942,316],{"class":477},[151,37944,9417],{"class":503},[151,37946,37947,37950,37952],{"class":469,"line":3754},[151,37948,37949],{"class":15210},"            schedules",[151,37951,1876],{"class":1869},[151,37953,37954],{"class":503},"[ValheimWorldScalingSchedule(\n",[151,37956,37957,37960,37962,37965,37968,37970,37973,37975,37978,37980,37983],{"class":469,"line":3760},[151,37958,37959],{"class":15210},"                start",[151,37961,1876],{"class":1869},[151,37963,37964],{"class":503},"appScaling.CronOptions(",[151,37966,37967],{"class":15210},"hour",[151,37969,1876],{"class":1869},[151,37971,37972],{"class":481},"'12'",[151,37974,106],{"class":503},[151,37976,37977],{"class":15210},"week_day",[151,37979,1876],{"class":1869},[151,37981,37982],{"class":481},"'1-5'",[151,37984,37985],{"class":503},"),\n",[151,37987,37988,37991,37993,37995,37997,37999,38002,38004,38006,38008,38010],{"class":469,"line":3773},[151,37989,37990],{"class":15210},"                stop",[151,37992,1876],{"class":1869},[151,37994,37964],{"class":503},[151,37996,37967],{"class":15210},[151,37998,1876],{"class":1869},[151,38000,38001],{"class":481},"'1'",[151,38003,106],{"class":503},[151,38005,37977],{"class":15210},[151,38007,1876],{"class":1869},[151,38009,37982],{"class":481},[151,38011,37985],{"class":503},[151,38013,38014],{"class":469,"line":3782},[151,38015,38016],{"class":503},"            )],\n",[151,38018,38019,38022,38024],{"class":469,"line":3791},[151,38020,38021],{"class":15210},"            environment",[151,38023,1876],{"class":1869},[151,38025,12966],{"class":503},[151,38027,38028,38031,38034,38037,38039,38042],{"class":469,"line":3803},[151,38029,38030],{"class":481},"                \"SERVER_NAME\"",[151,38032,38033],{"class":503},": os.environ.get(",[151,38035,38036],{"class":481},"\"SERVER_NAME\"",[151,38038,106],{"class":503},[151,38040,38041],{"class":481},"\"CDK Valheim\"",[151,38043,37985],{"class":503},[151,38045,38046,38049,38051,38054,38056,38059],{"class":469,"line":3811},[151,38047,38048],{"class":481},"                \"WORLD_NAME\"",[151,38050,38033],{"class":503},[151,38052,38053],{"class":481},"\"WORLD_NAME\"",[151,38055,106],{"class":503},[151,38057,38058],{"class":481},"\"Amazon\"",[151,38060,37985],{"class":503},[151,38062,38063,38066,38068,38071,38073,38076],{"class":469,"line":3820},[151,38064,38065],{"class":481},"                \"SERVER_PASS\"",[151,38067,38033],{"class":503},[151,38069,38070],{"class":481},"\"SERVER_PASS\"",[151,38072,106],{"class":503},[151,38074,38075],{"class":481},"\"fargate\"",[151,38077,37985],{"class":503},[151,38079,38080,38083,38085,38087],{"class":469,"line":7084},[151,38081,38082],{"class":481},"                \"BACKUPS\"",[151,38084,6208],{"class":503},[151,38086,34308],{"class":481},[151,38088,9417],{"class":503},[151,38090,38091],{"class":469,"line":7148},[151,38092,38093],{"class":503},"            })\n",[11,38095,38096,38097,38099],{},"We are almost ready to deploy a basic version of our Valheim server using the ",[30,38098,36515],{}," construct. If we were to run the following command:",[459,38101,38104],{"className":38102,"code":38103,"language":997},[995],"cdk deploy --app cdk/app.py --require-approval never\n",[30,38105,38103],{"__ignoreMap":464},[11,38107,38108,38109,38112],{},"from the root of our project, it should work. This assumes that we have default credentials configured in ",[30,38110,38111],{},"~/.aws/credentials"," and that we have also bootstrapped our AWS account with the resources it needs for CDK to work:",[459,38114,38117],{"className":38115,"code":38116,"language":997},[995],"cdk bootstrap --app cdk/app.py aws://$AWS_ACCOUNT_ID/$AWS_DEFAULT_REGION\n",[30,38118,38116],{"__ignoreMap":464},[736,38120,38122],{"id":38121},"setup-gitlab-ci-job-for-automated-deployments","Setup GitLab CI job for automated deployments",[11,38124,38125,38126,38129],{},"Instead of deploying from the command line, it would be better to run the deployment from a CI/CD pipeline. Add a ",[30,38127,38128],{},".gitlab-ci.yml"," file to the root of your project and populate it with the following YAML:",[459,38131,38133],{"className":14359,"code":38132,"language":14361,"meta":464,"style":464},"stages:\n  - deploy\n\nimage: python:3.8\n\ncdk_deploy:\n  stage: deploy\n  rules:\n    - if: \"$CI_COMMIT_TAG\"\n      when: always\n  before_script:\n    - apt-get -qq update && apt-get -y install nodejs npm\n    - npm i -g aws-cdk\n    - pip3 install -e cdk\n  script:\n    - cdk bootstrap --app cdk/app.py aws://$AWS_ACCOUNT_ID/$AWS_DEFAULT_REGION\n    - cdk deploy --app cdk/app.py --require-approval never\n",[30,38134,38135,38142,38148,38152,38161,38165,38172,38181,38188,38199,38209,38216,38223,38229,38236,38243,38249],{"__ignoreMap":464},[151,38136,38137,38140],{"class":469,"line":470},[151,38138,38139],{"class":14368},"stages",[151,38141,14372],{"class":503},[151,38143,38144,38146],{"class":469,"line":488},[151,38145,19688],{"class":503},[151,38147,20676],{"class":481},[151,38149,38150],{"class":469,"line":500},[151,38151,1090],{"emptyLinePlaceholder":609},[151,38153,38154,38156,38158],{"class":469,"line":509},[151,38155,19666],{"class":14368},[151,38157,6208],{"class":503},[151,38159,38160],{"class":481},"python:3.8\n",[151,38162,38163],{"class":469,"line":517},[151,38164,1090],{"emptyLinePlaceholder":609},[151,38166,38167,38170],{"class":469,"line":534},[151,38168,38169],{"class":14368},"cdk_deploy",[151,38171,14372],{"class":503},[151,38173,38174,38177,38179],{"class":469,"line":1413},[151,38175,38176],{"class":14368},"  stage",[151,38178,6208],{"class":503},[151,38180,20676],{"class":481},[151,38182,38183,38186],{"class":469,"line":1418},[151,38184,38185],{"class":14368},"  rules",[151,38187,14372],{"class":503},[151,38189,38190,38192,38194,38196],{"class":469,"line":2462},[151,38191,29541],{"class":503},[151,38193,17218],{"class":14368},[151,38195,6208],{"class":503},[151,38197,38198],{"class":481},"\"$CI_COMMIT_TAG\"\n",[151,38200,38201,38204,38206],{"class":469,"line":2471},[151,38202,38203],{"class":14368},"      when",[151,38205,6208],{"class":503},[151,38207,38208],{"class":481},"always\n",[151,38210,38211,38214],{"class":469,"line":2480},[151,38212,38213],{"class":14368},"  before_script",[151,38215,14372],{"class":503},[151,38217,38218,38220],{"class":469,"line":2489},[151,38219,29541],{"class":503},[151,38221,38222],{"class":481},"apt-get -qq update && apt-get -y install nodejs npm\n",[151,38224,38225,38227],{"class":469,"line":2497},[151,38226,29541],{"class":503},[151,38228,37580],{"class":481},[151,38230,38231,38233],{"class":469,"line":3140},[151,38232,29541],{"class":503},[151,38234,38235],{"class":481},"pip3 install -e cdk\n",[151,38237,38238,38241],{"class":469,"line":3149},[151,38239,38240],{"class":14368},"  script",[151,38242,14372],{"class":503},[151,38244,38245,38247],{"class":469,"line":3158},[151,38246,29541],{"class":503},[151,38248,38116],{"class":481},[151,38250,38251,38253],{"class":469,"line":3167},[151,38252,29541],{"class":503},[151,38254,38103],{"class":481},[11,38256,38257,38258,38261,38262,38265,38266,643],{},"Before we initialize a git repository in the root directory of our project, remove the ",[30,38259,38260],{},".git"," repo that CDK created when we initialized the project with ",[30,38263,38264],{},"rf -rf cdk/.git",". Now initialized a project in the root directory with ",[30,38267,38268],{},"git init",[11,38270,38271],{},"Next, create a GitLab repository and add the remote to this project with:",[459,38273,38276],{"className":38274,"code":38275,"language":997},[995],"git remote add origin git@gitlab.com:gitlab-username/project-name.git\n",[30,38277,38275],{"__ignoreMap":464},[11,38279,38280,38281,38284],{},"In the GitLab project's ",[30,38282,38283],{},"Settings > CI/CD > Variables"," section, add the following environment variables as protected variables:",[76,38286,38287,38291,38295,38299,38303,38308,38313,38318],{},[79,38288,38289],{},[30,38290,33070],{},[79,38292,38293],{},[30,38294,33091],{},[79,38296,38297],{},[30,38298,33082],{},[79,38300,38301],{},[30,38302,33076],{},[79,38304,38305],{},[30,38306,38307],{},"APPLICATION_PUBLIC_KEY",[79,38309,38310],{},[30,38311,38312],{},"SERVER_PASS",[79,38314,38315],{},[30,38316,38317],{},"SERVER_NAME",[79,38319,38320],{},[30,38321,38322],{},"WORLD_NAME",[11,38324,38325,38326,38329,38330,38332],{},"Now under ",[30,38327,38328],{},"Settings > Repository > Protected Tags",", add a wildcard (",[30,38331,23268],{},") so that all tags are protected and only maintainers can push tags. This allows us to use the protected environment variables only when a trusted maintainer pushes a tag to the repository.",[736,38334,38336],{"id":38335},"edit-the-name-of-the-stack-and-add-region-and-account-info","Edit the name of the stack and add region and account info",[11,38338,38339,38340,208],{},"We are almost ready to create a tag and push to GitLab, but before we do that let's change the name of the CloudFormation stack that CDK will create in ",[30,38341,38342],{},"cdk/app.py",[459,38344,38346],{"className":24401,"code":38345,"language":24403,"meta":464,"style":464},"#!/usr/bin/env python3\n\nimport os\n\nfrom aws_cdk import core as cdk\n\n# For consistency with TypeScript code, `cdk` is the preferred import name for\n# the CDK's core module.  The following line also imports it as `core` for use\n# with examples from the CDK Developer's Guide, which are in the process of\n# being updated to use `cdk`.  You may delete this import if you don't need it.\nfrom aws_cdk import core\n\nfrom cdk.cdk_stack import CdkStack\n\naws_region = os.environ.get(\"AWS_DEFAULT_REGION\", \"us-east-1\")\naws_account = os.environ.get(\"AWS_ACCOUNT_ID\", \"\")\n\n\napp = cdk.App()\nCdkStack(\n    app,\n    \"valheim-server-stack\",\n    env={\"region\": aws_region, \"account\": aws_account}\n)\n\napp.synth()\n",[30,38347,38348,38353,38357,38363,38367,38381,38385,38390,38395,38400,38405,38416,38420,38432,38436,38455,38474,38478,38482,38492,38497,38502,38509,38530,38534,38538],{"__ignoreMap":464},[151,38349,38350],{"class":469,"line":470},[151,38351,38352],{"class":1527},"#!/usr/bin/env python3\n",[151,38354,38355],{"class":469,"line":488},[151,38356,1090],{"emptyLinePlaceholder":609},[151,38358,38359,38361],{"class":469,"line":500},[151,38360,16859],{"class":1869},[151,38362,24070],{"class":503},[151,38364,38365],{"class":469,"line":509},[151,38366,1090],{"emptyLinePlaceholder":609},[151,38368,38369,38371,38373,38375,38377,38379],{"class":469,"line":517},[151,38370,16853],{"class":1869},[151,38372,37698],{"class":503},[151,38374,16859],{"class":1869},[151,38376,37703],{"class":503},[151,38378,16998],{"class":1869},[151,38380,37708],{"class":503},[151,38382,38383],{"class":469,"line":534},[151,38384,1090],{"emptyLinePlaceholder":609},[151,38386,38387],{"class":469,"line":1413},[151,38388,38389],{"class":1527},"# For consistency with TypeScript code, `cdk` is the preferred import name for\n",[151,38391,38392],{"class":469,"line":1418},[151,38393,38394],{"class":1527},"# the CDK's core module.  The following line also imports it as `core` for use\n",[151,38396,38397],{"class":469,"line":2462},[151,38398,38399],{"class":1527},"# with examples from the CDK Developer's Guide, which are in the process of\n",[151,38401,38402],{"class":469,"line":2471},[151,38403,38404],{"class":1527},"# being updated to use `cdk`.  You may delete this import if you don't need it.\n",[151,38406,38407,38409,38411,38413],{"class":469,"line":2480},[151,38408,16853],{"class":1869},[151,38410,37698],{"class":503},[151,38412,16859],{"class":1869},[151,38414,38415],{"class":503}," core\n",[151,38417,38418],{"class":469,"line":2489},[151,38419,1090],{"emptyLinePlaceholder":609},[151,38421,38422,38424,38427,38429],{"class":469,"line":2497},[151,38423,16853],{"class":1869},[151,38425,38426],{"class":503}," cdk.cdk_stack ",[151,38428,16859],{"class":1869},[151,38430,38431],{"class":503}," CdkStack\n",[151,38433,38434],{"class":469,"line":3140},[151,38435,1090],{"emptyLinePlaceholder":609},[151,38437,38438,38441,38443,38445,38448,38450,38453],{"class":469,"line":3149},[151,38439,38440],{"class":503},"aws_region ",[151,38442,1876],{"class":1869},[151,38444,36806],{"class":503},[151,38446,38447],{"class":481},"\"AWS_DEFAULT_REGION\"",[151,38449,106],{"class":503},[151,38451,38452],{"class":481},"\"us-east-1\"",[151,38454,3640],{"class":503},[151,38456,38457,38460,38462,38464,38467,38469,38472],{"class":469,"line":3158},[151,38458,38459],{"class":503},"aws_account ",[151,38461,1876],{"class":1869},[151,38463,36806],{"class":503},[151,38465,38466],{"class":481},"\"AWS_ACCOUNT_ID\"",[151,38468,106],{"class":503},[151,38470,38471],{"class":481},"\"\"",[151,38473,3640],{"class":503},[151,38475,38476],{"class":469,"line":3167},[151,38477,1090],{"emptyLinePlaceholder":609},[151,38479,38480],{"class":469,"line":3175},[151,38481,1090],{"emptyLinePlaceholder":609},[151,38483,38484,38487,38489],{"class":469,"line":3184},[151,38485,38486],{"class":503},"app ",[151,38488,1876],{"class":1869},[151,38490,38491],{"class":503}," cdk.App()\n",[151,38493,38494],{"class":469,"line":3193},[151,38495,38496],{"class":503},"CdkStack(\n",[151,38498,38499],{"class":469,"line":3720},[151,38500,38501],{"class":503},"    app,\n",[151,38503,38504,38507],{"class":469,"line":3729},[151,38505,38506],{"class":481},"    \"valheim-server-stack\"",[151,38508,9417],{"class":503},[151,38510,38511,38514,38516,38518,38521,38524,38527],{"class":469,"line":3735},[151,38512,38513],{"class":15210},"    env",[151,38515,1876],{"class":1869},[151,38517,5729],{"class":503},[151,38519,38520],{"class":481},"\"region\"",[151,38522,38523],{"class":503},": aws_region, ",[151,38525,38526],{"class":481},"\"account\"",[151,38528,38529],{"class":503},": aws_account}\n",[151,38531,38532],{"class":469,"line":3745},[151,38533,3640],{"class":503},[151,38535,38536],{"class":469,"line":3754},[151,38537,1090],{"emptyLinePlaceholder":609},[151,38539,38540],{"class":469,"line":3760},[151,38541,38542],{"class":503},"app.synth()\n",[11,38544,38545],{},"Now commit changes, create a tag and push it to GitLab:",[459,38547,38550],{"className":38548,"code":38549,"language":997},[995],"git add .\ngit commit -m \"initial commit\"\ngit tag v0.0.1\ngit push origin v0.0.1\n",[30,38551,38549],{"__ignoreMap":464},[11,38553,38554],{},"Check the logs of the GitLab CI pipeline that this creates in your GitLab project's CI/CD settings.",[11,38556,38557],{},"If everything runs successfully, you should be able to see your Valheim server listed in the list of community servers once it comes online, and you should be able to connect to it with the password you set in GitLab project variables.",[56,38559,38561],{"id":38560},"add-the-lambda-function-handler-code","Add the Lambda function handler code",[11,38563,38564,38565,38567,38568,10744,38571,38574,38575,10744,38578,38581],{},"We will have a simple Flask application respond the the Discord ",[30,38566,36573],{}," requests that send Interaction events. Let's add ",[30,38569,38570],{},"lambda-handler.py",[30,38572,38573],{},"lambda/functions/interactions/lambda-handler.py",", and ",[30,38576,38577],{},"requirements.txt",[30,38579,38580],{},"lambda/functions/interactions/requirements.txt",". Our project structure should look like this:",[459,38583,38586],{"className":38584,"code":38585,"language":997},[995],"$ tree -L 4\n.\n├── cdk\n│   ├── app.py\n│   ├── cdk\n│   │   ├── cdk_stack.py\n│   │   └── __init__.py\n│   ├── cdk.json\n│   ├── README.md\n│   ├── requirements.txt\n│   ├── setup.py\n│   └── source.bat\n├── lambda\n│   └── functions\n│       └── interactions\n│           ├── lambda-handler.py    \u003C-- here\n│           └── requirements.txt     \u003C-- and here\n├── README.md\n└── register_bot.py\n",[30,38587,38585],{"__ignoreMap":464},[11,38589,19225,38590,38592],{},[30,38591,38577],{}," defines the pip dependencies for our Lambda function. It should include the following:",[459,38594,38597],{"className":38595,"code":38596,"language":997},[995],"aws-wsgi==0.2.7\ndiscord-interactions==0.2.0\nFlask==1.1.2\n",[30,38598,38596],{"__ignoreMap":464},[76,38600,38601,38607,38612],{},[79,38602,38603,38606],{},[30,38604,38605],{},"aws-wsgi"," will transform API Gateway requests into WSGI application requests that Flask can handle",[79,38608,38609,38611],{},[30,38610,37564],{}," will help us with some security-related requirements",[79,38613,38614,38617],{},[30,38615,38616],{},"Flask"," will be our web application framework",[11,38619,38620,38621,208],{},"Here's the code for ",[30,38622,38570],{},[459,38624,38626],{"className":24401,"code":38625,"language":24403,"meta":464,"style":464},"import os\nimport logging\n\nimport awsgi\nimport boto3\nfrom discord_interactions import verify_key_decorator\nfrom flask import (\n    Flask,\n    jsonify,\n    request\n)\n\n\nclient = boto3.client('ecs')\n\n# Your public key can be found on your application in the Developer Portal\nPUBLIC_KEY = os.environ.get('APPLICATION_PUBLIC_KEY')\n\nlogger = logging.getLogger()\nlogger.setLevel(logging.INFO)\n\napp = Flask(__name__)\n\n\n@app.route('/discord', methods=['POST'])\n@verify_key_decorator(PUBLIC_KEY)\ndef index():\n    if request.json[\"type\"] == 1:\n        return jsonify({\"type\": 1})\n    else:\n        logger.info(request.json)\n        try:\n            interaction_option = request.json[\"data\"][\"options\"][0][\"value\"]\n        except KeyError:\n            logger.info(\"Could not parse the interaction option\")\n            interaction_option = \"status\"\n\n        logger.info(\"Interaction:\")\n        logger.info(interaction_option)\n\n        content = \"\"\n\n        if interaction_option == \"status\":\n            try:\n\n                resp = client.describe_services(\n                    cluster=os.environ.get(\"ECS_CLUSTER_ARN\", \"\"),\n                    services=[\n                        os.environ.get(\"ECS_SERVICE_NAME\", \"\"),\n                    ]\n                )\n                desired_count = resp[\"services\"][0][\"desiredCount\"]\n                running_count = resp[\"services\"][0][\"runningCount\"]\n                pending_count = resp[\"services\"][0][\"pendingCount\"]\n\n                content = f\"Desired: {desired_count} | Running: {running_count} | Pending: {pending_count}\"\n\n            except Error as e:\n                content = \"Could not get server status\"\n                logger.info(\"Could not get the server status\")\n                logger.info(e)\n\n        elif interaction_option == \"start\":\n            content = \"Starting the server\"\n\n            resp = client.update_service(\n                cluster=os.environ.get(\"ECS_CLUSTER_ARN\", \"\"),\n                service=os.environ.get(\"ECS_SERVICE_NAME\", \"\"),\n                desiredCount=1\n            )\n\n        elif interaction_option == \"stop\":\n            content = \"Stopping the server\"\n\n            resp = client.update_service(\n                cluster=os.environ.get(\"ECS_CLUSTER_ARN\", \"\"),\n                service=os.environ.get(\"ECS_SERVICE_NAME\", \"\"),\n                desiredCount=0\n            )\n\n        else:\n            content = \"Unknown command\"\n\n        logger.info(resp)\n\n        return jsonify({\n            \"type\": 4,\n            \"data\": {\n                \"tts\": False,\n                \"content\": content,\n                \"embeds\": [],\n                \"allowed_mentions\": { \"parse\": [] }\n            }\n        })\n\ndef handler(event, context):\n    return awsgi.response(\n        app,\n        event,\n        context,\n        base64_content_types={\"image/png\"}\n    )\n",[30,38627,38628,38634,38641,38645,38652,38659,38671,38682,38687,38692,38697,38701,38705,38709,38724,38728,38733,38747,38751,38761,38770,38774,38788,38792,38796,38821,38832,38841,38859,38874,38881,38886,38893,38920,38930,38940,38949,38953,38963,38968,38972,38982,38986,39000,39007,39011,39021,39040,39049,39063,39068,39072,39096,39118,39140,39144,39185,39189,39201,39210,39220,39225,39229,39243,39253,39257,39267,39284,39301,39310,39314,39318,39331,39340,39344,39352,39368,39384,39392,39396,39400,39406,39415,39419,39424,39428,39435,39445,39452,39464,39472,39480,39494,39499,39504,39508,39527,39534,39539,39544,39549,39563],{"__ignoreMap":464},[151,38629,38630,38632],{"class":469,"line":470},[151,38631,16859],{"class":1869},[151,38633,24070],{"class":503},[151,38635,38636,38638],{"class":469,"line":488},[151,38637,16859],{"class":1869},[151,38639,38640],{"class":503}," logging\n",[151,38642,38643],{"class":469,"line":500},[151,38644,1090],{"emptyLinePlaceholder":609},[151,38646,38647,38649],{"class":469,"line":509},[151,38648,16859],{"class":1869},[151,38650,38651],{"class":503}," awsgi\n",[151,38653,38654,38656],{"class":469,"line":517},[151,38655,16859],{"class":1869},[151,38657,38658],{"class":503}," boto3\n",[151,38660,38661,38663,38666,38668],{"class":469,"line":534},[151,38662,16853],{"class":1869},[151,38664,38665],{"class":503}," discord_interactions ",[151,38667,16859],{"class":1869},[151,38669,38670],{"class":503}," verify_key_decorator\n",[151,38672,38673,38675,38678,38680],{"class":469,"line":1413},[151,38674,16853],{"class":1869},[151,38676,38677],{"class":503}," flask ",[151,38679,16859],{"class":1869},[151,38681,37723],{"class":503},[151,38683,38684],{"class":469,"line":1418},[151,38685,38686],{"class":503},"    Flask,\n",[151,38688,38689],{"class":469,"line":2462},[151,38690,38691],{"class":503},"    jsonify,\n",[151,38693,38694],{"class":469,"line":2471},[151,38695,38696],{"class":503},"    request\n",[151,38698,38699],{"class":469,"line":2480},[151,38700,3640],{"class":503},[151,38702,38703],{"class":469,"line":2489},[151,38704,1090],{"emptyLinePlaceholder":609},[151,38706,38707],{"class":469,"line":2497},[151,38708,1090],{"emptyLinePlaceholder":609},[151,38710,38711,38714,38716,38719,38722],{"class":469,"line":3140},[151,38712,38713],{"class":503},"client ",[151,38715,1876],{"class":1869},[151,38717,38718],{"class":503}," boto3.client(",[151,38720,38721],{"class":481},"'ecs'",[151,38723,3640],{"class":503},[151,38725,38726],{"class":469,"line":3149},[151,38727,1090],{"emptyLinePlaceholder":609},[151,38729,38730],{"class":469,"line":3158},[151,38731,38732],{"class":1527},"# Your public key can be found on your application in the Developer Portal\n",[151,38734,38735,38738,38740,38742,38745],{"class":469,"line":3167},[151,38736,38737],{"class":477},"PUBLIC_KEY",[151,38739,19865],{"class":1869},[151,38741,36806],{"class":503},[151,38743,38744],{"class":481},"'APPLICATION_PUBLIC_KEY'",[151,38746,3640],{"class":503},[151,38748,38749],{"class":469,"line":3175},[151,38750,1090],{"emptyLinePlaceholder":609},[151,38752,38753,38756,38758],{"class":469,"line":3184},[151,38754,38755],{"class":503},"logger ",[151,38757,1876],{"class":1869},[151,38759,38760],{"class":503}," logging.getLogger()\n",[151,38762,38763,38766,38768],{"class":469,"line":3193},[151,38764,38765],{"class":503},"logger.setLevel(logging.",[151,38767,6608],{"class":477},[151,38769,3640],{"class":503},[151,38771,38772],{"class":469,"line":3720},[151,38773,1090],{"emptyLinePlaceholder":609},[151,38775,38776,38778,38780,38783,38786],{"class":469,"line":3729},[151,38777,38486],{"class":503},[151,38779,1876],{"class":1869},[151,38781,38782],{"class":503}," Flask(",[151,38784,38785],{"class":12360},"__name__",[151,38787,3640],{"class":503},[151,38789,38790],{"class":469,"line":3735},[151,38791,1090],{"emptyLinePlaceholder":609},[151,38793,38794],{"class":469,"line":3745},[151,38795,1090],{"emptyLinePlaceholder":609},[151,38797,38798,38801,38803,38806,38808,38811,38813,38815,38818],{"class":469,"line":3754},[151,38799,38800],{"class":473},"@app.route",[151,38802,12386],{"class":503},[151,38804,38805],{"class":481},"'/discord'",[151,38807,106],{"class":503},[151,38809,38810],{"class":15210},"methods",[151,38812,1876],{"class":1869},[151,38814,6698],{"class":503},[151,38816,38817],{"class":481},"'POST'",[151,38819,38820],{"class":503},"])\n",[151,38822,38823,38826,38828,38830],{"class":469,"line":3760},[151,38824,38825],{"class":473},"@verify_key_decorator",[151,38827,12386],{"class":503},[151,38829,38737],{"class":477},[151,38831,3640],{"class":503},[151,38833,38834,38836,38839],{"class":469,"line":3773},[151,38835,16925],{"class":12347},[151,38837,38838],{"class":473}," index",[151,38840,16931],{"class":503},[151,38842,38843,38845,38848,38851,38853,38855,38857],{"class":469,"line":3782},[151,38844,23327],{"class":1869},[151,38846,38847],{"class":503}," request.json[",[151,38849,38850],{"class":481},"\"type\"",[151,38852,16654],{"class":503},[151,38854,17223],{"class":1869},[151,38856,12448],{"class":477},[151,38858,14372],{"class":503},[151,38860,38861,38863,38866,38868,38870,38872],{"class":469,"line":3791},[151,38862,16833],{"class":1869},[151,38864,38865],{"class":503}," jsonify({",[151,38867,38850],{"class":481},[151,38869,6208],{"class":503},[151,38871,6760],{"class":477},[151,38873,19610],{"class":503},[151,38875,38876,38879],{"class":469,"line":3803},[151,38877,38878],{"class":1869},"    else",[151,38880,14372],{"class":503},[151,38882,38883],{"class":469,"line":3811},[151,38884,38885],{"class":503},"        logger.info(request.json)\n",[151,38887,38888,38891],{"class":469,"line":3820},[151,38889,38890],{"class":1869},"        try",[151,38892,14372],{"class":503},[151,38894,38895,38898,38900,38902,38904,38906,38909,38911,38913,38915,38918],{"class":469,"line":7084},[151,38896,38897],{"class":503},"            interaction_option ",[151,38899,1876],{"class":1869},[151,38901,38847],{"class":503},[151,38903,18472],{"class":481},[151,38905,6704],{"class":503},[151,38907,38908],{"class":481},"\"options\"",[151,38910,6704],{"class":503},[151,38912,9181],{"class":477},[151,38914,6704],{"class":503},[151,38916,38917],{"class":481},"\"value\"",[151,38919,3691],{"class":503},[151,38921,38922,38925,38928],{"class":469,"line":7148},[151,38923,38924],{"class":1869},"        except",[151,38926,38927],{"class":6205}," KeyError",[151,38929,14372],{"class":503},[151,38931,38932,38935,38938],{"class":469,"line":7211},[151,38933,38934],{"class":503},"            logger.info(",[151,38936,38937],{"class":481},"\"Could not parse the interaction option\"",[151,38939,3640],{"class":503},[151,38941,38942,38944,38946],{"class":469,"line":7273},[151,38943,38897],{"class":503},[151,38945,1876],{"class":1869},[151,38947,38948],{"class":481}," \"status\"\n",[151,38950,38951],{"class":469,"line":7335},[151,38952,1090],{"emptyLinePlaceholder":609},[151,38954,38955,38958,38961],{"class":469,"line":7398},[151,38956,38957],{"class":503},"        logger.info(",[151,38959,38960],{"class":481},"\"Interaction:\"",[151,38962,3640],{"class":503},[151,38964,38965],{"class":469,"line":7462},[151,38966,38967],{"class":503},"        logger.info(interaction_option)\n",[151,38969,38970],{"class":469,"line":7467},[151,38971,1090],{"emptyLinePlaceholder":609},[151,38973,38974,38977,38979],{"class":469,"line":7532},[151,38975,38976],{"class":503},"        content ",[151,38978,1876],{"class":1869},[151,38980,38981],{"class":481}," \"\"\n",[151,38983,38984],{"class":469,"line":7537},[151,38985,1090],{"emptyLinePlaceholder":609},[151,38987,38988,38990,38993,38995,38998],{"class":469,"line":7603},[151,38989,23357],{"class":1869},[151,38991,38992],{"class":503}," interaction_option ",[151,38994,17223],{"class":1869},[151,38996,38997],{"class":481}," \"status\"",[151,38999,14372],{"class":503},[151,39001,39002,39005],{"class":469,"line":7608},[151,39003,39004],{"class":1869},"            try",[151,39006,14372],{"class":503},[151,39008,39009],{"class":469,"line":7673},[151,39010,1090],{"emptyLinePlaceholder":609},[151,39012,39013,39016,39018],{"class":469,"line":7678},[151,39014,39015],{"class":503},"                resp ",[151,39017,1876],{"class":1869},[151,39019,39020],{"class":503}," client.describe_services(\n",[151,39022,39023,39026,39028,39031,39034,39036,39038],{"class":469,"line":7708},[151,39024,39025],{"class":15210},"                    cluster",[151,39027,1876],{"class":1869},[151,39029,39030],{"class":503},"os.environ.get(",[151,39032,39033],{"class":481},"\"ECS_CLUSTER_ARN\"",[151,39035,106],{"class":503},[151,39037,38471],{"class":481},[151,39039,37985],{"class":503},[151,39041,39042,39045,39047],{"class":469,"line":7713},[151,39043,39044],{"class":15210},"                    services",[151,39046,1876],{"class":1869},[151,39048,37620],{"class":503},[151,39050,39051,39054,39057,39059,39061],{"class":469,"line":7746},[151,39052,39053],{"class":503},"                        os.environ.get(",[151,39055,39056],{"class":481},"\"ECS_SERVICE_NAME\"",[151,39058,106],{"class":503},[151,39060,38471],{"class":481},[151,39062,37985],{"class":503},[151,39064,39065],{"class":469,"line":7751},[151,39066,39067],{"class":503},"                    ]\n",[151,39069,39070],{"class":469,"line":7816},[151,39071,16814],{"class":503},[151,39073,39074,39077,39079,39082,39085,39087,39089,39091,39094],{"class":469,"line":7821},[151,39075,39076],{"class":503},"                desired_count ",[151,39078,1876],{"class":1869},[151,39080,39081],{"class":503}," resp[",[151,39083,39084],{"class":481},"\"services\"",[151,39086,6704],{"class":503},[151,39088,9181],{"class":477},[151,39090,6704],{"class":503},[151,39092,39093],{"class":481},"\"desiredCount\"",[151,39095,3691],{"class":503},[151,39097,39098,39101,39103,39105,39107,39109,39111,39113,39116],{"class":469,"line":7847},[151,39099,39100],{"class":503},"                running_count ",[151,39102,1876],{"class":1869},[151,39104,39081],{"class":503},[151,39106,39084],{"class":481},[151,39108,6704],{"class":503},[151,39110,9181],{"class":477},[151,39112,6704],{"class":503},[151,39114,39115],{"class":481},"\"runningCount\"",[151,39117,3691],{"class":503},[151,39119,39120,39123,39125,39127,39129,39131,39133,39135,39138],{"class":469,"line":7852},[151,39121,39122],{"class":503},"                pending_count ",[151,39124,1876],{"class":1869},[151,39126,39081],{"class":503},[151,39128,39084],{"class":481},[151,39130,6704],{"class":503},[151,39132,9181],{"class":477},[151,39134,6704],{"class":503},[151,39136,39137],{"class":481},"\"pendingCount\"",[151,39139,3691],{"class":503},[151,39141,39142],{"class":469,"line":7887},[151,39143,1090],{"emptyLinePlaceholder":609},[151,39145,39146,39149,39151,39153,39156,39158,39161,39163,39166,39168,39171,39173,39176,39178,39181,39183],{"class":469,"line":7892},[151,39147,39148],{"class":503},"                content ",[151,39150,1876],{"class":1869},[151,39152,36853],{"class":12347},[151,39154,39155],{"class":481},"\"Desired: ",[151,39157,5729],{"class":477},[151,39159,39160],{"class":503},"desired_count",[151,39162,2001],{"class":477},[151,39164,39165],{"class":481}," | Running: ",[151,39167,5729],{"class":477},[151,39169,39170],{"class":503},"running_count",[151,39172,2001],{"class":477},[151,39174,39175],{"class":481}," | Pending: ",[151,39177,5729],{"class":477},[151,39179,39180],{"class":503},"pending_count",[151,39182,2001],{"class":477},[151,39184,16406],{"class":481},[151,39186,39187],{"class":469,"line":7924},[151,39188,1090],{"emptyLinePlaceholder":609},[151,39190,39191,39194,39197,39199],{"class":469,"line":7929},[151,39192,39193],{"class":1869},"            except",[151,39195,39196],{"class":503}," Error ",[151,39198,16998],{"class":1869},[151,39200,18350],{"class":503},[151,39202,39203,39205,39207],{"class":469,"line":7991},[151,39204,39148],{"class":503},[151,39206,1876],{"class":1869},[151,39208,39209],{"class":481}," \"Could not get server status\"\n",[151,39211,39212,39215,39218],{"class":469,"line":7996},[151,39213,39214],{"class":503},"                logger.info(",[151,39216,39217],{"class":481},"\"Could not get the server status\"",[151,39219,3640],{"class":503},[151,39221,39222],{"class":469,"line":8078},[151,39223,39224],{"class":503},"                logger.info(e)\n",[151,39226,39227],{"class":469,"line":8140},[151,39228,1090],{"emptyLinePlaceholder":609},[151,39230,39231,39234,39236,39238,39241],{"class":469,"line":8145},[151,39232,39233],{"class":1869},"        elif",[151,39235,38992],{"class":503},[151,39237,17223],{"class":1869},[151,39239,39240],{"class":481}," \"start\"",[151,39242,14372],{"class":503},[151,39244,39245,39248,39250],{"class":469,"line":8259},[151,39246,39247],{"class":503},"            content ",[151,39249,1876],{"class":1869},[151,39251,39252],{"class":481}," \"Starting the server\"\n",[151,39254,39255],{"class":469,"line":8264},[151,39256,1090],{"emptyLinePlaceholder":609},[151,39258,39259,39262,39264],{"class":469,"line":8613},[151,39260,39261],{"class":503},"            resp ",[151,39263,1876],{"class":1869},[151,39265,39266],{"class":503}," client.update_service(\n",[151,39268,39269,39272,39274,39276,39278,39280,39282],{"class":469,"line":8678},[151,39270,39271],{"class":15210},"                cluster",[151,39273,1876],{"class":1869},[151,39275,39030],{"class":503},[151,39277,39033],{"class":481},[151,39279,106],{"class":503},[151,39281,38471],{"class":481},[151,39283,37985],{"class":503},[151,39285,39286,39289,39291,39293,39295,39297,39299],{"class":469,"line":8742},[151,39287,39288],{"class":15210},"                service",[151,39290,1876],{"class":1869},[151,39292,39030],{"class":503},[151,39294,39056],{"class":481},[151,39296,106],{"class":503},[151,39298,38471],{"class":481},[151,39300,37985],{"class":503},[151,39302,39303,39306,39308],{"class":469,"line":8806},[151,39304,39305],{"class":15210},"                desiredCount",[151,39307,1876],{"class":1869},[151,39309,1963],{"class":477},[151,39311,39312],{"class":469,"line":8870},[151,39313,15381],{"class":503},[151,39315,39316],{"class":469,"line":8875},[151,39317,1090],{"emptyLinePlaceholder":609},[151,39319,39320,39322,39324,39326,39329],{"class":469,"line":8881},[151,39321,39233],{"class":1869},[151,39323,38992],{"class":503},[151,39325,17223],{"class":1869},[151,39327,39328],{"class":481}," \"stop\"",[151,39330,14372],{"class":503},[151,39332,39333,39335,39337],{"class":469,"line":8886},[151,39334,39247],{"class":503},[151,39336,1876],{"class":1869},[151,39338,39339],{"class":481}," \"Stopping the server\"\n",[151,39341,39342],{"class":469,"line":8892},[151,39343,1090],{"emptyLinePlaceholder":609},[151,39345,39346,39348,39350],{"class":469,"line":8963},[151,39347,39261],{"class":503},[151,39349,1876],{"class":1869},[151,39351,39266],{"class":503},[151,39353,39354,39356,39358,39360,39362,39364,39366],{"class":469,"line":8969},[151,39355,39271],{"class":15210},[151,39357,1876],{"class":1869},[151,39359,39030],{"class":503},[151,39361,39033],{"class":481},[151,39363,106],{"class":503},[151,39365,38471],{"class":481},[151,39367,37985],{"class":503},[151,39369,39370,39372,39374,39376,39378,39380,39382],{"class":469,"line":15001},[151,39371,39288],{"class":15210},[151,39373,1876],{"class":1869},[151,39375,39030],{"class":503},[151,39377,39056],{"class":481},[151,39379,106],{"class":503},[151,39381,38471],{"class":481},[151,39383,37985],{"class":503},[151,39385,39386,39388,39390],{"class":469,"line":15009},[151,39387,39305],{"class":15210},[151,39389,1876],{"class":1869},[151,39391,6480],{"class":477},[151,39393,39394],{"class":469,"line":15019},[151,39395,15381],{"class":503},[151,39397,39398],{"class":469,"line":15027},[151,39399,1090],{"emptyLinePlaceholder":609},[151,39401,39402,39404],{"class":469,"line":15037},[151,39403,23395],{"class":1869},[151,39405,14372],{"class":503},[151,39407,39408,39410,39412],{"class":469,"line":15045},[151,39409,39247],{"class":503},[151,39411,1876],{"class":1869},[151,39413,39414],{"class":481}," \"Unknown command\"\n",[151,39416,39417],{"class":469,"line":15055},[151,39418,1090],{"emptyLinePlaceholder":609},[151,39420,39421],{"class":469,"line":15060},[151,39422,39423],{"class":503},"        logger.info(resp)\n",[151,39425,39426],{"class":469,"line":15068},[151,39427,1090],{"emptyLinePlaceholder":609},[151,39429,39430,39432],{"class":469,"line":15076},[151,39431,16833],{"class":1869},[151,39433,39434],{"class":503}," jsonify({\n",[151,39436,39437,39439,39441,39443],{"class":469,"line":15085},[151,39438,36946],{"class":481},[151,39440,6208],{"class":503},[151,39442,9187],{"class":477},[151,39444,9417],{"class":503},[151,39446,39447,39450],{"class":469,"line":15095},[151,39448,39449],{"class":481},"            \"data\"",[151,39451,21223],{"class":503},[151,39453,39454,39457,39459,39462],{"class":469,"line":15105},[151,39455,39456],{"class":481},"                \"tts\"",[151,39458,6208],{"class":503},[151,39460,39461],{"class":477},"False",[151,39463,9417],{"class":503},[151,39465,39466,39469],{"class":469,"line":15110},[151,39467,39468],{"class":481},"                \"content\"",[151,39470,39471],{"class":503},": content,\n",[151,39473,39474,39477],{"class":469,"line":15118},[151,39475,39476],{"class":481},"                \"embeds\"",[151,39478,39479],{"class":503},": [],\n",[151,39481,39482,39485,39488,39491],{"class":469,"line":15128},[151,39483,39484],{"class":481},"                \"allowed_mentions\"",[151,39486,39487],{"class":503},": { ",[151,39489,39490],{"class":481},"\"parse\"",[151,39492,39493],{"class":503},": [] }\n",[151,39495,39496],{"class":469,"line":15139},[151,39497,39498],{"class":503},"            }\n",[151,39500,39501],{"class":469,"line":31954},[151,39502,39503],{"class":503},"        })\n",[151,39505,39506],{"class":469,"line":31960},[151,39507,1090],{"emptyLinePlaceholder":609},[151,39509,39510,39512,39515,39517,39520,39522,39525],{"class":469,"line":31965},[151,39511,16925],{"class":12347},[151,39513,39514],{"class":473}," handler",[151,39516,12386],{"class":503},[151,39518,39519],{"class":15232},"event",[151,39521,106],{"class":503},[151,39523,39524],{"class":15232},"context",[151,39526,15264],{"class":503},[151,39528,39529,39531],{"class":469,"line":31971},[151,39530,17496],{"class":1869},[151,39532,39533],{"class":503}," awsgi.response(\n",[151,39535,39536],{"class":469,"line":31983},[151,39537,39538],{"class":503},"        app,\n",[151,39540,39541],{"class":469,"line":31994},[151,39542,39543],{"class":503},"        event,\n",[151,39545,39546],{"class":469,"line":32007},[151,39547,39548],{"class":503},"        context,\n",[151,39550,39551,39554,39556,39558,39561],{"class":469,"line":32018},[151,39552,39553],{"class":15210},"        base64_content_types",[151,39555,1876],{"class":1869},[151,39557,5729],{"class":503},[151,39559,39560],{"class":481},"\"image/png\"",[151,39562,6274],{"class":503},[151,39564,39565],{"class":469,"line":32026},[151,39566,39567],{"class":503},"    )\n",[11,39569,39570,39571,313,39573,18952,39576,39578,39579,39582],{},"Notice how we pass the Flask ",[30,39572,26476],{},[30,39574,39575],{},"awsgi.response",[30,39577,38605],{}," (or ",[30,39580,39581],{},"awsgi"," as it is imported) is the go-between for API Gateway and WSGI.",[56,39584,39586],{"id":39585},"add-the-cdk-code-for-api-gateway-and-lambda-that-will-serve-our-discord-interaction-endpoint-url","Add the CDK code for API Gateway and Lambda that will serve our Discord Interaction Endpoint URL",[11,39588,39589,39590,39592,39593,39595,39596,208],{},"Now we can add the following code to ",[30,39591,37683],{}," to configure the API Gateway and Lambda function. Add the following to ",[30,39594,37683],{}," after our definition of ",[30,39597,39598],{},"self.valheim_world",[459,39600,39602],{"className":24401,"code":39601,"language":24403,"meta":464,"style":464},"        self.env_vars = {\n            \"APPLICATION_PUBLIC_KEY\": os.environ.get(\"APPLICATION_PUBLIC_KEY\"),\n            \"ECS_SERVICE_NAME\": self.valheim_world.service.service_name,\n            \"ECS_CLUSTER_ARN\": self.valheim_world.service.cluster.cluster_arn\n        }\n\n        self.flask_lambda_layer = _lambda.LayerVersion(\n            self,\n            \"FlaskAppLambdaLayer\",\n            code=_lambda.AssetCode(\"./layers/flask\"),\n            compatible_runtimes=[_lambda.Runtime.PYTHON_3_8,],\n        )\n\n        self.flask_app_lambda = _lambda.Function(\n            self,\n            \"FlaskAppLambda\",\n            runtime=_lambda.Runtime.PYTHON_3_8,\n            code=_lambda.AssetCode('./lambda/functions/interactions'),\n            function_name=\"flask-app-handler\",\n            handler=\"lambda-handler.handler\",\n            layers=[self.flask_lambda_layer],\n            timeout=core.Duration.seconds(60),\n            environment={**self.env_vars},\n        )\n\n        self.flask_app_lambda.role.add_managed_policy(\n            iam.ManagedPolicy.from_managed_policy_arn(\n                self,\n                'ECS_FullAccessPolicy',\n                managed_policy_arn='arn:aws:iam::aws:policy/AmazonECS_FullAccess'\n            )\n        )\n\n        # https://slmkitani.medium.com/passing-custom-headers-through-amazon-api-gateway-to-an-aws-lambda-function-f3a1cfdc0e29\n        self.request_templates = {\n            \"application/json\": '''{\n                \"method\": \"$context.httpMethod\",\n                \"body\" : $input.json(\"$\"),\n                \"headers\": {\n                    #foreach($param in $input.params().header.keySet())\n                    \"$param\": \"$util.escapeJavaScript($input.params().header.get($param))\"\n                    #if($foreach.hasNext),#end\n                    #end\n                }\n            }\n            '''\n        }\n\n        self.apigateway = apigw.RestApi(\n            self,\n            'FlaskAppEndpoint',\n        )\n\n        self.apigateway.root.add_method(\"ANY\")\n\n        self.discord_interaction_webhook = self.apigateway.root.add_resource(\"discord\")\n\n        self.discord_interaction_webhook_integration = apigw.LambdaIntegration(\n            self.flask_app_lambda,\n            request_templates=self.request_templates\n        )\n\n        self.discord_interaction_webhook.add_method(\n            'POST',\n            self.discord_interaction_webhook_integration\n        )\n",[30,39603,39604,39615,39627,39639,39651,39655,39659,39671,39677,39684,39699,39715,39719,39723,39735,39741,39748,39762,39775,39787,39799,39813,39828,39843,39847,39851,39858,39863,39870,39877,39887,39891,39895,39899,39904,39915,39925,39930,39935,39940,39945,39950,39955,39960,39964,39968,39973,39977,39981,39993,39999,40006,40010,40014,40026,40030,40049,40053,40065,40072,40084,40088,40092,40099,40106,40113],{"__ignoreMap":464},[151,39605,39606,39608,39611,39613],{"class":469,"line":470},[151,39607,37901],{"class":15289},[151,39609,39610],{"class":503},".env_vars ",[151,39612,1876],{"class":1869},[151,39614,19833],{"class":503},[151,39616,39617,39620,39622,39625],{"class":469,"line":488},[151,39618,39619],{"class":481},"            \"APPLICATION_PUBLIC_KEY\"",[151,39621,38033],{"class":503},[151,39623,39624],{"class":481},"\"APPLICATION_PUBLIC_KEY\"",[151,39626,37985],{"class":503},[151,39628,39629,39632,39634,39636],{"class":469,"line":500},[151,39630,39631],{"class":481},"            \"ECS_SERVICE_NAME\"",[151,39633,6208],{"class":503},[151,39635,15277],{"class":15289},[151,39637,39638],{"class":503},".valheim_world.service.service_name,\n",[151,39640,39641,39644,39646,39648],{"class":469,"line":509},[151,39642,39643],{"class":481},"            \"ECS_CLUSTER_ARN\"",[151,39645,6208],{"class":503},[151,39647,15277],{"class":15289},[151,39649,39650],{"class":503},".valheim_world.service.cluster.cluster_arn\n",[151,39652,39653],{"class":469,"line":517},[151,39654,23390],{"class":503},[151,39656,39657],{"class":469,"line":534},[151,39658,1090],{"emptyLinePlaceholder":609},[151,39660,39661,39663,39666,39668],{"class":469,"line":1413},[151,39662,37901],{"class":15289},[151,39664,39665],{"class":503},".flask_lambda_layer ",[151,39667,1876],{"class":1869},[151,39669,39670],{"class":503}," _lambda.LayerVersion(\n",[151,39672,39673,39675],{"class":469,"line":1418},[151,39674,15290],{"class":15289},[151,39676,9417],{"class":503},[151,39678,39679,39682],{"class":469,"line":2462},[151,39680,39681],{"class":481},"            \"FlaskAppLambdaLayer\"",[151,39683,9417],{"class":503},[151,39685,39686,39689,39691,39694,39697],{"class":469,"line":2471},[151,39687,39688],{"class":15210},"            code",[151,39690,1876],{"class":1869},[151,39692,39693],{"class":503},"_lambda.AssetCode(",[151,39695,39696],{"class":481},"\"./layers/flask\"",[151,39698,37985],{"class":503},[151,39700,39701,39704,39706,39709,39712],{"class":469,"line":2480},[151,39702,39703],{"class":15210},"            compatible_runtimes",[151,39705,1876],{"class":1869},[151,39707,39708],{"class":503},"[_lambda.Runtime.",[151,39710,39711],{"class":477},"PYTHON_3_8",[151,39713,39714],{"class":503},",],\n",[151,39716,39717],{"class":469,"line":2489},[151,39718,16824],{"class":503},[151,39720,39721],{"class":469,"line":2497},[151,39722,1090],{"emptyLinePlaceholder":609},[151,39724,39725,39727,39730,39732],{"class":469,"line":3140},[151,39726,37901],{"class":15289},[151,39728,39729],{"class":503},".flask_app_lambda ",[151,39731,1876],{"class":1869},[151,39733,39734],{"class":503}," _lambda.Function(\n",[151,39736,39737,39739],{"class":469,"line":3149},[151,39738,15290],{"class":15289},[151,39740,9417],{"class":503},[151,39742,39743,39746],{"class":469,"line":3158},[151,39744,39745],{"class":481},"            \"FlaskAppLambda\"",[151,39747,9417],{"class":503},[151,39749,39750,39753,39755,39758,39760],{"class":469,"line":3167},[151,39751,39752],{"class":15210},"            runtime",[151,39754,1876],{"class":1869},[151,39756,39757],{"class":503},"_lambda.Runtime.",[151,39759,39711],{"class":477},[151,39761,9417],{"class":503},[151,39763,39764,39766,39768,39770,39773],{"class":469,"line":3175},[151,39765,39688],{"class":15210},[151,39767,1876],{"class":1869},[151,39769,39693],{"class":503},[151,39771,39772],{"class":481},"'./lambda/functions/interactions'",[151,39774,37985],{"class":503},[151,39776,39777,39780,39782,39785],{"class":469,"line":3184},[151,39778,39779],{"class":15210},"            function_name",[151,39781,1876],{"class":1869},[151,39783,39784],{"class":481},"\"flask-app-handler\"",[151,39786,9417],{"class":503},[151,39788,39789,39792,39794,39797],{"class":469,"line":3193},[151,39790,39791],{"class":15210},"            handler",[151,39793,1876],{"class":1869},[151,39795,39796],{"class":481},"\"lambda-handler.handler\"",[151,39798,9417],{"class":503},[151,39800,39801,39804,39806,39808,39810],{"class":469,"line":3720},[151,39802,39803],{"class":15210},"            layers",[151,39805,1876],{"class":1869},[151,39807,6698],{"class":503},[151,39809,15277],{"class":15289},[151,39811,39812],{"class":503},".flask_lambda_layer],\n",[151,39814,39815,39818,39820,39823,39826],{"class":469,"line":3729},[151,39816,39817],{"class":15210},"            timeout",[151,39819,1876],{"class":1869},[151,39821,39822],{"class":503},"core.Duration.seconds(",[151,39824,39825],{"class":477},"60",[151,39827,37985],{"class":503},[151,39829,39830,39832,39834,39836,39838,39840],{"class":469,"line":3735},[151,39831,38021],{"class":15210},[151,39833,1876],{"class":1869},[151,39835,5729],{"class":503},[151,39837,24677],{"class":1869},[151,39839,15277],{"class":15289},[151,39841,39842],{"class":503},".env_vars},\n",[151,39844,39845],{"class":469,"line":3745},[151,39846,16824],{"class":503},[151,39848,39849],{"class":469,"line":3754},[151,39850,1090],{"emptyLinePlaceholder":609},[151,39852,39853,39855],{"class":469,"line":3760},[151,39854,37901],{"class":15289},[151,39856,39857],{"class":503},".flask_app_lambda.role.add_managed_policy(\n",[151,39859,39860],{"class":469,"line":3773},[151,39861,39862],{"class":503},"            iam.ManagedPolicy.from_managed_policy_arn(\n",[151,39864,39865,39868],{"class":469,"line":3782},[151,39866,39867],{"class":15289},"                self",[151,39869,9417],{"class":503},[151,39871,39872,39875],{"class":469,"line":3791},[151,39873,39874],{"class":481},"                'ECS_FullAccessPolicy'",[151,39876,9417],{"class":503},[151,39878,39879,39882,39884],{"class":469,"line":3803},[151,39880,39881],{"class":15210},"                managed_policy_arn",[151,39883,1876],{"class":1869},[151,39885,39886],{"class":481},"'arn:aws:iam::aws:policy/AmazonECS_FullAccess'\n",[151,39888,39889],{"class":469,"line":3811},[151,39890,15381],{"class":503},[151,39892,39893],{"class":469,"line":3820},[151,39894,16824],{"class":503},[151,39896,39897],{"class":469,"line":7084},[151,39898,1090],{"emptyLinePlaceholder":609},[151,39900,39901],{"class":469,"line":7148},[151,39902,39903],{"class":1527},"        # https://slmkitani.medium.com/passing-custom-headers-through-amazon-api-gateway-to-an-aws-lambda-function-f3a1cfdc0e29\n",[151,39905,39906,39908,39911,39913],{"class":469,"line":7211},[151,39907,37901],{"class":15289},[151,39909,39910],{"class":503},".request_templates ",[151,39912,1876],{"class":1869},[151,39914,19833],{"class":503},[151,39916,39917,39920,39922],{"class":469,"line":7273},[151,39918,39919],{"class":481},"            \"application/json\"",[151,39921,6208],{"class":503},[151,39923,39924],{"class":481},"'''{\n",[151,39926,39927],{"class":469,"line":7335},[151,39928,39929],{"class":481},"                \"method\": \"$context.httpMethod\",\n",[151,39931,39932],{"class":469,"line":7398},[151,39933,39934],{"class":481},"                \"body\" : $input.json(\"$\"),\n",[151,39936,39937],{"class":469,"line":7462},[151,39938,39939],{"class":481},"                \"headers\": {\n",[151,39941,39942],{"class":469,"line":7467},[151,39943,39944],{"class":481},"                    #foreach($param in $input.params().header.keySet())\n",[151,39946,39947],{"class":469,"line":7532},[151,39948,39949],{"class":481},"                    \"$param\": \"$util.escapeJavaScript($input.params().header.get($param))\"\n",[151,39951,39952],{"class":469,"line":7537},[151,39953,39954],{"class":481},"                    #if($foreach.hasNext),#end\n",[151,39956,39957],{"class":469,"line":7603},[151,39958,39959],{"class":481},"                    #end\n",[151,39961,39962],{"class":469,"line":7608},[151,39963,37060],{"class":481},[151,39965,39966],{"class":469,"line":7673},[151,39967,39498],{"class":481},[151,39969,39970],{"class":469,"line":7678},[151,39971,39972],{"class":481},"            '''\n",[151,39974,39975],{"class":469,"line":7708},[151,39976,23390],{"class":503},[151,39978,39979],{"class":469,"line":7713},[151,39980,1090],{"emptyLinePlaceholder":609},[151,39982,39983,39985,39988,39990],{"class":469,"line":7746},[151,39984,37901],{"class":15289},[151,39986,39987],{"class":503},".apigateway ",[151,39989,1876],{"class":1869},[151,39991,39992],{"class":503}," apigw.RestApi(\n",[151,39994,39995,39997],{"class":469,"line":7751},[151,39996,15290],{"class":15289},[151,39998,9417],{"class":503},[151,40000,40001,40004],{"class":469,"line":7816},[151,40002,40003],{"class":481},"            'FlaskAppEndpoint'",[151,40005,9417],{"class":503},[151,40007,40008],{"class":469,"line":7821},[151,40009,16824],{"class":503},[151,40011,40012],{"class":469,"line":7847},[151,40013,1090],{"emptyLinePlaceholder":609},[151,40015,40016,40018,40021,40024],{"class":469,"line":7852},[151,40017,37901],{"class":15289},[151,40019,40020],{"class":503},".apigateway.root.add_method(",[151,40022,40023],{"class":481},"\"ANY\"",[151,40025,3640],{"class":503},[151,40027,40028],{"class":469,"line":7887},[151,40029,1090],{"emptyLinePlaceholder":609},[151,40031,40032,40034,40037,40039,40041,40044,40047],{"class":469,"line":7892},[151,40033,37901],{"class":15289},[151,40035,40036],{"class":503},".discord_interaction_webhook ",[151,40038,1876],{"class":1869},[151,40040,15451],{"class":15289},[151,40042,40043],{"class":503},".apigateway.root.add_resource(",[151,40045,40046],{"class":481},"\"discord\"",[151,40048,3640],{"class":503},[151,40050,40051],{"class":469,"line":7924},[151,40052,1090],{"emptyLinePlaceholder":609},[151,40054,40055,40057,40060,40062],{"class":469,"line":7929},[151,40056,37901],{"class":15289},[151,40058,40059],{"class":503},".discord_interaction_webhook_integration ",[151,40061,1876],{"class":1869},[151,40063,40064],{"class":503}," apigw.LambdaIntegration(\n",[151,40066,40067,40069],{"class":469,"line":7991},[151,40068,15290],{"class":15289},[151,40070,40071],{"class":503},".flask_app_lambda,\n",[151,40073,40074,40077,40079,40081],{"class":469,"line":7996},[151,40075,40076],{"class":15210},"            request_templates",[151,40078,1876],{"class":1869},[151,40080,15277],{"class":15289},[151,40082,40083],{"class":503},".request_templates\n",[151,40085,40086],{"class":469,"line":8078},[151,40087,16824],{"class":503},[151,40089,40090],{"class":469,"line":8140},[151,40091,1090],{"emptyLinePlaceholder":609},[151,40093,40094,40096],{"class":469,"line":8145},[151,40095,37901],{"class":15289},[151,40097,40098],{"class":503},".discord_interaction_webhook.add_method(\n",[151,40100,40101,40104],{"class":469,"line":8259},[151,40102,40103],{"class":481},"            'POST'",[151,40105,9417],{"class":503},[151,40107,40108,40110],{"class":469,"line":8264},[151,40109,15290],{"class":15289},[151,40111,40112],{"class":503},".discord_interaction_webhook_integration\n",[151,40114,40115],{"class":469,"line":8613},[151,40116,16824],{"class":503},[11,40118,40119,40120,40122],{},"First we add some environment variables that will be made available to the Lambda function's execution environment. The ECS cluster and service name as well as our Discord application's ",[30,40121,38737],{}," are needed in the Lambda function for everything to work.",[11,40124,40125],{},"We have to give the lambda function permissions to make changes to ECS since it will be interacting with ECS via boto3.",[11,40127,40128,40131,40132,40134],{},[30,40129,40130],{},"self.request_templates"," is needed in order to pass the special security headers from the Discord ",[30,40133,36573],{}," request that are needed for security. I couldn't find a lot of resources on how to make this work, but I learned that this uses Apache Velocity Template Language.",[736,40136,40138],{"id":40137},"add-a-gitlab-ci-job-for-installing-dependencies-into-lambda-layer","Add a GitLab CI job for installing dependencies into Lambda Layer",[11,40140,40141],{},"There's one more step before we can push our code. We need to add another GitLab CI job that will install the Lambda dependencies so that they can be sent to the Lambda layer that we defined in our Lambda function. A Lambda Layer is where you install dependencies for this type of Lambda setup. Let's add the following stage:",[459,40143,40145],{"className":21928,"code":40144,"language":21930,"meta":464,"style":464},"stages:\n  - build\n  - deploy\n\nimage: python:3.8\n\npip_install:\n  stage: build\n  rules:\n    - if: \"$CI_COMMIT_TAG\"\n      when: always\n  artifacts:\n    paths:\n      - layers/flask/python\n  script:\n    - pip install -r lambda/functions/interactions/requirements.txt -t layers/flask/python\n",[30,40146,40147,40153,40160,40166,40170,40178,40182,40189,40197,40203,40213,40221,40228,40235,40242,40248],{"__ignoreMap":464},[151,40148,40149,40151],{"class":469,"line":470},[151,40150,38139],{"class":14368},[151,40152,14372],{"class":503},[151,40154,40155,40157],{"class":469,"line":488},[151,40156,19688],{"class":503},[151,40158,40159],{"class":481},"build\n",[151,40161,40162,40164],{"class":469,"line":500},[151,40163,19688],{"class":503},[151,40165,20676],{"class":481},[151,40167,40168],{"class":469,"line":509},[151,40169,1090],{"emptyLinePlaceholder":609},[151,40171,40172,40174,40176],{"class":469,"line":517},[151,40173,19666],{"class":14368},[151,40175,6208],{"class":503},[151,40177,38160],{"class":481},[151,40179,40180],{"class":469,"line":534},[151,40181,1090],{"emptyLinePlaceholder":609},[151,40183,40184,40187],{"class":469,"line":1413},[151,40185,40186],{"class":14368},"pip_install",[151,40188,14372],{"class":503},[151,40190,40191,40193,40195],{"class":469,"line":1418},[151,40192,38176],{"class":14368},[151,40194,6208],{"class":503},[151,40196,40159],{"class":481},[151,40198,40199,40201],{"class":469,"line":2462},[151,40200,38185],{"class":14368},[151,40202,14372],{"class":503},[151,40204,40205,40207,40209,40211],{"class":469,"line":2471},[151,40206,29541],{"class":503},[151,40208,17218],{"class":14368},[151,40210,6208],{"class":503},[151,40212,38198],{"class":481},[151,40214,40215,40217,40219],{"class":469,"line":2480},[151,40216,38203],{"class":14368},[151,40218,6208],{"class":503},[151,40220,38208],{"class":481},[151,40222,40223,40226],{"class":469,"line":2489},[151,40224,40225],{"class":14368},"  artifacts",[151,40227,14372],{"class":503},[151,40229,40230,40233],{"class":469,"line":2497},[151,40231,40232],{"class":14368},"    paths",[151,40234,14372],{"class":503},[151,40236,40237,40239],{"class":469,"line":3140},[151,40238,14459],{"class":503},[151,40240,40241],{"class":481},"layers/flask/python\n",[151,40243,40244,40246],{"class":469,"line":3149},[151,40245,38240],{"class":14368},[151,40247,14372],{"class":503},[151,40249,40250,40252],{"class":469,"line":3158},[151,40251,29541],{"class":503},[151,40253,40254],{"class":481},"pip install -r lambda/functions/interactions/requirements.txt -t layers/flask/python\n",[11,40256,40257,40258,40261,40262,40264,40265,22326,40268,40271,40272,22326,40275,40277],{},"Now we are installing dependencies into a target location (with the ",[30,40259,40260],{},"-t"," flag) that our Lambda Layer will be able to use in the ",[30,40263,38169],{}," GitLab CI job. This is because we have indicated the path to ",[30,40266,40267],{},"layers/flask/python",[30,40269,40270],{},"paths"," array of ",[30,40273,40274],{},"artifacts",[30,40276,40186],{}," job. There are other ways to add the pip dependencies to the Lambda Layers. We don't absolutely need this to be done in a separate CI job.",[11,40279,40280],{},"Now tag and push the code to GitLab and check to see that the pipeline runs successfully.",[736,40282,40284],{"id":40283},"add-the-api-gateway-url-to-discord-application-settings","Add the API Gateway URL to Discord Application settings",[11,40286,40287],{},"If everything runs smoothly, we should see a URL in the very last lines of the pipeline. This is the URL for our API Gateway endpoint:",[459,40289,40292],{"className":40290,"code":40291,"language":997},[995],"https://abc123xyz.execute-api.us-east-1.amazonaws.com/prod/\n",[30,40293,40291],{"__ignoreMap":464},[11,40295,40296,40297,40300,40301,22326,40303,40305],{},"We need to add ",[30,40298,40299],{},"discord"," to the end of this URL and then add that to our the ",[30,40302,37215],{},[30,40304,36651],{}," section of our Discord application's admin page:",[459,40307,40310],{"className":40308,"code":40309,"language":997},[995],"https://abc123xyz.execute-api.us-east-1.amazonaws.com/prod/discord\n",[30,40311,40309],{"__ignoreMap":464},[11,40313,40314,40315,40320],{},"When we add this URL in the application settings, Discord will make sure that our endpoint is properly verifying the request based on its headers. Check out ",[20,40316,40319],{"href":40317,"rel":40318},"https://github.com/discord/discord-interactions-python/blob/main/discord_interactions/__init__.py#L31",[24],"this function"," to see how it works:",[459,40322,40324],{"className":24401,"code":40323,"language":24403,"meta":464,"style":464},"def verify_key_decorator(client_public_key):\n    from flask import request, jsonify\n\n    # https://stackoverflow.com/questions/51691730/flask-middleware-for-specific-route\n    def _decorator(f):\n        @wraps(f)\n        def __decorator(*args, **kwargs):\n            # Verify request\n            signature = request.headers.get('X-Signature-Ed25519')\n            timestamp = request.headers.get('X-Signature-Timestamp')\n            if signature is None or timestamp is None or not verify_key(request.data, signature, timestamp, client_public_key):\n                return 'Bad request signature', 401\n\n            # Automatically respond to pings\n            if request.json and request.json.get('type') == InteractionType.PING:\n                return jsonify({\n                    'type': InteractionResponseType.PONG\n                })\n\n            # Pass through\n            return f(*args, **kwargs)\n        return __decorator\n    return _decorator\n",[30,40325,40326,40340,40352,40356,40361,40374,40382,40404,40409,40424,40438,40468,40481,40485,40490,40518,40524,40535,40540,40544,40549,40565,40572],{"__ignoreMap":464},[151,40327,40328,40330,40333,40335,40338],{"class":469,"line":470},[151,40329,16925],{"class":12347},[151,40331,40332],{"class":473}," verify_key_decorator",[151,40334,12386],{"class":503},[151,40336,40337],{"class":15232},"client_public_key",[151,40339,15264],{"class":503},[151,40341,40342,40345,40347,40349],{"class":469,"line":488},[151,40343,40344],{"class":1869},"    from",[151,40346,38677],{"class":503},[151,40348,16859],{"class":1869},[151,40350,40351],{"class":503}," request, jsonify\n",[151,40353,40354],{"class":469,"line":500},[151,40355,1090],{"emptyLinePlaceholder":609},[151,40357,40358],{"class":469,"line":509},[151,40359,40360],{"class":1527},"    # https://stackoverflow.com/questions/51691730/flask-middleware-for-specific-route\n",[151,40362,40363,40365,40368,40370,40372],{"class":469,"line":517},[151,40364,16566],{"class":12347},[151,40366,40367],{"class":473}," _decorator",[151,40369,12386],{"class":503},[151,40371,13214],{"class":15232},[151,40373,15264],{"class":503},[151,40375,40376,40379],{"class":469,"line":534},[151,40377,40378],{"class":473},"        @wraps",[151,40380,40381],{"class":503},"(f)\n",[151,40383,40384,40386,40389,40391,40393,40396,40398,40400,40402],{"class":469,"line":1413},[151,40385,15269],{"class":12347},[151,40387,40388],{"class":473}," __decorator",[151,40390,12386],{"class":503},[151,40392,23268],{"class":1869},[151,40394,40395],{"class":15232},"args",[151,40397,106],{"class":503},[151,40399,24677],{"class":1869},[151,40401,37866],{"class":15232},[151,40403,15264],{"class":503},[151,40405,40406],{"class":469,"line":1418},[151,40407,40408],{"class":1527},"            # Verify request\n",[151,40410,40411,40414,40416,40419,40422],{"class":469,"line":2462},[151,40412,40413],{"class":503},"            signature ",[151,40415,1876],{"class":1869},[151,40417,40418],{"class":503}," request.headers.get(",[151,40420,40421],{"class":481},"'X-Signature-Ed25519'",[151,40423,3640],{"class":503},[151,40425,40426,40429,40431,40433,40436],{"class":469,"line":2471},[151,40427,40428],{"class":503},"            timestamp ",[151,40430,1876],{"class":1869},[151,40432,40418],{"class":503},[151,40434,40435],{"class":481},"'X-Signature-Timestamp'",[151,40437,3640],{"class":503},[151,40439,40440,40443,40446,40449,40452,40454,40457,40459,40461,40463,40465],{"class":469,"line":2480},[151,40441,40442],{"class":1869},"            if",[151,40444,40445],{"class":503}," signature ",[151,40447,40448],{"class":1869},"is",[151,40450,40451],{"class":477}," None",[151,40453,2161],{"class":1869},[151,40455,40456],{"class":503}," timestamp ",[151,40458,40448],{"class":1869},[151,40460,40451],{"class":477},[151,40462,2161],{"class":1869},[151,40464,4191],{"class":1869},[151,40466,40467],{"class":503}," verify_key(request.data, signature, timestamp, client_public_key):\n",[151,40469,40470,40473,40476,40478],{"class":469,"line":2489},[151,40471,40472],{"class":1869},"                return",[151,40474,40475],{"class":481}," 'Bad request signature'",[151,40477,106],{"class":503},[151,40479,40480],{"class":477},"401\n",[151,40482,40483],{"class":469,"line":2497},[151,40484,1090],{"emptyLinePlaceholder":609},[151,40486,40487],{"class":469,"line":3140},[151,40488,40489],{"class":1527},"            # Automatically respond to pings\n",[151,40491,40492,40494,40497,40500,40503,40506,40508,40510,40513,40516],{"class":469,"line":3149},[151,40493,40442],{"class":1869},[151,40495,40496],{"class":503}," request.json ",[151,40498,40499],{"class":1869},"and",[151,40501,40502],{"class":503}," request.json.get(",[151,40504,40505],{"class":481},"'type'",[151,40507,16995],{"class":503},[151,40509,17223],{"class":1869},[151,40511,40512],{"class":503}," InteractionType.",[151,40514,40515],{"class":477},"PING",[151,40517,14372],{"class":503},[151,40519,40520,40522],{"class":469,"line":3158},[151,40521,40472],{"class":1869},[151,40523,39434],{"class":503},[151,40525,40526,40529,40532],{"class":469,"line":3167},[151,40527,40528],{"class":481},"                    'type'",[151,40530,40531],{"class":503},": InteractionResponseType.",[151,40533,40534],{"class":477},"PONG\n",[151,40536,40537],{"class":469,"line":3175},[151,40538,40539],{"class":503},"                })\n",[151,40541,40542],{"class":469,"line":3184},[151,40543,1090],{"emptyLinePlaceholder":609},[151,40545,40546],{"class":469,"line":3193},[151,40547,40548],{"class":1527},"            # Pass through\n",[151,40550,40551,40553,40556,40558,40561,40563],{"class":469,"line":3720},[151,40552,15386],{"class":1869},[151,40554,40555],{"class":503}," f(",[151,40557,23268],{"class":1869},[151,40559,40560],{"class":503},"args, ",[151,40562,24677],{"class":1869},[151,40564,37891],{"class":503},[151,40566,40567,40569],{"class":469,"line":3729},[151,40568,16833],{"class":1869},[151,40570,40571],{"class":503}," __decorator\n",[151,40573,40574,40576],{"class":469,"line":3735},[151,40575,17496],{"class":1869},[151,40577,40578],{"class":503}," _decorator\n",[11,40580,40581],{},"If it fails verification, we will not be able to add the URL and it will not work. You might want to add some additional logging to the Lambda function if you are not able to add the URL successfully.",[11,40583,40584,40585,643],{},"This is all covered in ",[20,40586,40589],{"href":40587,"rel":40588},"https://discord.com/developers/docs/interactions/slash-commands",[24],"the documentation for Discord Interactions",[11,40591,40592,40593,187,40596,643],{},"Now you should be able to run the Discord slash commands. You can get the status of your ECS cluster and scale it to either 1 or 0 for ",[30,40594,40595],{},"ON",[30,40597,40598],{},"OFF",[56,40600,40602],{"id":40601},"overview","Overview",[11,40604,40605],{},"Here's an overview of what we covered:",[11,40607,40608],{},[2718,40609],{"alt":20386,"src":36626},[700,40611,40613,40616,40619,40622,40625,40628,40636,40639,40645,40648,40657,40660,40663,40666,40668,40673,40679,40685,40694],{"start":40612},0,[79,40614,40615],{},"This is my computer. For development of this project (and most other projects) I used Windows with WSL2.",[79,40617,40618],{},"GitLab CI - This is used to run our automated pipelines whenever we push a tag.",[79,40620,40621],{},"The CDK CLI is used to create, update and delete the infrastructure in our AWS account.",[79,40623,40624],{},"Valheim - The client for the game server that we set up",[79,40626,40627],{},"The public IP address of the ECS Task that can be used to connect to our server on port 2456.",[79,40629,40630,40631,643],{},"The ECS Cluster that runs the actual docker container for the Valheim server. By default, the image used is ",[20,40632,40635],{"href":40633,"rel":40634},"https://hub.docker.com/r/lloesche/valheim-server",[24],"lloesche/valheim-server",[79,40637,40638],{},"EFS - This is the file system that is mounted onto the container of the ECS task where our game's world data is stored.",[79,40640,40641,40642,40644],{},"AWS Backup (Optional) - This is an optional feature of the ",[30,40643,36515],{}," construct that can make regular backups of our EFS file system.",[79,40646,40647],{},"Events (Optional) - AWS Events can be used to scale the number of ECS tasks between 0 and 1.",[79,40649,40650,40651,40656],{},"This is the ",[20,40652,40654],{"href":36522,"rel":40653},[24],[30,40655,36515],{}," construct that I use in this project.",[79,40658,40659],{},"S3 bucket for syncing data to and from EFS with DataSync (WIP)",[79,40661,40662],{},"DataSync for moving game data between EFS and S3.",[79,40664,40665],{},"The Slash Commands that we set up",[79,40667,33843],{},[79,40669,40670,40671,3888],{},"Discord Interactions sends and a ",[30,40672,36573],{},[79,40674,40675,40676,40678],{},"The API Gateway endpoint that we configured to handle Discord Interaction ",[30,40677,36573],{}," requests.",[79,40680,40681,40682,40684],{},"The Lambda function running a simple Flask app that responds to the Interaction ",[30,40683,36573],{}," request.",[79,40686,40687,40688,187,40690,40693],{},"boto3 - This is the AWS SDK Python library included in the Python execution environment that allows us to interact with the resources in our AWS account. In particular, the interactions we use from boto3 are the ",[30,40689,36577],{},[30,40691,40692],{},"describe_servics"," methods from the ECS module. This allows us to turn our server on and off and also get the status.",[79,40695,40696,40697,40700],{},"This represents the ",[30,40698,40699],{},"valheim-server-stack"," we defined in our CDK application.",[56,40702,21038],{"id":21037},[11,40704,40705],{},"There are still some things that I'm working on finalizing.",[76,40707,40708,40711,40714,40717,40720],{},[79,40709,40710],{},"DataSync for easily moving data between S3 and EFS",[79,40712,40713],{},"Report billing data with an additional slash command sub-command",[79,40715,40716],{},"Add tagging to the resources in our stack to make the billing command easier to implement.",[79,40718,40719],{},"Get feedback from the Discord, CDK and Valheim communities about what I can improve here",[79,40721,40722,40723],{},"Contribute to ",[20,40724,40726],{"href":36522,"rel":40725},[24],[30,40727,40728],{},"gotodeploy/cdk-valheim",[11,40730,40731],{},"Thanks for reading!",[589,40733,40734],{},"html pre.shiki code .sstjo, html code.shiki .sstjo{--shiki-default:#032F62;--shiki-dark:#9ECBFF;--shiki-sepia:#E6DB74}html pre.shiki code .sC2Qs, html code.shiki .sC2Qs{--shiki-default:#D73A49;--shiki-dark:#F97583;--shiki-sepia:#F92672}html pre.shiki code .sMOD_, html code.shiki .sMOD_{--shiki-default:#24292E;--shiki-dark:#E1E4E8;--shiki-sepia:#F8F8F2}html pre.shiki code .s7F3e, html code.shiki .s7F3e{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#AE81FF}html pre.shiki code .sq6CD, html code.shiki .sq6CD{--shiki-default:#D73A49;--shiki-default-font-style:inherit;--shiki-dark:#F97583;--shiki-dark-font-style:inherit;--shiki-sepia:#66D9EF;--shiki-sepia-font-style:italic}html pre.shiki code .sXSQT, html code.shiki .sXSQT{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#F8F8F2}html pre.shiki code .sTHNf, html code.shiki .sTHNf{--shiki-default:#E36209;--shiki-default-font-style:inherit;--shiki-dark:#FFAB70;--shiki-dark-font-style:inherit;--shiki-sepia:#FD971F;--shiki-sepia-font-style:italic}html pre.shiki code .sTrkL, html code.shiki .sTrkL{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#66D9EF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html .sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html.sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html pre.shiki code .s-m8C, html code.shiki .s-m8C{--shiki-default:#005CC5;--shiki-default-font-style:inherit;--shiki-dark:#79B8FF;--shiki-dark-font-style:inherit;--shiki-sepia:#66D9EF;--shiki-sepia-font-style:italic}html pre.shiki code .sCZoN, html code.shiki .sCZoN{--shiki-default:#032F62;--shiki-dark:#9ECBFF;--shiki-sepia:#CFCFC2}html pre.shiki code .sz2Vg, html code.shiki .sz2Vg{--shiki-default:#6F42C1;--shiki-default-text-decoration:inherit;--shiki-dark:#B392F0;--shiki-dark-text-decoration:inherit;--shiki-sepia:#A6E22E;--shiki-sepia-text-decoration:underline}html pre.shiki code .s30JN, html code.shiki .s30JN{--shiki-default:#6F42C1;--shiki-default-font-style:inherit;--shiki-default-text-decoration:inherit;--shiki-dark:#B392F0;--shiki-dark-font-style:inherit;--shiki-dark-text-decoration:inherit;--shiki-sepia:#A6E22E;--shiki-sepia-font-style:italic;--shiki-sepia-text-decoration:underline}html pre.shiki code .so59x, html code.shiki .so59x{--shiki-default:#24292E;--shiki-default-font-style:inherit;--shiki-dark:#E1E4E8;--shiki-dark-font-style:inherit;--shiki-sepia:#FD971F;--shiki-sepia-font-style:italic}html pre.shiki code .s8-w5, html code.shiki .s8-w5{--shiki-default:#6A737D;--shiki-dark:#6A737D;--shiki-sepia:#88846F}html pre.shiki code .sP7S_, html code.shiki .sP7S_{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#FD971F}html pre.shiki code .s5clZ, html code.shiki .s5clZ{--shiki-default:#22863A;--shiki-dark:#85E89D;--shiki-sepia:#F92672}html pre.shiki code .srTi1, html code.shiki .srTi1{--shiki-default:#6F42C1;--shiki-dark:#B392F0;--shiki-sepia:#A6E22E}",{"title":464,"searchDepth":488,"depth":488,"links":40736},[40737,40739,40740,40744,40745,40750,40751,40755,40756],{"id":36512,"depth":488,"text":40738},"cdk-valheim construct on GitHub",{"id":36554,"depth":488,"text":36555},{"id":36632,"depth":488,"text":36633,"children":40741},[40742,40743],{"id":36689,"depth":500,"text":36690},{"id":36745,"depth":500,"text":36746},{"id":37226,"depth":488,"text":37227},{"id":37568,"depth":488,"text":37569,"children":40746},[40747,40748,40749],{"id":37597,"depth":500,"text":37598},{"id":38121,"depth":500,"text":38122},{"id":38335,"depth":500,"text":38336},{"id":38560,"depth":488,"text":38561},{"id":39585,"depth":488,"text":39586,"children":40752},[40753,40754],{"id":40137,"depth":500,"text":40138},{"id":40283,"depth":500,"text":40284},{"id":40601,"depth":488,"text":40602},{"id":21037,"depth":488,"text":21038},"2021-03-18","This is a detailed guide showing how to setup a dedicated Valheim server using serverless technologies on AWS","/static/valheim/hall.png",{},"/2021/03/18/on-demand-dedicated-serverless-valheim-server-with-cdk-discrod-interactions",{"title":36489,"description":40758},"2021/03/18/on-demand-dedicated-serverless-valheim-server-with-cdk-discrod-interactions",[40765,23779,12886,40766,40767,40299],"valheim","flask","serverless","Ny58nS27CCHSGv1vGhH9bDRLah_mbbqRXYmqUP1uRG0",{"id":40770,"title":40771,"body":40772,"comments":602,"date":46083,"description":40771,"draft":602,"extension":605,"external":606,"image":44447,"meta":46084,"navigation":609,"path":46085,"seo":46086,"stem":46087,"tags":46088,"__hash__":46092},"blog/2021/01/16/i-scraped-analyzed-and-generated-yc-companies-founders-and-work-at-a-startup-job-postings.md","Scraping, analyzing and generating companies, founders and job postings from YC's Work at a Startup",{"type":8,"value":40773,"toc":46068},[40774,40783,40786,40789,40793,40800,41028,41032,41039,41156,41159,41190,41197,41200,41406,41410,41413,41417,41420,41541,41546,41549,43750,43753,43756,44000,44005,44009,44012,44017,44416,44419,44424,44428,44431,44436,44440,44443,44448,44949,44952,44956,44965,44968,44973,44976,45288,45291,45297,45495,45500,45504,45507,45512,45516,45519,45531,45545,45548,45554,45557,45631,45642,45645,45651,45657,45660,45679,45682,45691,45698,45718,45721,45842,45845,45851,45854,45859,45862,45867,45872,45878,45907,45917,45920,46003,46008,46012,46018,46024,46027,46033,46036,46041,46048,46055,46062,46065],[11,40775,40776,40777,40782],{},"I always enjoy reading about new batches of YC companies. I came across YC's ",[20,40778,40781],{"href":40779,"rel":40780},"https://www.workatastartup.com/",[24],"Work at a Startup"," (WaaS) recently while browsing HN and got pretty curious about all of the available data points on companies, jobs and founders.",[11,40784,40785],{},"This article will outline my process for collecting, cleaning, visualizing and analyzing the dataset.",[11,40787,40788],{},"After filling out my profile, WaaS recommended 750 matching YC startups which collectively list 1614 open positions. I think this is all of the available job openings and hiring companies, but I'm not sure.",[56,40790,40792],{"id":40791},"scraping-data","Scraping data",[11,40794,40795,40796,40799],{},"I've used a few different tools to scrape data and automate web browsers. For collecting this data, I ended up just writing some JavaScript directly in the browser console and ",[30,40797,40798],{},"Ctrl+S","aved the page HTML and assets (company logos and founder photos).",[459,40801,40803],{"className":12338,"code":40802,"language":12340,"meta":464,"style":464},"// expand each companies to see full details and all postings\nconst toggleDetails = document.getElementsByClassName(\"checkbox-inline\")[0]\ntoggleDetails.click()\n\n// automated scrolling, run this until it gets to the end\nconst scroll = setInterval(() => {window.scrollTo(0,document.body.scrollHeight);}, 3000)\n\n// when it no longer scrolls, clear the interval\nclearInterval(scroll)\n    \u003Cvue-apex-charts\n      v-if=\"chartOptions && series\"\n      type=\"bar\"\n      height=\"350\"\n      :options=\"chartOptions\"\n      :series=\"series\"\n    >\u003C/vue-apex-charts>\n// expand each job listing:\nconst jobs = document.getElementsByClassName(\"job-name\")\nfor (let job of jobs) {\n    job.click()\n}\n\n// now Ctrl+S to save the HTML and images\n",[30,40804,40805,40810,40837,40847,40851,40856,40890,40894,40899,40907,40914,40924,40934,40944,40949,40954,40964,40969,40989,41006,41015,41019,41023],{"__ignoreMap":464},[151,40806,40807],{"class":469,"line":470},[151,40808,40809],{"class":1527},"// expand each companies to see full details and all postings\n",[151,40811,40812,40814,40817,40819,40822,40825,40827,40830,40833,40835],{"class":469,"line":488},[151,40813,12348],{"class":12347},[151,40815,40816],{"class":12360}," toggleDetails",[151,40818,19865],{"class":1869},[151,40820,40821],{"class":503}," document.",[151,40823,40824],{"class":473},"getElementsByClassName",[151,40826,12386],{"class":503},[151,40828,40829],{"class":481},"\"checkbox-inline\"",[151,40831,40832],{"class":503},")[",[151,40834,9181],{"class":477},[151,40836,3691],{"class":503},[151,40838,40839,40842,40845],{"class":469,"line":500},[151,40840,40841],{"class":503},"toggleDetails.",[151,40843,40844],{"class":473},"click",[151,40846,12461],{"class":503},[151,40848,40849],{"class":469,"line":509},[151,40850,1090],{"emptyLinePlaceholder":609},[151,40852,40853],{"class":469,"line":517},[151,40854,40855],{"class":1527},"// automated scrolling, run this until it gets to the end\n",[151,40857,40858,40860,40863,40865,40868,40870,40872,40875,40878,40880,40882,40885,40888],{"class":469,"line":534},[151,40859,12348],{"class":12347},[151,40861,40862],{"class":12360}," scroll",[151,40864,19865],{"class":1869},[151,40866,40867],{"class":473}," setInterval",[151,40869,34120],{"class":503},[151,40871,17166],{"class":12347},[151,40873,40874],{"class":503}," {window.",[151,40876,40877],{"class":473},"scrollTo",[151,40879,12386],{"class":503},[151,40881,9181],{"class":477},[151,40883,40884],{"class":503},",document.body.scrollHeight);}, ",[151,40886,40887],{"class":477},"3000",[151,40889,3640],{"class":503},[151,40891,40892],{"class":469,"line":1413},[151,40893,1090],{"emptyLinePlaceholder":609},[151,40895,40896],{"class":469,"line":1418},[151,40897,40898],{"class":1527},"// when it no longer scrolls, clear the interval\n",[151,40900,40901,40904],{"class":469,"line":2462},[151,40902,40903],{"class":473},"clearInterval",[151,40905,40906],{"class":503},"(scroll)\n",[151,40908,40909,40911],{"class":469,"line":2471},[151,40910,34669],{"class":503},[151,40912,40913],{"class":6205},"vue-apex-charts\n",[151,40915,40916,40919,40921],{"class":469,"line":2480},[151,40917,40918],{"class":473},"      v-if",[151,40920,1876],{"class":1869},[151,40922,40923],{"class":481},"\"chartOptions && series\"\n",[151,40925,40926,40929,40931],{"class":469,"line":2489},[151,40927,40928],{"class":473},"      type",[151,40930,1876],{"class":1869},[151,40932,40933],{"class":481},"\"bar\"\n",[151,40935,40936,40939,40941],{"class":469,"line":2497},[151,40937,40938],{"class":473},"      height",[151,40940,1876],{"class":1869},[151,40942,40943],{"class":481},"\"350\"\n",[151,40945,40946],{"class":469,"line":3140},[151,40947,40948],{"class":6607},"      :options=\"chartOptions\"\n",[151,40950,40951],{"class":469,"line":3149},[151,40952,40953],{"class":6607},"      :series=\"series\"\n",[151,40955,40956,40959,40962],{"class":469,"line":3158},[151,40957,40958],{"class":503},"    >\u003C/",[151,40960,40961],{"class":6205},"vue-apex-charts",[151,40963,3742],{"class":503},[151,40965,40966],{"class":469,"line":3167},[151,40967,40968],{"class":1527},"// expand each job listing:\n",[151,40970,40971,40973,40976,40978,40980,40982,40984,40987],{"class":469,"line":3175},[151,40972,12348],{"class":12347},[151,40974,40975],{"class":12360}," jobs",[151,40977,19865],{"class":1869},[151,40979,40821],{"class":503},[151,40981,40824],{"class":473},[151,40983,12386],{"class":503},[151,40985,40986],{"class":481},"\"job-name\"",[151,40988,3640],{"class":503},[151,40990,40991,40993,40995,40997,41000,41003],{"class":469,"line":3184},[151,40992,16732],{"class":1869},[151,40994,129],{"class":503},[151,40996,34469],{"class":12347},[151,40998,40999],{"class":503}," job ",[151,41001,41002],{"class":1869},"of",[151,41004,41005],{"class":503}," jobs) {\n",[151,41007,41008,41011,41013],{"class":469,"line":3193},[151,41009,41010],{"class":503},"    job.",[151,41012,40844],{"class":473},[151,41014,12461],{"class":503},[151,41016,41017],{"class":469,"line":3720},[151,41018,6274],{"class":503},[151,41020,41021],{"class":469,"line":3729},[151,41022,1090],{"emptyLinePlaceholder":609},[151,41024,41025],{"class":469,"line":3735},[151,41026,41027],{"class":1527},"// now Ctrl+S to save the HTML and images\n",[56,41029,41031],{"id":41030},"parsing-the-html","Parsing the HTML",[11,41033,41034,41035,41038],{},"Next I'll parse the company data into a python list of dictionaries and then ",[30,41036,41037],{},"dumps"," it into a JSON file. This code is a little bit scrappy, here's the pseudo code:",[459,41040,41042],{"className":24401,"code":41041,"language":24403,"meta":464,"style":464},"# pseudo code for parsing data\nhtml = open(\"data.html\")\nparsed_html = parseHtml(html)\n\ncompanies = []\nfor company in parsed_html.find_all(\"company\")\n    # company stats, founders and jobs\n    company_details = extract_company_details(company)\n    companies.append(company_details)\n\nwith open(\"output.json\", \"wb\") as f:\n    f.write(json.dumps(companies))\n",[30,41043,41044,41049,41065,41075,41079,41088,41105,41110,41120,41125,41129,41151],{"__ignoreMap":464},[151,41045,41046],{"class":469,"line":470},[151,41047,41048],{"class":1527},"# pseudo code for parsing data\n",[151,41050,41051,41054,41056,41058,41060,41063],{"class":469,"line":488},[151,41052,41053],{"class":503},"html ",[151,41055,1876],{"class":1869},[151,41057,16970],{"class":2226},[151,41059,12386],{"class":503},[151,41061,41062],{"class":481},"\"data.html\"",[151,41064,3640],{"class":503},[151,41066,41067,41070,41072],{"class":469,"line":500},[151,41068,41069],{"class":503},"parsed_html ",[151,41071,1876],{"class":1869},[151,41073,41074],{"class":503}," parseHtml(html)\n",[151,41076,41077],{"class":469,"line":509},[151,41078,1090],{"emptyLinePlaceholder":609},[151,41080,41081,41084,41086],{"class":469,"line":517},[151,41082,41083],{"class":503},"companies ",[151,41085,1876],{"class":1869},[151,41087,16606],{"class":503},[151,41089,41090,41092,41095,41097,41100,41103],{"class":469,"line":534},[151,41091,16732],{"class":1869},[151,41093,41094],{"class":503}," company ",[151,41096,16417],{"class":1869},[151,41098,41099],{"class":503}," parsed_html.find_all(",[151,41101,41102],{"class":481},"\"company\"",[151,41104,3640],{"class":503},[151,41106,41107],{"class":469,"line":1413},[151,41108,41109],{"class":1527},"    # company stats, founders and jobs\n",[151,41111,41112,41115,41117],{"class":469,"line":1418},[151,41113,41114],{"class":503},"    company_details ",[151,41116,1876],{"class":1869},[151,41118,41119],{"class":503}," extract_company_details(company)\n",[151,41121,41122],{"class":469,"line":2462},[151,41123,41124],{"class":503},"    companies.append(company_details)\n",[151,41126,41127],{"class":469,"line":2471},[151,41128,1090],{"emptyLinePlaceholder":609},[151,41130,41131,41133,41135,41137,41140,41142,41145,41147,41149],{"class":469,"line":2480},[151,41132,24959],{"class":1869},[151,41134,16970],{"class":2226},[151,41136,12386],{"class":503},[151,41138,41139],{"class":481},"\"output.json\"",[151,41141,106],{"class":503},[151,41143,41144],{"class":481},"\"wb\"",[151,41146,16995],{"class":503},[151,41148,16998],{"class":1869},[151,41150,17001],{"class":503},[151,41152,41153],{"class":469,"line":2489},[151,41154,41155],{"class":503},"    f.write(json.dumps(companies))\n",[11,41157,41158],{},"To scrape the data I used my go-to library for this type of task: BeautifulSoup. There were a few tricky parts:",[76,41160,41161,41164,41178],{},[79,41162,41163],{},"Job details (visa requirements, salary, equity) were all labelled with the same class and they were inconsistent (sometimes salary or equity or both were excluded, for example).",[79,41165,41166,41167,41170,41171,41174,41175],{},"Equity was mostly a range of percentages such as ",[30,41168,41169],{},"1% - 2%"," and sometimes a single percentage like ",[30,41172,41173],{},"1.5%",". Some salary ranges had typos like ",[30,41176,41177],{},"$90k - $10k",[79,41179,41180,41181,106,41184,187,41187,23724],{},"Years of experience required was also inconsistent with mixed types like ",[30,41182,41183],{},"3+ Years",[30,41185,41186],{},"Any (recent grad ok)",[30,41188,41189],{},"Senior or Juniors",[11,41191,41192,41193,41196],{},"These were all pretty easy to account for, it just required some additional logic to handle default values for ",[30,41194,41195],{},"\u003Cdiv>","s that were not included as well as mixed data types and representations where there were inconsistencies.",[11,41198,41199],{},"The resulting JSON structure for the big array of companies looks like this:",[459,41201,41203],{"className":6194,"code":41202,"language":6196,"meta":464,"style":464},"[\n    {\n        \"company_name\": \"Startup A\",\n        \"logo\": \"logo.png\",\n        \"jobs\": [\n            {\n                \"title\": \"Software Engineer\",\n                \"skills\": [\"python\", \"javascript\"],\n                \"salary\": {\n                    \"min\": 90000,\n                    \"max\": 110000,\n                    \"avg\": 100000\n                }\n            }\n        ],\n        \"founders\": [\n            {\n                \"name\": \"Founder Name\",\n                \"linkedin\": \"https://linkedin.com/founder\",\n                \"education\": \"University A\",\n                \"image\": \"abc.png\"\n            }\n        ]\n    }\n]\n",[30,41204,41205,41209,41213,41225,41237,41244,41249,41261,41278,41285,41297,41309,41319,41323,41327,41332,41339,41343,41355,41367,41379,41389,41393,41398,41402],{"__ignoreMap":464},[151,41206,41207],{"class":469,"line":470},[151,41208,37620],{"class":503},[151,41210,41211],{"class":469,"line":488},[151,41212,9404],{"class":503},[151,41214,41215,41218,41220,41223],{"class":469,"line":500},[151,41216,41217],{"class":6205},"        \"company_name\"",[151,41219,6208],{"class":503},[151,41221,41222],{"class":6211},"\"Startup A\"",[151,41224,9417],{"class":503},[151,41226,41227,41230,41232,41235],{"class":469,"line":509},[151,41228,41229],{"class":6205},"        \"logo\"",[151,41231,6208],{"class":503},[151,41233,41234],{"class":6211},"\"logo.png\"",[151,41236,9417],{"class":503},[151,41238,41239,41242],{"class":469,"line":517},[151,41240,41241],{"class":6205},"        \"jobs\"",[151,41243,9399],{"class":503},[151,41245,41246],{"class":469,"line":534},[151,41247,41248],{"class":503},"            {\n",[151,41250,41251,41254,41256,41259],{"class":469,"line":1413},[151,41252,41253],{"class":6205},"                \"title\"",[151,41255,6208],{"class":503},[151,41257,41258],{"class":6211},"\"Software Engineer\"",[151,41260,9417],{"class":503},[151,41262,41263,41266,41268,41271,41273,41276],{"class":469,"line":1418},[151,41264,41265],{"class":6205},"                \"skills\"",[151,41267,8365],{"class":503},[151,41269,41270],{"class":6211},"\"python\"",[151,41272,106],{"class":503},[151,41274,41275],{"class":6211},"\"javascript\"",[151,41277,18746],{"class":503},[151,41279,41280,41283],{"class":469,"line":2462},[151,41281,41282],{"class":6205},"                \"salary\"",[151,41284,21223],{"class":503},[151,41286,41287,41290,41292,41295],{"class":469,"line":2471},[151,41288,41289],{"class":6205},"                    \"min\"",[151,41291,6208],{"class":503},[151,41293,41294],{"class":477},"90000",[151,41296,9417],{"class":503},[151,41298,41299,41302,41304,41307],{"class":469,"line":2480},[151,41300,41301],{"class":6205},"                    \"max\"",[151,41303,6208],{"class":503},[151,41305,41306],{"class":477},"110000",[151,41308,9417],{"class":503},[151,41310,41311,41314,41316],{"class":469,"line":2489},[151,41312,41313],{"class":6205},"                    \"avg\"",[151,41315,6208],{"class":503},[151,41317,41318],{"class":477},"100000\n",[151,41320,41321],{"class":469,"line":2497},[151,41322,37060],{"class":503},[151,41324,41325],{"class":469,"line":3140},[151,41326,39498],{"class":503},[151,41328,41329],{"class":469,"line":3149},[151,41330,41331],{"class":503},"        ],\n",[151,41333,41334,41337],{"class":469,"line":3158},[151,41335,41336],{"class":6205},"        \"founders\"",[151,41338,9399],{"class":503},[151,41340,41341],{"class":469,"line":3167},[151,41342,41248],{"class":503},[151,41344,41345,41348,41350,41353],{"class":469,"line":3175},[151,41346,41347],{"class":6205},"                \"name\"",[151,41349,6208],{"class":503},[151,41351,41352],{"class":6211},"\"Founder Name\"",[151,41354,9417],{"class":503},[151,41356,41357,41360,41362,41365],{"class":469,"line":3184},[151,41358,41359],{"class":6205},"                \"linkedin\"",[151,41361,6208],{"class":503},[151,41363,41364],{"class":6211},"\"https://linkedin.com/founder\"",[151,41366,9417],{"class":503},[151,41368,41369,41372,41374,41377],{"class":469,"line":3193},[151,41370,41371],{"class":6205},"                \"education\"",[151,41373,6208],{"class":503},[151,41375,41376],{"class":6211},"\"University A\"",[151,41378,9417],{"class":503},[151,41380,41381,41384,41386],{"class":469,"line":3720},[151,41382,41383],{"class":6205},"                \"image\"",[151,41385,6208],{"class":503},[151,41387,41388],{"class":6211},"\"abc.png\"\n",[151,41390,41391],{"class":469,"line":3729},[151,41392,39498],{"class":503},[151,41394,41395],{"class":469,"line":3735},[151,41396,41397],{"class":503},"        ]\n",[151,41399,41400],{"class":469,"line":3745},[151,41401,9461],{"class":503},[151,41403,41404],{"class":469,"line":3754},[151,41405,3691],{"class":503},[56,41407,41409],{"id":41408},"analysis","Analysis",[11,41411,41412],{},"Here are some of the biggest questions I wanted to answer along with some simple python I used for extracting data from the main dictionary/JSON object containing all companies and jobs. For the following code, assume I have read the JSON file back into a python dictionary.",[736,41414,41416],{"id":41415},"what-are-the-most-in-demand-skills-for-yc-jobs","What are the most in demand skills for YC Jobs?",[11,41418,41419],{},"I think skills are included mostly for engineering roles (not so much for sales, marketing, etc.). Here are the top skills:",[459,41421,41423],{"className":24401,"code":41422,"language":24403,"meta":464,"style":464},"skills = []\nfor company in company_list:\n    if company[\"jobs\"] is not None:\n        for job in company[\"jobs\"]:\n            if job[\"job_skills\"] is not None:\n                for skill in job[\"job_skills\"]:\n                    skills.append(skill)\n\ntop_skills = Counter(skills).most_common()\nprint(top_skills)\n",[30,41424,41425,41434,41445,41465,41479,41499,41515,41520,41524,41534],{"__ignoreMap":464},[151,41426,41427,41430,41432],{"class":469,"line":470},[151,41428,41429],{"class":503},"skills ",[151,41431,1876],{"class":1869},[151,41433,16606],{"class":503},[151,41435,41436,41438,41440,41442],{"class":469,"line":488},[151,41437,16732],{"class":1869},[151,41439,41094],{"class":503},[151,41441,16417],{"class":1869},[151,41443,41444],{"class":503}," company_list:\n",[151,41446,41447,41449,41452,41455,41457,41459,41461,41463],{"class":469,"line":500},[151,41448,23327],{"class":1869},[151,41450,41451],{"class":503}," company[",[151,41453,41454],{"class":481},"\"jobs\"",[151,41456,16654],{"class":503},[151,41458,40448],{"class":1869},[151,41460,4191],{"class":1869},[151,41462,40451],{"class":477},[151,41464,14372],{"class":503},[151,41466,41467,41469,41471,41473,41475,41477],{"class":469,"line":509},[151,41468,16616],{"class":1869},[151,41470,40999],{"class":503},[151,41472,16417],{"class":1869},[151,41474,41451],{"class":503},[151,41476,41454],{"class":481},[151,41478,17073],{"class":503},[151,41480,41481,41483,41486,41489,41491,41493,41495,41497],{"class":469,"line":517},[151,41482,40442],{"class":1869},[151,41484,41485],{"class":503}," job[",[151,41487,41488],{"class":481},"\"job_skills\"",[151,41490,16654],{"class":503},[151,41492,40448],{"class":1869},[151,41494,4191],{"class":1869},[151,41496,40451],{"class":477},[151,41498,14372],{"class":503},[151,41500,41501,41504,41507,41509,41511,41513],{"class":469,"line":534},[151,41502,41503],{"class":1869},"                for",[151,41505,41506],{"class":503}," skill ",[151,41508,16417],{"class":1869},[151,41510,41485],{"class":503},[151,41512,41488],{"class":481},[151,41514,17073],{"class":503},[151,41516,41517],{"class":469,"line":1413},[151,41518,41519],{"class":503},"                    skills.append(skill)\n",[151,41521,41522],{"class":469,"line":1418},[151,41523,1090],{"emptyLinePlaceholder":609},[151,41525,41526,41529,41531],{"class":469,"line":2462},[151,41527,41528],{"class":503},"top_skills ",[151,41530,1876],{"class":1869},[151,41532,41533],{"class":503}," Counter(skills).most_common()\n",[151,41535,41536,41538],{"class":469,"line":2471},[151,41537,18513],{"class":2226},[151,41539,41540],{"class":503},"(top_skills)\n",[142,41542,41543],{},[41544,41545],"skill-count",{},[11,41547,41548],{},"I'll try to briefly describe what I know about each of these if I know what it means (without Googling!):",[459,41550,41552],{"className":24401,"code":41551,"language":24403,"meta":464,"style":464},"\n[\n ('JAVASCRIPT', 330), # I have been using JS a lot recently with Vue\n ('REACT', 323), # As a prefer to use Vue, I haven't used React in a while\n ('PYTHON', 312), # I'm a big Python fan, it was the first language I touched, happy to see it near the top!\n ('AMAZON WEB SERVICES (AWS)', 200), # I like AWS a lot. I have really been enjoying using CDK to build infrastructure\n ('NODE.JS', 195), # I would to do more with node this year. I generally use Python for web apps\n ('POSTGRESQL', 132), # I have used Postgres ever since I started using Django and like it a lot\n ('TYPESCRIPT', 114), # this is another goal of mine for 2021, it seems like an inevitability\n ('JAVA', 79), # I have never used Java\n ('SQL', 74), # I usually don't write my own SQL queries; I view SQL through the lense of an ORM\n ('RUBY ON RAILS', 72), # I haven't used RoR\n ('CSS', 71), # I like CSS Frameworks. Recently I'm into Tailwind and Material UI. This site uses Tailwind\n ('HTML', 71),\n ('DOCKER', 66), # I am a big container fan! It is my preferred way to run software, locally and in the cloud\n ('KUBERNETES', 58), # I read the Manning book on k8s. I prefer ECS or Swarm, but I might try using it more\n ('GO', 58), # I haven't ever used Go, but it doesn't look too bad coming from Python\n ('REACT NATIVE', 58),\n ('C++', 55), # Also haven't used this, but I have a book on it\n ('GRAPHQL', 48), # I tried GraphQL and built a HN clone in Django. I prefer REST but I get the appeal (for frontend developers)\n ('GOOGLE CLOUD', 46), # I'm not a big GCP as I mostly use AWS and Digital Ocean but I would like to try Cloud Run\n ('RUBY', 44), # I haven't used it\n ('DJANGO', 44), # Django is my go-to tool for building web apps and APIs. I love the admin, ORM and DRF\n ('MACHINE LEARNING', 44), # I am familiar with some ML techniques but not very well practiced\n ('MONGODB', 43), # Have used it before, but I try to use the postgres JSONField for storing NoSQL data when possible\n ('IOS', 38), # I have an iPhone, but haven't used a Mac in a long time, mostly on Linux and Windows\n ('MYSQL', 36), # I tend to use Postgres, I don't think I've ever used this\n ('ANDROID', 35), # Not something I have worked with\n ('DATA ANALYTICS', 32), # I do a lot of this\n ('GIT', 30), # I'm very slowly trying to learn advanced git features. I have many abandoned \"rebase-practice\" repos\n ('ANGULAR', 29), # I tried it once for about an hour, I know that people love to hate it, I'm just not sure why\n ('SWIFT', 29), # This is apparently a very popular language and has use cases outside of mobile development, but I've never used it\n ('LINUX', 28), # I spend lot of time using Linux machines, mostly Ubuntu.\n ('SOFTWARE ARCHITECTURE', 24), # I like using diagrams.net to draw application infrastructure\n ('KOTLIN', 23), # I think this is a framework for Java/Android?\n ('TENSORFLOW', 22), # Google DL/ML library that I'll try to use later in this article\n ('DISTRIBUTED SYSTEMS', 22), # Using AWS, I guess I have technically designed distributed systems but I wouldn't call it one of my skills\n ('PHP', 22), # I almost learned it to support a WordPress site but opted to use JAMStack instead\n ('DATA WAREHOUSING', 22), # I have used Google BigQuery before which I think counts for this skill\n ('DEEP LEARNING', 20), # I'm going to try to use this later in this article\n ('DATA MODELING', 20), # To me this means writing Django models, or thinking about how to structure an API/json data, etc.\n ('C#', 19), # Micrsoft language used for different things including game dev with Unity\n ('FLASK', 19), # I'm familiar with it but mostly prefer Django's batteries included philosophy\n ('C', 19), # In learning about Linux I have read a bit of C, but never written any\n ('REDIS', 18), # Fast, in memory key-value store with multiple data types. I use it for a few different things\n ('MICROSERVICES', 18), #\n ('COMPUTER VISION', 17), # I once used OpenCV on my Raspberry Pi\n ('EXPRESS', 15), # I'd like to learn how to use this in 2021\n ('BASH/SHELL', 13), # I'm not very fluent in bash but\n ('OBJECTIVE-C', 13), # I haven't used this but I know it is popular for iOS development\n ('FIREBASE', 12), # I have played around with Firebase, but I haven't built anything with it\n ('SCALA', 11), # Functional programming language for JVM, I haven't used it\n ('SOFTWARE SECURITY', 11),\n ('UNITY', 11), # I used this once before to play around with VR development for HTC Vive\n ('R', 11), # I haven't used R before, and I would probably reach for a Python library for doing statistics or ML-related things\n ('KAFKA', 10), # I haven't used it but I'm familiar with the ideas behind Kafka.\n ('SPARK', 10), # I haven't used it and I'm not really sure what it is\n ('ELASTICSEARCH', 10), # I haven't used it before\n ('ETL', 10), # Extract, Transform and Load\n ('NATURAL LANGUAGE PROCESSING', 10),\n ('HEROKU', 10), # I used this when first learning about Django, haven't used it in a while\n ('NGINX', 9), # I use NGINX in most of my web apps as a reverse proxy\n ('JENKINS', 9), # I haven't used it, I am a big GitLab fan and will use that whenever possible\n ('RUST', 9), # I have read the Rust book and have played around with WASM\n ('IMAGE PROCESSING', 8),\n ('SERVERLESS', 8), # I currenty use Fargate and have also used Lambda and SQS and some other serverless AWS tools\n ('BLOCKCHAIN', 8),\n ('OPENCV', 8),\n ('CAD DESIGN', 7), # I am a big fan of SketchUp and I'm familiar with Blender as well, but I'm not sure if these qualify as CAD design\n ('JQUERY', 7), # It was the first library I worked with when I started learning javascript\n ('HADOOP', 6), # It is related to map-reduce, I haven't used it before and I don't really know what it is. I know it is related to HDFS\n ('.NET CORE', 6), # I haven't used it\n ('TCP/IP', 6), # I'm familiary with the basics\n ('ELIXIR', 6), # I don't know, I think it is a framework for Erlang\n ('INTERNET OF THINGS (IOT)', 5),\n ('SASS', 5), # I think it is a framework for CSS. I frequently see node-sass errors from npm\n ('OPENGL', 5),\n ('DYNAMODB', 5), # I am familiar with it but haven't used it\n ('GOOGLE APP ENGINE', 5), # I haven't used it before\n ('UNIX', 4),\n ('SPRING FRAMEWORK', 4), # A Java web framework that I haven't used\n ('CUDA', 4), # I have used it indirectly when I used nvidia-docker to used Tensorflow\n ('DART', 4), # I don't know what this is\n ('ERLANG', 4), # A language that handles concurrency very well\n ('RABBITMQ', 4), # Message queue, I have used it before but tend to use Redis as a message broker\n ('KERAS', 4), # A helper/wrapper library for Tensorflow\n ('SCSS', 4), # I think this is a language that compiles to CSS. CSS frameworks that I use use this\n ('ML', 4), # I've read some books and experimented but I'm not a regular practitioner\n ('MATLAB', 4), # A tool her programming with higher math\n ('SPRING', 4), # A Java web framework. Not sure if different from Spring Boot. Never used it.\n ('CASSANDRA', 4), # FB scalable database (NoSQL I think?) that does sharding really well\n ('HIVE', 3), # Not sure what hive is. I think it related to Hadoop\n ('PUPPET', 3), # Configuration management tool that I haven't ever used\n ('REDSHIFT', 3), # AWS version of Google BigQuery\n ('SQL SERVER', 3), # Not sure what this refers to, specifically.\n ('GROOVY', 3), # I think this is a Java framework?\n ('VERILOG', 3), # I've never heard of this.\n ('TORCH/PYTORCH', 3), # FB python deep learning library.\n ('CLOJURE', 3), # A LISP derivate, functional language\n ('MICROSOFT AZURE', 3), # I've used Azure AD and thats it.\n ('HBASE', 2), # I don't know what this is\n ('RDS/AURORA', 2), # I use RDS and experimented with Aurora but don't know when/why to use it\n ('FIRMWARE', 2), # What's between hardware and software\n ('ABAP', 2), # I don't know\n ('ARDUINO', 2), # I have one, but don't use it\n ('MICROCONTROLLERS', 2), # Arduino might be an example of what this is\n ('SOLIDITY', 2), # Don't know what this is\n ('UNREAL ENGINE', 2), # Unity competitor, used for game development\n ('COFFEESCRIPT', 2), # I think it is a dialect of JS, but I'm not sure\n ('LUA', 2), # I think this is what redis is written in, but I'm not sure how to descbribe what it is\n ('MACOS', 2), # I haven't used MacOS in a long time. I'm tempted to try M1, but I also want to buid a new PC...\n ('NEO4J', 2), # A graph database, I'm not sure what a typical use case is for this\n ('INFORMATION SECURITY', 2), # Unknown unknowns\n ('REINFORCEMENT LEARNING (RL)', 2), # Not sure what this refers to specifically\n ('DEVICE DRIVERS', 2), # Probably involves writing kernel modules\n ('EMBEDDED LINUX', 2), # Not sure if Raspberry Pi is an example of this or not\n ('ELASTIC STACK (ELK)', 1), # Useful for viewing log data (Elastic, Logstash and Kibana), haven't used it\n ('IIS', 1), # I don't know\n ('ORACLE', 1), # A big software company and a proprietary database (I think Django supports it)\n ('F#', 1), # A programming language that I don't know anything about\n ('SQLITE', 1), # A light-weight SQL-compatible file-based database\n ('HASKELL', 1), # A functional programming language that I don't know\n ('SCHEME', 1), # I'm not sure, it may be something related to LISP\n ('MS SQL', 1), # Never used this\n ('MARIADB', 1), # An open source SQL database\n ('MAVEN', 1), # I think it is a Java Framework\n ('SEARCH', 1), # I don't know what this refers to\n ('OCAML', 1), # I think I once read that high-frequency traders like to use this language, but I'm not sure why\n ('JULIA', 1), # A programming language used for math and statistics\n ('GPU PROGRAMMING', 1), # I haven't done this before, probably uses C++\n ('HACK', 1), # FB's version of PHP\n ('XAMARIN', 1), # I dont't know what this is\n ('CORDOVA', 1), # I think it is a tool for generating native apps from JS\n ('SAS', 1), # I don't know what this is\n ('ASSEMBLY', 1), # Low level language that gives instructions to CPU\n ('XML', 1), # A data format, I use it for this site's sitemap and RSS feed\n ('MEMCACHED', 1), # Used for caching. I haven't used it; I typically use redis where this might be an option\n ('LESS', 1), # I think is is related to CSS?\n ('AMAZON ECHO', 1) # I once built an open source Echo on a raspberry pi\n]\n",[30,41553,41554,41558,41562,41579,41596,41613,41630,41647,41664,41681,41698,41715,41731,41748,41761,41778,41795,41811,41824,41841,41858,41874,41891,41907,41923,41940,41956,41973,41990,42006,42023,42039,42055,42071,42087,42104,42121,42137,42153,42169,42185,42201,42218,42234,42250,42266,42282,42299,42316,42333,42349,42366,42383,42396,42412,42428,42444,42460,42476,42492,42505,42521,42537,42553,42569,42582,42598,42611,42624,42640,42656,42672,42687,42703,42719,42732,42748,42761,42777,42792,42805,42821,42837,42853,42869,42885,42901,42917,42933,42949,42965,42981,42997,43013,43029,43045,43061,43077,43093,43109,43125,43140,43156,43172,43188,43204,43220,43236,43252,43268,43284,43300,43316,43332,43348,43364,43380,43396,43411,43427,43443,43459,43475,43491,43507,43523,43539,43555,43571,43587,43603,43619,43635,43651,43666,43682,43698,43714,43730,43746],{"__ignoreMap":464},[151,41555,41556],{"class":469,"line":470},[151,41557,1090],{"emptyLinePlaceholder":609},[151,41559,41560],{"class":469,"line":488},[151,41561,37620],{"class":503},[151,41563,41564,41566,41569,41571,41574,41576],{"class":469,"line":500},[151,41565,129],{"class":503},[151,41567,41568],{"class":481},"'JAVASCRIPT'",[151,41570,106],{"class":503},[151,41572,41573],{"class":477},"330",[151,41575,24817],{"class":503},[151,41577,41578],{"class":1527},"# I have been using JS a lot recently with Vue\n",[151,41580,41581,41583,41586,41588,41591,41593],{"class":469,"line":509},[151,41582,129],{"class":503},[151,41584,41585],{"class":481},"'REACT'",[151,41587,106],{"class":503},[151,41589,41590],{"class":477},"323",[151,41592,24817],{"class":503},[151,41594,41595],{"class":1527},"# As a prefer to use Vue, I haven't used React in a while\n",[151,41597,41598,41600,41603,41605,41608,41610],{"class":469,"line":517},[151,41599,129],{"class":503},[151,41601,41602],{"class":481},"'PYTHON'",[151,41604,106],{"class":503},[151,41606,41607],{"class":477},"312",[151,41609,24817],{"class":503},[151,41611,41612],{"class":1527},"# I'm a big Python fan, it was the first language I touched, happy to see it near the top!\n",[151,41614,41615,41617,41620,41622,41625,41627],{"class":469,"line":534},[151,41616,129],{"class":503},[151,41618,41619],{"class":481},"'AMAZON WEB SERVICES (AWS)'",[151,41621,106],{"class":503},[151,41623,41624],{"class":477},"200",[151,41626,24817],{"class":503},[151,41628,41629],{"class":1527},"# I like AWS a lot. I have really been enjoying using CDK to build infrastructure\n",[151,41631,41632,41634,41637,41639,41642,41644],{"class":469,"line":1413},[151,41633,129],{"class":503},[151,41635,41636],{"class":481},"'NODE.JS'",[151,41638,106],{"class":503},[151,41640,41641],{"class":477},"195",[151,41643,24817],{"class":503},[151,41645,41646],{"class":1527},"# I would to do more with node this year. I generally use Python for web apps\n",[151,41648,41649,41651,41654,41656,41659,41661],{"class":469,"line":1418},[151,41650,129],{"class":503},[151,41652,41653],{"class":481},"'POSTGRESQL'",[151,41655,106],{"class":503},[151,41657,41658],{"class":477},"132",[151,41660,24817],{"class":503},[151,41662,41663],{"class":1527},"# I have used Postgres ever since I started using Django and like it a lot\n",[151,41665,41666,41668,41671,41673,41676,41678],{"class":469,"line":2462},[151,41667,129],{"class":503},[151,41669,41670],{"class":481},"'TYPESCRIPT'",[151,41672,106],{"class":503},[151,41674,41675],{"class":477},"114",[151,41677,24817],{"class":503},[151,41679,41680],{"class":1527},"# this is another goal of mine for 2021, it seems like an inevitability\n",[151,41682,41683,41685,41688,41690,41693,41695],{"class":469,"line":2471},[151,41684,129],{"class":503},[151,41686,41687],{"class":481},"'JAVA'",[151,41689,106],{"class":503},[151,41691,41692],{"class":477},"79",[151,41694,24817],{"class":503},[151,41696,41697],{"class":1527},"# I have never used Java\n",[151,41699,41700,41702,41705,41707,41710,41712],{"class":469,"line":2480},[151,41701,129],{"class":503},[151,41703,41704],{"class":481},"'SQL'",[151,41706,106],{"class":503},[151,41708,41709],{"class":477},"74",[151,41711,24817],{"class":503},[151,41713,41714],{"class":1527},"# I usually don't write my own SQL queries; I view SQL through the lense of an ORM\n",[151,41716,41717,41719,41722,41724,41726,41728],{"class":469,"line":2489},[151,41718,129],{"class":503},[151,41720,41721],{"class":481},"'RUBY ON RAILS'",[151,41723,106],{"class":503},[151,41725,9232],{"class":477},[151,41727,24817],{"class":503},[151,41729,41730],{"class":1527},"# I haven't used RoR\n",[151,41732,41733,41735,41738,41740,41743,41745],{"class":469,"line":2497},[151,41734,129],{"class":503},[151,41736,41737],{"class":481},"'CSS'",[151,41739,106],{"class":503},[151,41741,41742],{"class":477},"71",[151,41744,24817],{"class":503},[151,41746,41747],{"class":1527},"# I like CSS Frameworks. Recently I'm into Tailwind and Material UI. This site uses Tailwind\n",[151,41749,41750,41752,41755,41757,41759],{"class":469,"line":3140},[151,41751,129],{"class":503},[151,41753,41754],{"class":481},"'HTML'",[151,41756,106],{"class":503},[151,41758,41742],{"class":477},[151,41760,37985],{"class":503},[151,41762,41763,41765,41768,41770,41773,41775],{"class":469,"line":3149},[151,41764,129],{"class":503},[151,41766,41767],{"class":481},"'DOCKER'",[151,41769,106],{"class":503},[151,41771,41772],{"class":477},"66",[151,41774,24817],{"class":503},[151,41776,41777],{"class":1527},"# I am a big container fan! It is my preferred way to run software, locally and in the cloud\n",[151,41779,41780,41782,41785,41787,41790,41792],{"class":469,"line":3158},[151,41781,129],{"class":503},[151,41783,41784],{"class":481},"'KUBERNETES'",[151,41786,106],{"class":503},[151,41788,41789],{"class":477},"58",[151,41791,24817],{"class":503},[151,41793,41794],{"class":1527},"# I read the Manning book on k8s. I prefer ECS or Swarm, but I might try using it more\n",[151,41796,41797,41799,41802,41804,41806,41808],{"class":469,"line":3167},[151,41798,129],{"class":503},[151,41800,41801],{"class":481},"'GO'",[151,41803,106],{"class":503},[151,41805,41789],{"class":477},[151,41807,24817],{"class":503},[151,41809,41810],{"class":1527},"# I haven't ever used Go, but it doesn't look too bad coming from Python\n",[151,41812,41813,41815,41818,41820,41822],{"class":469,"line":3175},[151,41814,129],{"class":503},[151,41816,41817],{"class":481},"'REACT NATIVE'",[151,41819,106],{"class":503},[151,41821,41789],{"class":477},[151,41823,37985],{"class":503},[151,41825,41826,41828,41831,41833,41836,41838],{"class":469,"line":3184},[151,41827,129],{"class":503},[151,41829,41830],{"class":481},"'C++'",[151,41832,106],{"class":503},[151,41834,41835],{"class":477},"55",[151,41837,24817],{"class":503},[151,41839,41840],{"class":1527},"# Also haven't used this, but I have a book on it\n",[151,41842,41843,41845,41848,41850,41853,41855],{"class":469,"line":3193},[151,41844,129],{"class":503},[151,41846,41847],{"class":481},"'GRAPHQL'",[151,41849,106],{"class":503},[151,41851,41852],{"class":477},"48",[151,41854,24817],{"class":503},[151,41856,41857],{"class":1527},"# I tried GraphQL and built a HN clone in Django. I prefer REST but I get the appeal (for frontend developers)\n",[151,41859,41860,41862,41865,41867,41869,41871],{"class":469,"line":3720},[151,41861,129],{"class":503},[151,41863,41864],{"class":481},"'GOOGLE CLOUD'",[151,41866,106],{"class":503},[151,41868,6591],{"class":477},[151,41870,24817],{"class":503},[151,41872,41873],{"class":1527},"# I'm not a big GCP as I mostly use AWS and Digital Ocean but I would like to try Cloud Run\n",[151,41875,41876,41878,41881,41883,41886,41888],{"class":469,"line":3729},[151,41877,129],{"class":503},[151,41879,41880],{"class":481},"'RUBY'",[151,41882,106],{"class":503},[151,41884,41885],{"class":477},"44",[151,41887,24817],{"class":503},[151,41889,41890],{"class":1527},"# I haven't used it\n",[151,41892,41893,41895,41898,41900,41902,41904],{"class":469,"line":3735},[151,41894,129],{"class":503},[151,41896,41897],{"class":481},"'DJANGO'",[151,41899,106],{"class":503},[151,41901,41885],{"class":477},[151,41903,24817],{"class":503},[151,41905,41906],{"class":1527},"# Django is my go-to tool for building web apps and APIs. I love the admin, ORM and DRF\n",[151,41908,41909,41911,41914,41916,41918,41920],{"class":469,"line":3745},[151,41910,129],{"class":503},[151,41912,41913],{"class":481},"'MACHINE LEARNING'",[151,41915,106],{"class":503},[151,41917,41885],{"class":477},[151,41919,24817],{"class":503},[151,41921,41922],{"class":1527},"# I am familiar with some ML techniques but not very well practiced\n",[151,41924,41925,41927,41930,41932,41935,41937],{"class":469,"line":3754},[151,41926,129],{"class":503},[151,41928,41929],{"class":481},"'MONGODB'",[151,41931,106],{"class":503},[151,41933,41934],{"class":477},"43",[151,41936,24817],{"class":503},[151,41938,41939],{"class":1527},"# Have used it before, but I try to use the postgres JSONField for storing NoSQL data when possible\n",[151,41941,41942,41944,41947,41949,41951,41953],{"class":469,"line":3760},[151,41943,129],{"class":503},[151,41945,41946],{"class":481},"'IOS'",[151,41948,106],{"class":503},[151,41950,7907],{"class":477},[151,41952,24817],{"class":503},[151,41954,41955],{"class":1527},"# I have an iPhone, but haven't used a Mac in a long time, mostly on Linux and Windows\n",[151,41957,41958,41960,41963,41965,41968,41970],{"class":469,"line":3773},[151,41959,129],{"class":503},[151,41961,41962],{"class":481},"'MYSQL'",[151,41964,106],{"class":503},[151,41966,41967],{"class":477},"36",[151,41969,24817],{"class":503},[151,41971,41972],{"class":1527},"# I tend to use Postgres, I don't think I've ever used this\n",[151,41974,41975,41977,41980,41982,41985,41987],{"class":469,"line":3782},[151,41976,129],{"class":503},[151,41978,41979],{"class":481},"'ANDROID'",[151,41981,106],{"class":503},[151,41983,41984],{"class":477},"35",[151,41986,24817],{"class":503},[151,41988,41989],{"class":1527},"# Not something I have worked with\n",[151,41991,41992,41994,41997,41999,42001,42003],{"class":469,"line":3791},[151,41993,129],{"class":503},[151,41995,41996],{"class":481},"'DATA ANALYTICS'",[151,41998,106],{"class":503},[151,42000,9302],{"class":477},[151,42002,24817],{"class":503},[151,42004,42005],{"class":1527},"# I do a lot of this\n",[151,42007,42008,42010,42013,42015,42018,42020],{"class":469,"line":3803},[151,42009,129],{"class":503},[151,42011,42012],{"class":481},"'GIT'",[151,42014,106],{"class":503},[151,42016,42017],{"class":477},"30",[151,42019,24817],{"class":503},[151,42021,42022],{"class":1527},"# I'm very slowly trying to learn advanced git features. I have many abandoned \"rebase-practice\" repos\n",[151,42024,42025,42027,42030,42032,42034,42036],{"class":469,"line":3811},[151,42026,129],{"class":503},[151,42028,42029],{"class":481},"'ANGULAR'",[151,42031,106],{"class":503},[151,42033,7836],{"class":477},[151,42035,24817],{"class":503},[151,42037,42038],{"class":1527},"# I tried it once for about an hour, I know that people love to hate it, I'm just not sure why\n",[151,42040,42041,42043,42046,42048,42050,42052],{"class":469,"line":3820},[151,42042,129],{"class":503},[151,42044,42045],{"class":481},"'SWIFT'",[151,42047,106],{"class":503},[151,42049,7836],{"class":477},[151,42051,24817],{"class":503},[151,42053,42054],{"class":1527},"# This is apparently a very popular language and has use cases outside of mobile development, but I've never used it\n",[151,42056,42057,42059,42062,42064,42066,42068],{"class":469,"line":7084},[151,42058,129],{"class":503},[151,42060,42061],{"class":481},"'LINUX'",[151,42063,106],{"class":503},[151,42065,7702],{"class":477},[151,42067,24817],{"class":503},[151,42069,42070],{"class":1527},"# I spend lot of time using Linux machines, mostly Ubuntu.\n",[151,42072,42073,42075,42078,42080,42082,42084],{"class":469,"line":7148},[151,42074,129],{"class":503},[151,42076,42077],{"class":481},"'SOFTWARE ARCHITECTURE'",[151,42079,106],{"class":503},[151,42081,7728],{"class":477},[151,42083,24817],{"class":503},[151,42085,42086],{"class":1527},"# I like using diagrams.net to draw application infrastructure\n",[151,42088,42089,42091,42094,42096,42099,42101],{"class":469,"line":7211},[151,42090,129],{"class":503},[151,42092,42093],{"class":481},"'KOTLIN'",[151,42095,106],{"class":503},[151,42097,42098],{"class":477},"23",[151,42100,24817],{"class":503},[151,42102,42103],{"class":1527},"# I think this is a framework for Java/Android?\n",[151,42105,42106,42108,42111,42113,42116,42118],{"class":469,"line":7273},[151,42107,129],{"class":503},[151,42109,42110],{"class":481},"'TENSORFLOW'",[151,42112,106],{"class":503},[151,42114,42115],{"class":477},"22",[151,42117,24817],{"class":503},[151,42119,42120],{"class":1527},"# Google DL/ML library that I'll try to use later in this article\n",[151,42122,42123,42125,42128,42130,42132,42134],{"class":469,"line":7335},[151,42124,129],{"class":503},[151,42126,42127],{"class":481},"'DISTRIBUTED SYSTEMS'",[151,42129,106],{"class":503},[151,42131,42115],{"class":477},[151,42133,24817],{"class":503},[151,42135,42136],{"class":1527},"# Using AWS, I guess I have technically designed distributed systems but I wouldn't call it one of my skills\n",[151,42138,42139,42141,42144,42146,42148,42150],{"class":469,"line":7398},[151,42140,129],{"class":503},[151,42142,42143],{"class":481},"'PHP'",[151,42145,106],{"class":503},[151,42147,42115],{"class":477},[151,42149,24817],{"class":503},[151,42151,42152],{"class":1527},"# I almost learned it to support a WordPress site but opted to use JAMStack instead\n",[151,42154,42155,42157,42160,42162,42164,42166],{"class":469,"line":7462},[151,42156,129],{"class":503},[151,42158,42159],{"class":481},"'DATA WAREHOUSING'",[151,42161,106],{"class":503},[151,42163,42115],{"class":477},[151,42165,24817],{"class":503},[151,42167,42168],{"class":1527},"# I have used Google BigQuery before which I think counts for this skill\n",[151,42170,42171,42173,42176,42178,42180,42182],{"class":469,"line":7467},[151,42172,129],{"class":503},[151,42174,42175],{"class":481},"'DEEP LEARNING'",[151,42177,106],{"class":503},[151,42179,9097],{"class":477},[151,42181,24817],{"class":503},[151,42183,42184],{"class":1527},"# I'm going to try to use this later in this article\n",[151,42186,42187,42189,42192,42194,42196,42198],{"class":469,"line":7532},[151,42188,129],{"class":503},[151,42190,42191],{"class":481},"'DATA MODELING'",[151,42193,106],{"class":503},[151,42195,9097],{"class":477},[151,42197,24817],{"class":503},[151,42199,42200],{"class":1527},"# To me this means writing Django models, or thinking about how to structure an API/json data, etc.\n",[151,42202,42203,42205,42208,42210,42213,42215],{"class":469,"line":7537},[151,42204,129],{"class":503},[151,42206,42207],{"class":481},"'C#'",[151,42209,106],{"class":503},[151,42211,42212],{"class":477},"19",[151,42214,24817],{"class":503},[151,42216,42217],{"class":1527},"# Micrsoft language used for different things including game dev with Unity\n",[151,42219,42220,42222,42225,42227,42229,42231],{"class":469,"line":7603},[151,42221,129],{"class":503},[151,42223,42224],{"class":481},"'FLASK'",[151,42226,106],{"class":503},[151,42228,42212],{"class":477},[151,42230,24817],{"class":503},[151,42232,42233],{"class":1527},"# I'm familiar with it but mostly prefer Django's batteries included philosophy\n",[151,42235,42236,42238,42241,42243,42245,42247],{"class":469,"line":7608},[151,42237,129],{"class":503},[151,42239,42240],{"class":481},"'C'",[151,42242,106],{"class":503},[151,42244,42212],{"class":477},[151,42246,24817],{"class":503},[151,42248,42249],{"class":1527},"# In learning about Linux I have read a bit of C, but never written any\n",[151,42251,42252,42254,42257,42259,42261,42263],{"class":469,"line":7673},[151,42253,129],{"class":503},[151,42255,42256],{"class":481},"'REDIS'",[151,42258,106],{"class":503},[151,42260,7696],{"class":477},[151,42262,24817],{"class":503},[151,42264,42265],{"class":1527},"# Fast, in memory key-value store with multiple data types. I use it for a few different things\n",[151,42267,42268,42270,42273,42275,42277,42279],{"class":469,"line":7678},[151,42269,129],{"class":503},[151,42271,42272],{"class":481},"'MICROSERVICES'",[151,42274,106],{"class":503},[151,42276,7696],{"class":477},[151,42278,24817],{"class":503},[151,42280,42281],{"class":1527},"#\n",[151,42283,42284,42286,42289,42291,42294,42296],{"class":469,"line":7708},[151,42285,129],{"class":503},[151,42287,42288],{"class":481},"'COMPUTER VISION'",[151,42290,106],{"class":503},[151,42292,42293],{"class":477},"17",[151,42295,24817],{"class":503},[151,42297,42298],{"class":1527},"# I once used OpenCV on my Raspberry Pi\n",[151,42300,42301,42303,42306,42308,42311,42313],{"class":469,"line":7713},[151,42302,129],{"class":503},[151,42304,42305],{"class":481},"'EXPRESS'",[151,42307,106],{"class":503},[151,42309,42310],{"class":477},"15",[151,42312,24817],{"class":503},[151,42314,42315],{"class":1527},"# I'd like to learn how to use this in 2021\n",[151,42317,42318,42320,42323,42325,42328,42330],{"class":469,"line":7746},[151,42319,129],{"class":503},[151,42321,42322],{"class":481},"'BASH/SHELL'",[151,42324,106],{"class":503},[151,42326,42327],{"class":477},"13",[151,42329,24817],{"class":503},[151,42331,42332],{"class":1527},"# I'm not very fluent in bash but\n",[151,42334,42335,42337,42340,42342,42344,42346],{"class":469,"line":7751},[151,42336,129],{"class":503},[151,42338,42339],{"class":481},"'OBJECTIVE-C'",[151,42341,106],{"class":503},[151,42343,42327],{"class":477},[151,42345,24817],{"class":503},[151,42347,42348],{"class":1527},"# I haven't used this but I know it is popular for iOS development\n",[151,42350,42351,42353,42356,42358,42361,42363],{"class":469,"line":7816},[151,42352,129],{"class":503},[151,42354,42355],{"class":481},"'FIREBASE'",[151,42357,106],{"class":503},[151,42359,42360],{"class":477},"12",[151,42362,24817],{"class":503},[151,42364,42365],{"class":1527},"# I have played around with Firebase, but I haven't built anything with it\n",[151,42367,42368,42370,42373,42375,42378,42380],{"class":469,"line":7821},[151,42369,129],{"class":503},[151,42371,42372],{"class":481},"'SCALA'",[151,42374,106],{"class":503},[151,42376,42377],{"class":477},"11",[151,42379,24817],{"class":503},[151,42381,42382],{"class":1527},"# Functional programming language for JVM, I haven't used it\n",[151,42384,42385,42387,42390,42392,42394],{"class":469,"line":7847},[151,42386,129],{"class":503},[151,42388,42389],{"class":481},"'SOFTWARE SECURITY'",[151,42391,106],{"class":503},[151,42393,42377],{"class":477},[151,42395,37985],{"class":503},[151,42397,42398,42400,42403,42405,42407,42409],{"class":469,"line":7852},[151,42399,129],{"class":503},[151,42401,42402],{"class":481},"'UNITY'",[151,42404,106],{"class":503},[151,42406,42377],{"class":477},[151,42408,24817],{"class":503},[151,42410,42411],{"class":1527},"# I used this once before to play around with VR development for HTC Vive\n",[151,42413,42414,42416,42419,42421,42423,42425],{"class":469,"line":7887},[151,42415,129],{"class":503},[151,42417,42418],{"class":481},"'R'",[151,42420,106],{"class":503},[151,42422,42377],{"class":477},[151,42424,24817],{"class":503},[151,42426,42427],{"class":1527},"# I haven't used R before, and I would probably reach for a Python library for doing statistics or ML-related things\n",[151,42429,42430,42432,42435,42437,42439,42441],{"class":469,"line":7892},[151,42431,129],{"class":503},[151,42433,42434],{"class":481},"'KAFKA'",[151,42436,106],{"class":503},[151,42438,12423],{"class":477},[151,42440,24817],{"class":503},[151,42442,42443],{"class":1527},"# I haven't used it but I'm familiar with the ideas behind Kafka.\n",[151,42445,42446,42448,42451,42453,42455,42457],{"class":469,"line":7924},[151,42447,129],{"class":503},[151,42449,42450],{"class":481},"'SPARK'",[151,42452,106],{"class":503},[151,42454,12423],{"class":477},[151,42456,24817],{"class":503},[151,42458,42459],{"class":1527},"# I haven't used it and I'm not really sure what it is\n",[151,42461,42462,42464,42467,42469,42471,42473],{"class":469,"line":7929},[151,42463,129],{"class":503},[151,42465,42466],{"class":481},"'ELASTICSEARCH'",[151,42468,106],{"class":503},[151,42470,12423],{"class":477},[151,42472,24817],{"class":503},[151,42474,42475],{"class":1527},"# I haven't used it before\n",[151,42477,42478,42480,42483,42485,42487,42489],{"class":469,"line":7991},[151,42479,129],{"class":503},[151,42481,42482],{"class":481},"'ETL'",[151,42484,106],{"class":503},[151,42486,12423],{"class":477},[151,42488,24817],{"class":503},[151,42490,42491],{"class":1527},"# Extract, Transform and Load\n",[151,42493,42494,42496,42499,42501,42503],{"class":469,"line":7996},[151,42495,129],{"class":503},[151,42497,42498],{"class":481},"'NATURAL LANGUAGE PROCESSING'",[151,42500,106],{"class":503},[151,42502,12423],{"class":477},[151,42504,37985],{"class":503},[151,42506,42507,42509,42512,42514,42516,42518],{"class":469,"line":8078},[151,42508,129],{"class":503},[151,42510,42511],{"class":481},"'HEROKU'",[151,42513,106],{"class":503},[151,42515,12423],{"class":477},[151,42517,24817],{"class":503},[151,42519,42520],{"class":1527},"# I used this when first learning about Django, haven't used it in a while\n",[151,42522,42523,42525,42528,42530,42532,42534],{"class":469,"line":8140},[151,42524,129],{"class":503},[151,42526,42527],{"class":481},"'NGINX'",[151,42529,106],{"class":503},[151,42531,7918],{"class":477},[151,42533,24817],{"class":503},[151,42535,42536],{"class":1527},"# I use NGINX in most of my web apps as a reverse proxy\n",[151,42538,42539,42541,42544,42546,42548,42550],{"class":469,"line":8145},[151,42540,129],{"class":503},[151,42542,42543],{"class":481},"'JENKINS'",[151,42545,106],{"class":503},[151,42547,7918],{"class":477},[151,42549,24817],{"class":503},[151,42551,42552],{"class":1527},"# I haven't used it, I am a big GitLab fan and will use that whenever possible\n",[151,42554,42555,42557,42560,42562,42564,42566],{"class":469,"line":8259},[151,42556,129],{"class":503},[151,42558,42559],{"class":481},"'RUST'",[151,42561,106],{"class":503},[151,42563,7918],{"class":477},[151,42565,24817],{"class":503},[151,42567,42568],{"class":1527},"# I have read the Rust book and have played around with WASM\n",[151,42570,42571,42573,42576,42578,42580],{"class":469,"line":8264},[151,42572,129],{"class":503},[151,42574,42575],{"class":481},"'IMAGE PROCESSING'",[151,42577,106],{"class":503},[151,42579,24369],{"class":477},[151,42581,37985],{"class":503},[151,42583,42584,42586,42589,42591,42593,42595],{"class":469,"line":8613},[151,42585,129],{"class":503},[151,42587,42588],{"class":481},"'SERVERLESS'",[151,42590,106],{"class":503},[151,42592,24369],{"class":477},[151,42594,24817],{"class":503},[151,42596,42597],{"class":1527},"# I currenty use Fargate and have also used Lambda and SQS and some other serverless AWS tools\n",[151,42599,42600,42602,42605,42607,42609],{"class":469,"line":8678},[151,42601,129],{"class":503},[151,42603,42604],{"class":481},"'BLOCKCHAIN'",[151,42606,106],{"class":503},[151,42608,24369],{"class":477},[151,42610,37985],{"class":503},[151,42612,42613,42615,42618,42620,42622],{"class":469,"line":8742},[151,42614,129],{"class":503},[151,42616,42617],{"class":481},"'OPENCV'",[151,42619,106],{"class":503},[151,42621,24369],{"class":477},[151,42623,37985],{"class":503},[151,42625,42626,42628,42631,42633,42635,42637],{"class":469,"line":8806},[151,42627,129],{"class":503},[151,42629,42630],{"class":481},"'CAD DESIGN'",[151,42632,106],{"class":503},[151,42634,25043],{"class":477},[151,42636,24817],{"class":503},[151,42638,42639],{"class":1527},"# I am a big fan of SketchUp and I'm familiar with Blender as well, but I'm not sure if these qualify as CAD design\n",[151,42641,42642,42644,42647,42649,42651,42653],{"class":469,"line":8870},[151,42643,129],{"class":503},[151,42645,42646],{"class":481},"'JQUERY'",[151,42648,106],{"class":503},[151,42650,25043],{"class":477},[151,42652,24817],{"class":503},[151,42654,42655],{"class":1527},"# It was the first library I worked with when I started learning javascript\n",[151,42657,42658,42660,42663,42665,42667,42669],{"class":469,"line":8875},[151,42659,129],{"class":503},[151,42661,42662],{"class":481},"'HADOOP'",[151,42664,106],{"class":503},[151,42666,25038],{"class":477},[151,42668,24817],{"class":503},[151,42670,42671],{"class":1527},"# It is related to map-reduce, I haven't used it before and I don't really know what it is. I know it is related to HDFS\n",[151,42673,42674,42676,42679,42681,42683,42685],{"class":469,"line":8881},[151,42675,129],{"class":503},[151,42677,42678],{"class":481},"'.NET CORE'",[151,42680,106],{"class":503},[151,42682,25038],{"class":477},[151,42684,24817],{"class":503},[151,42686,41890],{"class":1527},[151,42688,42689,42691,42694,42696,42698,42700],{"class":469,"line":8886},[151,42690,129],{"class":503},[151,42692,42693],{"class":481},"'TCP/IP'",[151,42695,106],{"class":503},[151,42697,25038],{"class":477},[151,42699,24817],{"class":503},[151,42701,42702],{"class":1527},"# I'm familiary with the basics\n",[151,42704,42705,42707,42710,42712,42714,42716],{"class":469,"line":8892},[151,42706,129],{"class":503},[151,42708,42709],{"class":481},"'ELIXIR'",[151,42711,106],{"class":503},[151,42713,25038],{"class":477},[151,42715,24817],{"class":503},[151,42717,42718],{"class":1527},"# I don't know, I think it is a framework for Erlang\n",[151,42720,42721,42723,42726,42728,42730],{"class":469,"line":8963},[151,42722,129],{"class":503},[151,42724,42725],{"class":481},"'INTERNET OF THINGS (IOT)'",[151,42727,106],{"class":503},[151,42729,24380],{"class":477},[151,42731,37985],{"class":503},[151,42733,42734,42736,42739,42741,42743,42745],{"class":469,"line":8969},[151,42735,129],{"class":503},[151,42737,42738],{"class":481},"'SASS'",[151,42740,106],{"class":503},[151,42742,24380],{"class":477},[151,42744,24817],{"class":503},[151,42746,42747],{"class":1527},"# I think it is a framework for CSS. I frequently see node-sass errors from npm\n",[151,42749,42750,42752,42755,42757,42759],{"class":469,"line":15001},[151,42751,129],{"class":503},[151,42753,42754],{"class":481},"'OPENGL'",[151,42756,106],{"class":503},[151,42758,24380],{"class":477},[151,42760,37985],{"class":503},[151,42762,42763,42765,42768,42770,42772,42774],{"class":469,"line":15009},[151,42764,129],{"class":503},[151,42766,42767],{"class":481},"'DYNAMODB'",[151,42769,106],{"class":503},[151,42771,24380],{"class":477},[151,42773,24817],{"class":503},[151,42775,42776],{"class":1527},"# I am familiar with it but haven't used it\n",[151,42778,42779,42781,42784,42786,42788,42790],{"class":469,"line":15019},[151,42780,129],{"class":503},[151,42782,42783],{"class":481},"'GOOGLE APP ENGINE'",[151,42785,106],{"class":503},[151,42787,24380],{"class":477},[151,42789,24817],{"class":503},[151,42791,42475],{"class":1527},[151,42793,42794,42796,42799,42801,42803],{"class":469,"line":15027},[151,42795,129],{"class":503},[151,42797,42798],{"class":481},"'UNIX'",[151,42800,106],{"class":503},[151,42802,9187],{"class":477},[151,42804,37985],{"class":503},[151,42806,42807,42809,42812,42814,42816,42818],{"class":469,"line":15037},[151,42808,129],{"class":503},[151,42810,42811],{"class":481},"'SPRING FRAMEWORK'",[151,42813,106],{"class":503},[151,42815,9187],{"class":477},[151,42817,24817],{"class":503},[151,42819,42820],{"class":1527},"# A Java web framework that I haven't used\n",[151,42822,42823,42825,42828,42830,42832,42834],{"class":469,"line":15045},[151,42824,129],{"class":503},[151,42826,42827],{"class":481},"'CUDA'",[151,42829,106],{"class":503},[151,42831,9187],{"class":477},[151,42833,24817],{"class":503},[151,42835,42836],{"class":1527},"# I have used it indirectly when I used nvidia-docker to used Tensorflow\n",[151,42838,42839,42841,42844,42846,42848,42850],{"class":469,"line":15055},[151,42840,129],{"class":503},[151,42842,42843],{"class":481},"'DART'",[151,42845,106],{"class":503},[151,42847,9187],{"class":477},[151,42849,24817],{"class":503},[151,42851,42852],{"class":1527},"# I don't know what this is\n",[151,42854,42855,42857,42860,42862,42864,42866],{"class":469,"line":15060},[151,42856,129],{"class":503},[151,42858,42859],{"class":481},"'ERLANG'",[151,42861,106],{"class":503},[151,42863,9187],{"class":477},[151,42865,24817],{"class":503},[151,42867,42868],{"class":1527},"# A language that handles concurrency very well\n",[151,42870,42871,42873,42876,42878,42880,42882],{"class":469,"line":15068},[151,42872,129],{"class":503},[151,42874,42875],{"class":481},"'RABBITMQ'",[151,42877,106],{"class":503},[151,42879,9187],{"class":477},[151,42881,24817],{"class":503},[151,42883,42884],{"class":1527},"# Message queue, I have used it before but tend to use Redis as a message broker\n",[151,42886,42887,42889,42892,42894,42896,42898],{"class":469,"line":15076},[151,42888,129],{"class":503},[151,42890,42891],{"class":481},"'KERAS'",[151,42893,106],{"class":503},[151,42895,9187],{"class":477},[151,42897,24817],{"class":503},[151,42899,42900],{"class":1527},"# A helper/wrapper library for Tensorflow\n",[151,42902,42903,42905,42908,42910,42912,42914],{"class":469,"line":15085},[151,42904,129],{"class":503},[151,42906,42907],{"class":481},"'SCSS'",[151,42909,106],{"class":503},[151,42911,9187],{"class":477},[151,42913,24817],{"class":503},[151,42915,42916],{"class":1527},"# I think this is a language that compiles to CSS. CSS frameworks that I use use this\n",[151,42918,42919,42921,42924,42926,42928,42930],{"class":469,"line":15095},[151,42920,129],{"class":503},[151,42922,42923],{"class":481},"'ML'",[151,42925,106],{"class":503},[151,42927,9187],{"class":477},[151,42929,24817],{"class":503},[151,42931,42932],{"class":1527},"# I've read some books and experimented but I'm not a regular practitioner\n",[151,42934,42935,42937,42940,42942,42944,42946],{"class":469,"line":15105},[151,42936,129],{"class":503},[151,42938,42939],{"class":481},"'MATLAB'",[151,42941,106],{"class":503},[151,42943,9187],{"class":477},[151,42945,24817],{"class":503},[151,42947,42948],{"class":1527},"# A tool her programming with higher math\n",[151,42950,42951,42953,42956,42958,42960,42962],{"class":469,"line":15110},[151,42952,129],{"class":503},[151,42954,42955],{"class":481},"'SPRING'",[151,42957,106],{"class":503},[151,42959,9187],{"class":477},[151,42961,24817],{"class":503},[151,42963,42964],{"class":1527},"# A Java web framework. Not sure if different from Spring Boot. Never used it.\n",[151,42966,42967,42969,42972,42974,42976,42978],{"class":469,"line":15118},[151,42968,129],{"class":503},[151,42970,42971],{"class":481},"'CASSANDRA'",[151,42973,106],{"class":503},[151,42975,9187],{"class":477},[151,42977,24817],{"class":503},[151,42979,42980],{"class":1527},"# FB scalable database (NoSQL I think?) that does sharding really well\n",[151,42982,42983,42985,42988,42990,42992,42994],{"class":469,"line":15128},[151,42984,129],{"class":503},[151,42986,42987],{"class":481},"'HIVE'",[151,42989,106],{"class":503},[151,42991,6557],{"class":477},[151,42993,24817],{"class":503},[151,42995,42996],{"class":1527},"# Not sure what hive is. I think it related to Hadoop\n",[151,42998,42999,43001,43004,43006,43008,43010],{"class":469,"line":15139},[151,43000,129],{"class":503},[151,43002,43003],{"class":481},"'PUPPET'",[151,43005,106],{"class":503},[151,43007,6557],{"class":477},[151,43009,24817],{"class":503},[151,43011,43012],{"class":1527},"# Configuration management tool that I haven't ever used\n",[151,43014,43015,43017,43020,43022,43024,43026],{"class":469,"line":31954},[151,43016,129],{"class":503},[151,43018,43019],{"class":481},"'REDSHIFT'",[151,43021,106],{"class":503},[151,43023,6557],{"class":477},[151,43025,24817],{"class":503},[151,43027,43028],{"class":1527},"# AWS version of Google BigQuery\n",[151,43030,43031,43033,43036,43038,43040,43042],{"class":469,"line":31960},[151,43032,129],{"class":503},[151,43034,43035],{"class":481},"'SQL SERVER'",[151,43037,106],{"class":503},[151,43039,6557],{"class":477},[151,43041,24817],{"class":503},[151,43043,43044],{"class":1527},"# Not sure what this refers to, specifically.\n",[151,43046,43047,43049,43052,43054,43056,43058],{"class":469,"line":31965},[151,43048,129],{"class":503},[151,43050,43051],{"class":481},"'GROOVY'",[151,43053,106],{"class":503},[151,43055,6557],{"class":477},[151,43057,24817],{"class":503},[151,43059,43060],{"class":1527},"# I think this is a Java framework?\n",[151,43062,43063,43065,43068,43070,43072,43074],{"class":469,"line":31971},[151,43064,129],{"class":503},[151,43066,43067],{"class":481},"'VERILOG'",[151,43069,106],{"class":503},[151,43071,6557],{"class":477},[151,43073,24817],{"class":503},[151,43075,43076],{"class":1527},"# I've never heard of this.\n",[151,43078,43079,43081,43084,43086,43088,43090],{"class":469,"line":31983},[151,43080,129],{"class":503},[151,43082,43083],{"class":481},"'TORCH/PYTORCH'",[151,43085,106],{"class":503},[151,43087,6557],{"class":477},[151,43089,24817],{"class":503},[151,43091,43092],{"class":1527},"# FB python deep learning library.\n",[151,43094,43095,43097,43100,43102,43104,43106],{"class":469,"line":31994},[151,43096,129],{"class":503},[151,43098,43099],{"class":481},"'CLOJURE'",[151,43101,106],{"class":503},[151,43103,6557],{"class":477},[151,43105,24817],{"class":503},[151,43107,43108],{"class":1527},"# A LISP derivate, functional language\n",[151,43110,43111,43113,43116,43118,43120,43122],{"class":469,"line":32007},[151,43112,129],{"class":503},[151,43114,43115],{"class":481},"'MICROSOFT AZURE'",[151,43117,106],{"class":503},[151,43119,6557],{"class":477},[151,43121,24817],{"class":503},[151,43123,43124],{"class":1527},"# I've used Azure AD and thats it.\n",[151,43126,43127,43129,43132,43134,43136,43138],{"class":469,"line":32018},[151,43128,129],{"class":503},[151,43130,43131],{"class":481},"'HBASE'",[151,43133,106],{"class":503},[151,43135,6619],{"class":477},[151,43137,24817],{"class":503},[151,43139,42852],{"class":1527},[151,43141,43142,43144,43147,43149,43151,43153],{"class":469,"line":32026},[151,43143,129],{"class":503},[151,43145,43146],{"class":481},"'RDS/AURORA'",[151,43148,106],{"class":503},[151,43150,6619],{"class":477},[151,43152,24817],{"class":503},[151,43154,43155],{"class":1527},"# I use RDS and experimented with Aurora but don't know when/why to use it\n",[151,43157,43158,43160,43163,43165,43167,43169],{"class":469,"line":32031},[151,43159,129],{"class":503},[151,43161,43162],{"class":481},"'FIRMWARE'",[151,43164,106],{"class":503},[151,43166,6619],{"class":477},[151,43168,24817],{"class":503},[151,43170,43171],{"class":1527},"# What's between hardware and software\n",[151,43173,43174,43176,43179,43181,43183,43185],{"class":469,"line":32036},[151,43175,129],{"class":503},[151,43177,43178],{"class":481},"'ABAP'",[151,43180,106],{"class":503},[151,43182,6619],{"class":477},[151,43184,24817],{"class":503},[151,43186,43187],{"class":1527},"# I don't know\n",[151,43189,43190,43192,43195,43197,43199,43201],{"class":469,"line":32042},[151,43191,129],{"class":503},[151,43193,43194],{"class":481},"'ARDUINO'",[151,43196,106],{"class":503},[151,43198,6619],{"class":477},[151,43200,24817],{"class":503},[151,43202,43203],{"class":1527},"# I have one, but don't use it\n",[151,43205,43206,43208,43211,43213,43215,43217],{"class":469,"line":32054},[151,43207,129],{"class":503},[151,43209,43210],{"class":481},"'MICROCONTROLLERS'",[151,43212,106],{"class":503},[151,43214,6619],{"class":477},[151,43216,24817],{"class":503},[151,43218,43219],{"class":1527},"# Arduino might be an example of what this is\n",[151,43221,43222,43224,43227,43229,43231,43233],{"class":469,"line":32067},[151,43223,129],{"class":503},[151,43225,43226],{"class":481},"'SOLIDITY'",[151,43228,106],{"class":503},[151,43230,6619],{"class":477},[151,43232,24817],{"class":503},[151,43234,43235],{"class":1527},"# Don't know what this is\n",[151,43237,43238,43240,43243,43245,43247,43249],{"class":469,"line":32086},[151,43239,129],{"class":503},[151,43241,43242],{"class":481},"'UNREAL ENGINE'",[151,43244,106],{"class":503},[151,43246,6619],{"class":477},[151,43248,24817],{"class":503},[151,43250,43251],{"class":1527},"# Unity competitor, used for game development\n",[151,43253,43254,43256,43259,43261,43263,43265],{"class":469,"line":32097},[151,43255,129],{"class":503},[151,43257,43258],{"class":481},"'COFFEESCRIPT'",[151,43260,106],{"class":503},[151,43262,6619],{"class":477},[151,43264,24817],{"class":503},[151,43266,43267],{"class":1527},"# I think it is a dialect of JS, but I'm not sure\n",[151,43269,43270,43272,43275,43277,43279,43281],{"class":469,"line":25585},[151,43271,129],{"class":503},[151,43273,43274],{"class":481},"'LUA'",[151,43276,106],{"class":503},[151,43278,6619],{"class":477},[151,43280,24817],{"class":503},[151,43282,43283],{"class":1527},"# I think this is what redis is written in, but I'm not sure how to descbribe what it is\n",[151,43285,43286,43288,43291,43293,43295,43297],{"class":469,"line":32112},[151,43287,129],{"class":503},[151,43289,43290],{"class":481},"'MACOS'",[151,43292,106],{"class":503},[151,43294,6619],{"class":477},[151,43296,24817],{"class":503},[151,43298,43299],{"class":1527},"# I haven't used MacOS in a long time. I'm tempted to try M1, but I also want to buid a new PC...\n",[151,43301,43302,43304,43307,43309,43311,43313],{"class":469,"line":32117},[151,43303,129],{"class":503},[151,43305,43306],{"class":481},"'NEO4J'",[151,43308,106],{"class":503},[151,43310,6619],{"class":477},[151,43312,24817],{"class":503},[151,43314,43315],{"class":1527},"# A graph database, I'm not sure what a typical use case is for this\n",[151,43317,43318,43320,43323,43325,43327,43329],{"class":469,"line":32123},[151,43319,129],{"class":503},[151,43321,43322],{"class":481},"'INFORMATION SECURITY'",[151,43324,106],{"class":503},[151,43326,6619],{"class":477},[151,43328,24817],{"class":503},[151,43330,43331],{"class":1527},"# Unknown unknowns\n",[151,43333,43334,43336,43339,43341,43343,43345],{"class":469,"line":32151},[151,43335,129],{"class":503},[151,43337,43338],{"class":481},"'REINFORCEMENT LEARNING (RL)'",[151,43340,106],{"class":503},[151,43342,6619],{"class":477},[151,43344,24817],{"class":503},[151,43346,43347],{"class":1527},"# Not sure what this refers to specifically\n",[151,43349,43350,43352,43355,43357,43359,43361],{"class":469,"line":32156},[151,43351,129],{"class":503},[151,43353,43354],{"class":481},"'DEVICE DRIVERS'",[151,43356,106],{"class":503},[151,43358,6619],{"class":477},[151,43360,24817],{"class":503},[151,43362,43363],{"class":1527},"# Probably involves writing kernel modules\n",[151,43365,43366,43368,43371,43373,43375,43377],{"class":469,"line":32162},[151,43367,129],{"class":503},[151,43369,43370],{"class":481},"'EMBEDDED LINUX'",[151,43372,106],{"class":503},[151,43374,6619],{"class":477},[151,43376,24817],{"class":503},[151,43378,43379],{"class":1527},"# Not sure if Raspberry Pi is an example of this or not\n",[151,43381,43382,43384,43387,43389,43391,43393],{"class":469,"line":32168},[151,43383,129],{"class":503},[151,43385,43386],{"class":481},"'ELASTIC STACK (ELK)'",[151,43388,106],{"class":503},[151,43390,6760],{"class":477},[151,43392,24817],{"class":503},[151,43394,43395],{"class":1527},"# Useful for viewing log data (Elastic, Logstash and Kibana), haven't used it\n",[151,43397,43398,43400,43403,43405,43407,43409],{"class":469,"line":32180},[151,43399,129],{"class":503},[151,43401,43402],{"class":481},"'IIS'",[151,43404,106],{"class":503},[151,43406,6760],{"class":477},[151,43408,24817],{"class":503},[151,43410,43187],{"class":1527},[151,43412,43413,43415,43418,43420,43422,43424],{"class":469,"line":32192},[151,43414,129],{"class":503},[151,43416,43417],{"class":481},"'ORACLE'",[151,43419,106],{"class":503},[151,43421,6760],{"class":477},[151,43423,24817],{"class":503},[151,43425,43426],{"class":1527},"# A big software company and a proprietary database (I think Django supports it)\n",[151,43428,43429,43431,43434,43436,43438,43440],{"class":469,"line":32207},[151,43430,129],{"class":503},[151,43432,43433],{"class":481},"'F#'",[151,43435,106],{"class":503},[151,43437,6760],{"class":477},[151,43439,24817],{"class":503},[151,43441,43442],{"class":1527},"# A programming language that I don't know anything about\n",[151,43444,43445,43447,43450,43452,43454,43456],{"class":469,"line":32217},[151,43446,129],{"class":503},[151,43448,43449],{"class":481},"'SQLITE'",[151,43451,106],{"class":503},[151,43453,6760],{"class":477},[151,43455,24817],{"class":503},[151,43457,43458],{"class":1527},"# A light-weight SQL-compatible file-based database\n",[151,43460,43461,43463,43466,43468,43470,43472],{"class":469,"line":32226},[151,43462,129],{"class":503},[151,43464,43465],{"class":481},"'HASKELL'",[151,43467,106],{"class":503},[151,43469,6760],{"class":477},[151,43471,24817],{"class":503},[151,43473,43474],{"class":1527},"# A functional programming language that I don't know\n",[151,43476,43477,43479,43482,43484,43486,43488],{"class":469,"line":32231},[151,43478,129],{"class":503},[151,43480,43481],{"class":481},"'SCHEME'",[151,43483,106],{"class":503},[151,43485,6760],{"class":477},[151,43487,24817],{"class":503},[151,43489,43490],{"class":1527},"# I'm not sure, it may be something related to LISP\n",[151,43492,43493,43495,43498,43500,43502,43504],{"class":469,"line":32236},[151,43494,129],{"class":503},[151,43496,43497],{"class":481},"'MS SQL'",[151,43499,106],{"class":503},[151,43501,6760],{"class":477},[151,43503,24817],{"class":503},[151,43505,43506],{"class":1527},"# Never used this\n",[151,43508,43509,43511,43514,43516,43518,43520],{"class":469,"line":32244},[151,43510,129],{"class":503},[151,43512,43513],{"class":481},"'MARIADB'",[151,43515,106],{"class":503},[151,43517,6760],{"class":477},[151,43519,24817],{"class":503},[151,43521,43522],{"class":1527},"# An open source SQL database\n",[151,43524,43525,43527,43530,43532,43534,43536],{"class":469,"line":32249},[151,43526,129],{"class":503},[151,43528,43529],{"class":481},"'MAVEN'",[151,43531,106],{"class":503},[151,43533,6760],{"class":477},[151,43535,24817],{"class":503},[151,43537,43538],{"class":1527},"# I think it is a Java Framework\n",[151,43540,43541,43543,43546,43548,43550,43552],{"class":469,"line":32255},[151,43542,129],{"class":503},[151,43544,43545],{"class":481},"'SEARCH'",[151,43547,106],{"class":503},[151,43549,6760],{"class":477},[151,43551,24817],{"class":503},[151,43553,43554],{"class":1527},"# I don't know what this refers to\n",[151,43556,43557,43559,43562,43564,43566,43568],{"class":469,"line":32272},[151,43558,129],{"class":503},[151,43560,43561],{"class":481},"'OCAML'",[151,43563,106],{"class":503},[151,43565,6760],{"class":477},[151,43567,24817],{"class":503},[151,43569,43570],{"class":1527},"# I think I once read that high-frequency traders like to use this language, but I'm not sure why\n",[151,43572,43573,43575,43578,43580,43582,43584],{"class":469,"line":32277},[151,43574,129],{"class":503},[151,43576,43577],{"class":481},"'JULIA'",[151,43579,106],{"class":503},[151,43581,6760],{"class":477},[151,43583,24817],{"class":503},[151,43585,43586],{"class":1527},"# A programming language used for math and statistics\n",[151,43588,43589,43591,43594,43596,43598,43600],{"class":469,"line":32283},[151,43590,129],{"class":503},[151,43592,43593],{"class":481},"'GPU PROGRAMMING'",[151,43595,106],{"class":503},[151,43597,6760],{"class":477},[151,43599,24817],{"class":503},[151,43601,43602],{"class":1527},"# I haven't done this before, probably uses C++\n",[151,43604,43605,43607,43610,43612,43614,43616],{"class":469,"line":32295},[151,43606,129],{"class":503},[151,43608,43609],{"class":481},"'HACK'",[151,43611,106],{"class":503},[151,43613,6760],{"class":477},[151,43615,24817],{"class":503},[151,43617,43618],{"class":1527},"# FB's version of PHP\n",[151,43620,43621,43623,43626,43628,43630,43632],{"class":469,"line":32307},[151,43622,129],{"class":503},[151,43624,43625],{"class":481},"'XAMARIN'",[151,43627,106],{"class":503},[151,43629,6760],{"class":477},[151,43631,24817],{"class":503},[151,43633,43634],{"class":1527},"# I dont't know what this is\n",[151,43636,43637,43639,43642,43644,43646,43648],{"class":469,"line":32320},[151,43638,129],{"class":503},[151,43640,43641],{"class":481},"'CORDOVA'",[151,43643,106],{"class":503},[151,43645,6760],{"class":477},[151,43647,24817],{"class":503},[151,43649,43650],{"class":1527},"# I think it is a tool for generating native apps from JS\n",[151,43652,43653,43655,43658,43660,43662,43664],{"class":469,"line":32330},[151,43654,129],{"class":503},[151,43656,43657],{"class":481},"'SAS'",[151,43659,106],{"class":503},[151,43661,6760],{"class":477},[151,43663,24817],{"class":503},[151,43665,42852],{"class":1527},[151,43667,43668,43670,43673,43675,43677,43679],{"class":469,"line":32352},[151,43669,129],{"class":503},[151,43671,43672],{"class":481},"'ASSEMBLY'",[151,43674,106],{"class":503},[151,43676,6760],{"class":477},[151,43678,24817],{"class":503},[151,43680,43681],{"class":1527},"# Low level language that gives instructions to CPU\n",[151,43683,43684,43686,43689,43691,43693,43695],{"class":469,"line":32366},[151,43685,129],{"class":503},[151,43687,43688],{"class":481},"'XML'",[151,43690,106],{"class":503},[151,43692,6760],{"class":477},[151,43694,24817],{"class":503},[151,43696,43697],{"class":1527},"# A data format, I use it for this site's sitemap and RSS feed\n",[151,43699,43700,43702,43705,43707,43709,43711],{"class":469,"line":32371},[151,43701,129],{"class":503},[151,43703,43704],{"class":481},"'MEMCACHED'",[151,43706,106],{"class":503},[151,43708,6760],{"class":477},[151,43710,24817],{"class":503},[151,43712,43713],{"class":1527},"# Used for caching. I haven't used it; I typically use redis where this might be an option\n",[151,43715,43716,43718,43721,43723,43725,43727],{"class":469,"line":32376},[151,43717,129],{"class":503},[151,43719,43720],{"class":481},"'LESS'",[151,43722,106],{"class":503},[151,43724,6760],{"class":477},[151,43726,24817],{"class":503},[151,43728,43729],{"class":1527},"# I think is is related to CSS?\n",[151,43731,43732,43734,43737,43739,43741,43743],{"class":469,"line":32389},[151,43733,129],{"class":503},[151,43735,43736],{"class":481},"'AMAZON ECHO'",[151,43738,106],{"class":503},[151,43740,6760],{"class":477},[151,43742,16995],{"class":503},[151,43744,43745],{"class":1527},"# I once built an open source Echo on a raspberry pi\n",[151,43747,43748],{"class":469,"line":32394},[151,43749,3691],{"class":503},[11,43751,43752],{},"The list above gives a count of the different skills in all job postings sorted by the most common skills. But what about the most common skills listed together with any given skill? This would allow us to answer questions like \"what skills appear most frequently along with JavaScript?\"",[11,43754,43755],{},"We can find this with by doing:",[459,43757,43759],{"className":24401,"code":43758,"language":24403,"meta":464,"style":464},"skills_frequency = defaultdict(lambda: defaultdict(lambda: 0))\n\nfor company in company_list:\n    if company[\"jobs\"] is not None:\n        for job in company[\"jobs\"]:\n            if job[\"job_skills\"] is not None:\n                job_skills = job[\"job_skills\"]\n\n                skill_tuples = itertools.permutations(job_skills, 2)\n\n                for skill_tuple in skill_tuples:\n                    first = skill_tuple[0]\n                    second = skill_tuple[1]\n\n                    skills_frequency[first][second] += 1\n\npprint.pprint(\n    {\n        key: sorted(value.items(), key=lambda kv: -kv[1])\n        for key, value in skills_frequency.items()\n    }\n)\n",[30,43760,43761,43785,43789,43799,43817,43831,43849,43862,43866,43880,43884,43896,43910,43923,43927,43936,43940,43945,43949,43980,43992,43996],{"__ignoreMap":464},[151,43762,43763,43766,43768,43771,43774,43777,43779,43781,43783],{"class":469,"line":470},[151,43764,43765],{"class":503},"skills_frequency ",[151,43767,1876],{"class":1869},[151,43769,43770],{"class":503}," defaultdict(",[151,43772,43773],{"class":12347},"lambda",[151,43775,43776],{"class":503},": defaultdict(",[151,43778,43773],{"class":12347},[151,43780,6208],{"class":503},[151,43782,9181],{"class":477},[151,43784,12451],{"class":503},[151,43786,43787],{"class":469,"line":488},[151,43788,1090],{"emptyLinePlaceholder":609},[151,43790,43791,43793,43795,43797],{"class":469,"line":500},[151,43792,16732],{"class":1869},[151,43794,41094],{"class":503},[151,43796,16417],{"class":1869},[151,43798,41444],{"class":503},[151,43800,43801,43803,43805,43807,43809,43811,43813,43815],{"class":469,"line":509},[151,43802,23327],{"class":1869},[151,43804,41451],{"class":503},[151,43806,41454],{"class":481},[151,43808,16654],{"class":503},[151,43810,40448],{"class":1869},[151,43812,4191],{"class":1869},[151,43814,40451],{"class":477},[151,43816,14372],{"class":503},[151,43818,43819,43821,43823,43825,43827,43829],{"class":469,"line":517},[151,43820,16616],{"class":1869},[151,43822,40999],{"class":503},[151,43824,16417],{"class":1869},[151,43826,41451],{"class":503},[151,43828,41454],{"class":481},[151,43830,17073],{"class":503},[151,43832,43833,43835,43837,43839,43841,43843,43845,43847],{"class":469,"line":534},[151,43834,40442],{"class":1869},[151,43836,41485],{"class":503},[151,43838,41488],{"class":481},[151,43840,16654],{"class":503},[151,43842,40448],{"class":1869},[151,43844,4191],{"class":1869},[151,43846,40451],{"class":477},[151,43848,14372],{"class":503},[151,43850,43851,43854,43856,43858,43860],{"class":469,"line":1413},[151,43852,43853],{"class":503},"                job_skills ",[151,43855,1876],{"class":1869},[151,43857,41485],{"class":503},[151,43859,41488],{"class":481},[151,43861,3691],{"class":503},[151,43863,43864],{"class":469,"line":1418},[151,43865,1090],{"emptyLinePlaceholder":609},[151,43867,43868,43871,43873,43876,43878],{"class":469,"line":2462},[151,43869,43870],{"class":503},"                skill_tuples ",[151,43872,1876],{"class":1869},[151,43874,43875],{"class":503}," itertools.permutations(job_skills, ",[151,43877,6619],{"class":477},[151,43879,3640],{"class":503},[151,43881,43882],{"class":469,"line":2471},[151,43883,1090],{"emptyLinePlaceholder":609},[151,43885,43886,43888,43891,43893],{"class":469,"line":2480},[151,43887,41503],{"class":1869},[151,43889,43890],{"class":503}," skill_tuple ",[151,43892,16417],{"class":1869},[151,43894,43895],{"class":503}," skill_tuples:\n",[151,43897,43898,43901,43903,43906,43908],{"class":469,"line":2489},[151,43899,43900],{"class":503},"                    first ",[151,43902,1876],{"class":1869},[151,43904,43905],{"class":503}," skill_tuple[",[151,43907,9181],{"class":477},[151,43909,3691],{"class":503},[151,43911,43912,43915,43917,43919,43921],{"class":469,"line":2497},[151,43913,43914],{"class":503},"                    second ",[151,43916,1876],{"class":1869},[151,43918,43905],{"class":503},[151,43920,6760],{"class":477},[151,43922,3691],{"class":503},[151,43924,43925],{"class":469,"line":3140},[151,43926,1090],{"emptyLinePlaceholder":609},[151,43928,43929,43932,43934],{"class":469,"line":3149},[151,43930,43931],{"class":503},"                    skills_frequency[first][second] ",[151,43933,24780],{"class":1869},[151,43935,3181],{"class":477},[151,43937,43938],{"class":469,"line":3158},[151,43939,1090],{"emptyLinePlaceholder":609},[151,43941,43942],{"class":469,"line":3167},[151,43943,43944],{"class":503},"pprint.pprint(\n",[151,43946,43947],{"class":469,"line":3175},[151,43948,9404],{"class":503},[151,43950,43951,43954,43957,43960,43962,43964,43966,43969,43971,43973,43976,43978],{"class":469,"line":3184},[151,43952,43953],{"class":503},"        key: ",[151,43955,43956],{"class":2226},"sorted",[151,43958,43959],{"class":503},"(value.items(), ",[151,43961,18175],{"class":15210},[151,43963,1876],{"class":1869},[151,43965,43773],{"class":12347},[151,43967,43968],{"class":15232}," kv",[151,43970,6208],{"class":503},[151,43972,12445],{"class":1869},[151,43974,43975],{"class":503},"kv[",[151,43977,6760],{"class":477},[151,43979,38820],{"class":503},[151,43981,43982,43984,43987,43989],{"class":469,"line":3193},[151,43983,16616],{"class":1869},[151,43985,43986],{"class":503}," key, value ",[151,43988,16417],{"class":1869},[151,43990,43991],{"class":503}," skills_frequency.items()\n",[151,43993,43994],{"class":469,"line":3720},[151,43995,9461],{"class":503},[151,43997,43998],{"class":469,"line":3729},[151,43999,3640],{"class":503},[142,44001,44002],{},[44003,44004],"skill-frequencies",{},[736,44006,44008],{"id":44007},"what-are-these-companies-working-on","What are these companies working on?",[11,44010,44011],{},"Here's a wordcloud made from the short company descriptions:",[11,44013,44014],{},[2718,44015],{"alt":20386,"src":44016},"/static/company_desc_wc.png",[459,44018,44020],{"className":24401,"code":44019,"language":24403,"meta":464,"style":464},"import json\nimport random\n\nfrom collections import Counter\nfrom os import path\n\nimport matplotlib.pyplot as plt\nimport numpy as np\n\nfrom PIL import Image\nfrom wordcloud import WordCloud, STOPWORDS\n\nHTML_FILE = \"waas_data.json\"\nwith open(HTML_FILE, 'r') as j:\n     company_list = json.loads(j.read())\n\ncompany_names = [company.get(\"company_name\", \" \").lower() for company in company_list]\ncompany_description_list = [company.get(\"company_desc\", \" \").lower().replace(\".\", \"\") for company in company_list]\ncompany_descriptions = \" \".join(company_description_list)\n\nwc = WordCloud(background_color=\"white\", width=1920, height=1080, max_words=500, stopwords=STOPWORDS, margin=10,\n               random_state=1).generate(company_descriptions)\n\ndefault_colors = wc.to_array()\n\nplt.figure(figsize=(40, 40))\nplt.imshow(wc, interpolation=\"bilinear\")\n\nplt.axis(\"off\")\nplt.savefig('company_description_wc.png')\nplt.show()\n",[30,44021,44022,44028,44035,44039,44051,44063,44067,44079,44089,44093,44106,44121,44125,44135,44157,44167,44171,44200,44236,44249,44253,44321,44333,44337,44347,44351,44372,44387,44391,44401,44411],{"__ignoreMap":464},[151,44023,44024,44026],{"class":469,"line":470},[151,44025,16859],{"class":1869},[151,44027,24063],{"class":503},[151,44029,44030,44032],{"class":469,"line":488},[151,44031,16859],{"class":1869},[151,44033,44034],{"class":503}," random\n",[151,44036,44037],{"class":469,"line":500},[151,44038,1090],{"emptyLinePlaceholder":609},[151,44040,44041,44043,44046,44048],{"class":469,"line":509},[151,44042,16853],{"class":1869},[151,44044,44045],{"class":503}," collections ",[151,44047,16859],{"class":1869},[151,44049,44050],{"class":503}," Counter\n",[151,44052,44053,44055,44058,44060],{"class":469,"line":517},[151,44054,16853],{"class":1869},[151,44056,44057],{"class":503}," os ",[151,44059,16859],{"class":1869},[151,44061,44062],{"class":503}," path\n",[151,44064,44065],{"class":469,"line":534},[151,44066,1090],{"emptyLinePlaceholder":609},[151,44068,44069,44071,44074,44076],{"class":469,"line":1413},[151,44070,16859],{"class":1869},[151,44072,44073],{"class":503}," matplotlib.pyplot ",[151,44075,16998],{"class":1869},[151,44077,44078],{"class":503}," plt\n",[151,44080,44081,44083,44085,44087],{"class":469,"line":1418},[151,44082,16859],{"class":1869},[151,44084,24412],{"class":503},[151,44086,16998],{"class":1869},[151,44088,24417],{"class":503},[151,44090,44091],{"class":469,"line":2462},[151,44092,1090],{"emptyLinePlaceholder":609},[151,44094,44095,44097,44100,44103],{"class":469,"line":2471},[151,44096,16853],{"class":1869},[151,44098,44099],{"class":477}," PIL",[151,44101,44102],{"class":1869}," import",[151,44104,44105],{"class":503}," Image\n",[151,44107,44108,44110,44113,44115,44118],{"class":469,"line":2480},[151,44109,16853],{"class":1869},[151,44111,44112],{"class":503}," wordcloud ",[151,44114,16859],{"class":1869},[151,44116,44117],{"class":503}," WordCloud, ",[151,44119,44120],{"class":477},"STOPWORDS\n",[151,44122,44123],{"class":469,"line":2489},[151,44124,1090],{"emptyLinePlaceholder":609},[151,44126,44127,44130,44132],{"class":469,"line":2497},[151,44128,44129],{"class":477},"HTML_FILE",[151,44131,19865],{"class":1869},[151,44133,44134],{"class":481}," \"waas_data.json\"\n",[151,44136,44137,44139,44141,44143,44145,44147,44150,44152,44154],{"class":469,"line":3140},[151,44138,24959],{"class":1869},[151,44140,16970],{"class":2226},[151,44142,12386],{"class":503},[151,44144,44129],{"class":477},[151,44146,106],{"class":503},[151,44148,44149],{"class":481},"'r'",[151,44151,16995],{"class":503},[151,44153,16998],{"class":1869},[151,44155,44156],{"class":503}," j:\n",[151,44158,44159,44162,44164],{"class":469,"line":3149},[151,44160,44161],{"class":503},"     company_list ",[151,44163,1876],{"class":1869},[151,44165,44166],{"class":503}," json.loads(j.read())\n",[151,44168,44169],{"class":469,"line":3158},[151,44170,1090],{"emptyLinePlaceholder":609},[151,44172,44173,44176,44178,44181,44184,44186,44188,44191,44193,44195,44197],{"class":469,"line":3167},[151,44174,44175],{"class":503},"company_names ",[151,44177,1876],{"class":1869},[151,44179,44180],{"class":503}," [company.get(",[151,44182,44183],{"class":481},"\"company_name\"",[151,44185,106],{"class":503},[151,44187,24311],{"class":481},[151,44189,44190],{"class":503},").lower() ",[151,44192,16732],{"class":1869},[151,44194,41094],{"class":503},[151,44196,16417],{"class":1869},[151,44198,44199],{"class":503}," company_list]\n",[151,44201,44202,44205,44207,44209,44212,44214,44216,44219,44222,44224,44226,44228,44230,44232,44234],{"class":469,"line":3175},[151,44203,44204],{"class":503},"company_description_list ",[151,44206,1876],{"class":1869},[151,44208,44180],{"class":503},[151,44210,44211],{"class":481},"\"company_desc\"",[151,44213,106],{"class":503},[151,44215,24311],{"class":481},[151,44217,44218],{"class":503},").lower().replace(",[151,44220,44221],{"class":481},"\".\"",[151,44223,106],{"class":503},[151,44225,38471],{"class":481},[151,44227,16995],{"class":503},[151,44229,16732],{"class":1869},[151,44231,41094],{"class":503},[151,44233,16417],{"class":1869},[151,44235,44199],{"class":503},[151,44237,44238,44241,44243,44246],{"class":469,"line":3184},[151,44239,44240],{"class":503},"company_descriptions ",[151,44242,1876],{"class":1869},[151,44244,44245],{"class":481}," \" \"",[151,44247,44248],{"class":503},".join(company_description_list)\n",[151,44250,44251],{"class":469,"line":3193},[151,44252,1090],{"emptyLinePlaceholder":609},[151,44254,44255,44258,44260,44263,44266,44268,44271,44273,44276,44278,44281,44283,44286,44288,44291,44293,44296,44298,44300,44302,44305,44307,44310,44312,44315,44317,44319],{"class":469,"line":3720},[151,44256,44257],{"class":503},"wc ",[151,44259,1876],{"class":1869},[151,44261,44262],{"class":503}," WordCloud(",[151,44264,44265],{"class":15210},"background_color",[151,44267,1876],{"class":1869},[151,44269,44270],{"class":481},"\"white\"",[151,44272,106],{"class":503},[151,44274,44275],{"class":15210},"width",[151,44277,1876],{"class":1869},[151,44279,44280],{"class":477},"1920",[151,44282,106],{"class":503},[151,44284,44285],{"class":15210},"height",[151,44287,1876],{"class":1869},[151,44289,44290],{"class":477},"1080",[151,44292,106],{"class":503},[151,44294,44295],{"class":15210},"max_words",[151,44297,1876],{"class":1869},[151,44299,12208],{"class":477},[151,44301,106],{"class":503},[151,44303,44304],{"class":15210},"stopwords",[151,44306,1876],{"class":1869},[151,44308,44309],{"class":477},"STOPWORDS",[151,44311,106],{"class":503},[151,44313,44314],{"class":15210},"margin",[151,44316,1876],{"class":1869},[151,44318,12423],{"class":477},[151,44320,9417],{"class":503},[151,44322,44323,44326,44328,44330],{"class":469,"line":3729},[151,44324,44325],{"class":15210},"               random_state",[151,44327,1876],{"class":1869},[151,44329,6760],{"class":477},[151,44331,44332],{"class":503},").generate(company_descriptions)\n",[151,44334,44335],{"class":469,"line":3735},[151,44336,1090],{"emptyLinePlaceholder":609},[151,44338,44339,44342,44344],{"class":469,"line":3745},[151,44340,44341],{"class":503},"default_colors ",[151,44343,1876],{"class":1869},[151,44345,44346],{"class":503}," wc.to_array()\n",[151,44348,44349],{"class":469,"line":3754},[151,44350,1090],{"emptyLinePlaceholder":609},[151,44352,44353,44356,44359,44361,44363,44366,44368,44370],{"class":469,"line":3760},[151,44354,44355],{"class":503},"plt.figure(",[151,44357,44358],{"class":15210},"figsize",[151,44360,1876],{"class":1869},[151,44362,12386],{"class":503},[151,44364,44365],{"class":477},"40",[151,44367,106],{"class":503},[151,44369,44365],{"class":477},[151,44371,12451],{"class":503},[151,44373,44374,44377,44380,44382,44385],{"class":469,"line":3773},[151,44375,44376],{"class":503},"plt.imshow(wc, ",[151,44378,44379],{"class":15210},"interpolation",[151,44381,1876],{"class":1869},[151,44383,44384],{"class":481},"\"bilinear\"",[151,44386,3640],{"class":503},[151,44388,44389],{"class":469,"line":3782},[151,44390,1090],{"emptyLinePlaceholder":609},[151,44392,44393,44396,44399],{"class":469,"line":3791},[151,44394,44395],{"class":503},"plt.axis(",[151,44397,44398],{"class":481},"\"off\"",[151,44400,3640],{"class":503},[151,44402,44403,44406,44409],{"class":469,"line":3803},[151,44404,44405],{"class":503},"plt.savefig(",[151,44407,44408],{"class":481},"'company_description_wc.png'",[151,44410,3640],{"class":503},[151,44412,44413],{"class":469,"line":3811},[151,44414,44415],{"class":503},"plt.show()\n",[11,44417,44418],{},"Here's a breakdown of YC companies by category and sub category:",[142,44420,44421],{},[44422,44423],"category-breakdown-chart",{},[736,44425,44427],{"id":44426},"salary-equity-and-years-of-experience","Salary, Equity and Years of Experience",[11,44429,44430],{},"Here's a scatterplot showing average salary and average equity for positions categorized by years of experience required.",[142,44432,44433],{},[44434,44435],"salary-equity-scatter",{},[736,44437,44439],{"id":44438},"logos","Logos",[11,44441,44442],{},"Here's a look at about 600 of the 750 logos that were made available in the list of companies. The logos are sorted by their average hex color, which puts them on a gradient of dark to light:",[11,44444,44445],{},[2718,44446],{"alt":20386,"src":44447},"/static/waas/yc.png",[459,44449,44451],{"className":24401,"code":44450,"language":24403,"meta":464,"style":464},"import os\nimport PIL\nfrom PIL import Image\nfrom IPython.display import display, Image as IPyImage\nimport matplotlib.pyplot as plt\nimport matplotlib.image as mpimg\n%matplotlib inline\n\nLOGO_DIR = 'data/waas_full_details_dump_files/'\nyc_logos = [LOGO_DIR + x for x in os.listdir(LOGO_DIR) if x.endswith('.png')]\n\ndef average_img_hex(img):\n    \"\"\"\n    https://www.hackzine.org/getting-average-image-color-from-python.html\n    \"\"\"\n    img = Image.open(img)\n\n    # leave out images not in RGB/RGBA mode\n    if img.mode in [\"LA\", \"P\", \"L\"]:\n        return\n\n    # resize the image to 1 pixel and get the average hex value\n    img2 = img.resize((1, 1))\n    color = img2.getpixel((0, 0))\n    average_hex = '#{:02x}{:02x}{:02x}'.format(*color)\n\n    return average_hex\n\n# sort images by average hex value\nsorted_images = sorted(\n    [(average_img_hex(img), img) for img in yc_logos if average_img_hex(img) is not None],\n    key=lambda x: x[0]\n)\n\nimages = sorted_images[:600]\n\nfig, axes = plt.subplots(20, 30, figsize=(30, 15), sharex=False, sharey=False)\n\nfor img, ax in zip(images, axes.flat):\n    ax.imshow(mpimg.imread(img[1]))\n    ax.axis('off')\n\nplt.savefig('yc.png')\nplt.show()\n\n",[30,44452,44453,44459,44466,44476,44493,44503,44515,44523,44527,44537,44577,44581,44594,44598,44603,44607,44617,44621,44626,44652,44657,44661,44666,44684,44702,44738,44742,44749,44753,44758,44770,44798,44816,44820,44824,44839,44843,44893,44897,44912,44922,44932,44936,44945],{"__ignoreMap":464},[151,44454,44455,44457],{"class":469,"line":470},[151,44456,16859],{"class":1869},[151,44458,24070],{"class":503},[151,44460,44461,44463],{"class":469,"line":488},[151,44462,16859],{"class":1869},[151,44464,44465],{"class":477}," PIL\n",[151,44467,44468,44470,44472,44474],{"class":469,"line":500},[151,44469,16853],{"class":1869},[151,44471,44099],{"class":477},[151,44473,44102],{"class":1869},[151,44475,44105],{"class":503},[151,44477,44478,44480,44483,44485,44488,44490],{"class":469,"line":509},[151,44479,16853],{"class":1869},[151,44481,44482],{"class":503}," IPython.display ",[151,44484,16859],{"class":1869},[151,44486,44487],{"class":503}," display, Image ",[151,44489,16998],{"class":1869},[151,44491,44492],{"class":503}," IPyImage\n",[151,44494,44495,44497,44499,44501],{"class":469,"line":517},[151,44496,16859],{"class":1869},[151,44498,44073],{"class":503},[151,44500,16998],{"class":1869},[151,44502,44078],{"class":503},[151,44504,44505,44507,44510,44512],{"class":469,"line":534},[151,44506,16859],{"class":1869},[151,44508,44509],{"class":503}," matplotlib.image ",[151,44511,16998],{"class":1869},[151,44513,44514],{"class":503}," mpimg\n",[151,44516,44517,44520],{"class":469,"line":1413},[151,44518,44519],{"class":1869},"%",[151,44521,44522],{"class":503},"matplotlib inline\n",[151,44524,44525],{"class":469,"line":1418},[151,44526,1090],{"emptyLinePlaceholder":609},[151,44528,44529,44532,44534],{"class":469,"line":2462},[151,44530,44531],{"class":477},"LOGO_DIR",[151,44533,19865],{"class":1869},[151,44535,44536],{"class":481}," 'data/waas_full_details_dump_files/'\n",[151,44538,44539,44542,44544,44546,44548,44550,44553,44555,44557,44559,44562,44564,44566,44568,44571,44574],{"class":469,"line":2471},[151,44540,44541],{"class":503},"yc_logos ",[151,44543,1876],{"class":1869},[151,44545,6604],{"class":503},[151,44547,44531],{"class":477},[151,44549,23378],{"class":1869},[151,44551,44552],{"class":503}," x ",[151,44554,16732],{"class":1869},[151,44556,44552],{"class":503},[151,44558,16417],{"class":1869},[151,44560,44561],{"class":503}," os.listdir(",[151,44563,44531],{"class":477},[151,44565,16995],{"class":503},[151,44567,17218],{"class":1869},[151,44569,44570],{"class":503}," x.endswith(",[151,44572,44573],{"class":481},"'.png'",[151,44575,44576],{"class":503},")]\n",[151,44578,44579],{"class":469,"line":2480},[151,44580,1090],{"emptyLinePlaceholder":609},[151,44582,44583,44585,44588,44590,44592],{"class":469,"line":2489},[151,44584,16925],{"class":12347},[151,44586,44587],{"class":473}," average_img_hex",[151,44589,12386],{"class":503},[151,44591,2718],{"class":15232},[151,44593,15264],{"class":503},[151,44595,44596],{"class":469,"line":2497},[151,44597,17384],{"class":481},[151,44599,44600],{"class":469,"line":3140},[151,44601,44602],{"class":481},"    https://www.hackzine.org/getting-average-image-color-from-python.html\n",[151,44604,44605],{"class":469,"line":3149},[151,44606,17384],{"class":481},[151,44608,44609,44612,44614],{"class":469,"line":3158},[151,44610,44611],{"class":503},"    img ",[151,44613,1876],{"class":1869},[151,44615,44616],{"class":503}," Image.open(img)\n",[151,44618,44619],{"class":469,"line":3167},[151,44620,1090],{"emptyLinePlaceholder":609},[151,44622,44623],{"class":469,"line":3175},[151,44624,44625],{"class":1527},"    # leave out images not in RGB/RGBA mode\n",[151,44627,44628,44630,44633,44635,44637,44640,44642,44645,44647,44650],{"class":469,"line":3184},[151,44629,23327],{"class":1869},[151,44631,44632],{"class":503}," img.mode ",[151,44634,16417],{"class":1869},[151,44636,6604],{"class":503},[151,44638,44639],{"class":481},"\"LA\"",[151,44641,106],{"class":503},[151,44643,44644],{"class":481},"\"P\"",[151,44646,106],{"class":503},[151,44648,44649],{"class":481},"\"L\"",[151,44651,17073],{"class":503},[151,44653,44654],{"class":469,"line":3193},[151,44655,44656],{"class":1869},"        return\n",[151,44658,44659],{"class":469,"line":3720},[151,44660,1090],{"emptyLinePlaceholder":609},[151,44662,44663],{"class":469,"line":3729},[151,44664,44665],{"class":1527},"    # resize the image to 1 pixel and get the average hex value\n",[151,44667,44668,44671,44673,44676,44678,44680,44682],{"class":469,"line":3735},[151,44669,44670],{"class":503},"    img2 ",[151,44672,1876],{"class":1869},[151,44674,44675],{"class":503}," img.resize((",[151,44677,6760],{"class":477},[151,44679,106],{"class":503},[151,44681,6760],{"class":477},[151,44683,12451],{"class":503},[151,44685,44686,44689,44691,44694,44696,44698,44700],{"class":469,"line":3745},[151,44687,44688],{"class":503},"    color ",[151,44690,1876],{"class":1869},[151,44692,44693],{"class":503}," img2.getpixel((",[151,44695,9181],{"class":477},[151,44697,106],{"class":503},[151,44699,9181],{"class":477},[151,44701,12451],{"class":503},[151,44703,44704,44707,44709,44712,44714,44717,44720,44722,44724,44726,44728,44730,44733,44735],{"class":469,"line":3754},[151,44705,44706],{"class":503},"    average_hex ",[151,44708,1876],{"class":1869},[151,44710,44711],{"class":481}," '#",[151,44713,5729],{"class":477},[151,44715,44716],{"class":12347},":02x",[151,44718,44719],{"class":477},"}{",[151,44721,44716],{"class":12347},[151,44723,44719],{"class":477},[151,44725,44716],{"class":12347},[151,44727,2001],{"class":477},[151,44729,13223],{"class":481},[151,44731,44732],{"class":503},".format(",[151,44734,23268],{"class":1869},[151,44736,44737],{"class":503},"color)\n",[151,44739,44740],{"class":469,"line":3760},[151,44741,1090],{"emptyLinePlaceholder":609},[151,44743,44744,44746],{"class":469,"line":3773},[151,44745,17496],{"class":1869},[151,44747,44748],{"class":503}," average_hex\n",[151,44750,44751],{"class":469,"line":3782},[151,44752,1090],{"emptyLinePlaceholder":609},[151,44754,44755],{"class":469,"line":3791},[151,44756,44757],{"class":1527},"# sort images by average hex value\n",[151,44759,44760,44763,44765,44768],{"class":469,"line":3803},[151,44761,44762],{"class":503},"sorted_images ",[151,44764,1876],{"class":1869},[151,44766,44767],{"class":2226}," sorted",[151,44769,15410],{"class":503},[151,44771,44772,44775,44777,44780,44782,44785,44787,44790,44792,44794,44796],{"class":469,"line":3811},[151,44773,44774],{"class":503},"    [(average_img_hex(img), img) ",[151,44776,16732],{"class":1869},[151,44778,44779],{"class":503}," img ",[151,44781,16417],{"class":1869},[151,44783,44784],{"class":503}," yc_logos ",[151,44786,17218],{"class":1869},[151,44788,44789],{"class":503}," average_img_hex(img) ",[151,44791,40448],{"class":1869},[151,44793,4191],{"class":1869},[151,44795,40451],{"class":477},[151,44797,18746],{"class":503},[151,44799,44800,44803,44805,44807,44809,44812,44814],{"class":469,"line":3820},[151,44801,44802],{"class":15210},"    key",[151,44804,1876],{"class":1869},[151,44806,43773],{"class":12347},[151,44808,27729],{"class":15232},[151,44810,44811],{"class":503},": x[",[151,44813,9181],{"class":477},[151,44815,3691],{"class":503},[151,44817,44818],{"class":469,"line":7084},[151,44819,3640],{"class":503},[151,44821,44822],{"class":469,"line":7148},[151,44823,1090],{"emptyLinePlaceholder":609},[151,44825,44826,44829,44831,44834,44837],{"class":469,"line":7211},[151,44827,44828],{"class":503},"images ",[151,44830,1876],{"class":1869},[151,44832,44833],{"class":503}," sorted_images[:",[151,44835,44836],{"class":477},"600",[151,44838,3691],{"class":503},[151,44840,44841],{"class":469,"line":7273},[151,44842,1090],{"emptyLinePlaceholder":609},[151,44844,44845,44848,44850,44853,44855,44857,44859,44861,44863,44865,44867,44869,44871,44873,44875,44878,44880,44882,44884,44887,44889,44891],{"class":469,"line":7335},[151,44846,44847],{"class":503},"fig, axes ",[151,44849,1876],{"class":1869},[151,44851,44852],{"class":503}," plt.subplots(",[151,44854,9097],{"class":477},[151,44856,106],{"class":503},[151,44858,42017],{"class":477},[151,44860,106],{"class":503},[151,44862,44358],{"class":15210},[151,44864,1876],{"class":1869},[151,44866,12386],{"class":503},[151,44868,42017],{"class":477},[151,44870,106],{"class":503},[151,44872,42310],{"class":477},[151,44874,24817],{"class":503},[151,44876,44877],{"class":15210},"sharex",[151,44879,1876],{"class":1869},[151,44881,39461],{"class":477},[151,44883,106],{"class":503},[151,44885,44886],{"class":15210},"sharey",[151,44888,1876],{"class":1869},[151,44890,39461],{"class":477},[151,44892,3640],{"class":503},[151,44894,44895],{"class":469,"line":7398},[151,44896,1090],{"emptyLinePlaceholder":609},[151,44898,44899,44901,44904,44906,44909],{"class":469,"line":7462},[151,44900,16732],{"class":1869},[151,44902,44903],{"class":503}," img, ax ",[151,44905,16417],{"class":1869},[151,44907,44908],{"class":2226}," zip",[151,44910,44911],{"class":503},"(images, axes.flat):\n",[151,44913,44914,44917,44919],{"class":469,"line":7467},[151,44915,44916],{"class":503},"    ax.imshow(mpimg.imread(img[",[151,44918,6760],{"class":477},[151,44920,44921],{"class":503},"]))\n",[151,44923,44924,44927,44930],{"class":469,"line":7532},[151,44925,44926],{"class":503},"    ax.axis(",[151,44928,44929],{"class":481},"'off'",[151,44931,3640],{"class":503},[151,44933,44934],{"class":469,"line":7537},[151,44935,1090],{"emptyLinePlaceholder":609},[151,44937,44938,44940,44943],{"class":469,"line":7603},[151,44939,44405],{"class":503},[151,44941,44942],{"class":481},"'yc.png'",[151,44944,3640],{"class":503},[151,44946,44947],{"class":469,"line":7608},[151,44948,44415],{"class":503},[11,44950,44951],{},"There are a lot of logos that have a similar design to Stripe's logo. Rose/peach/pamplemousse colored logos also seem to be popular.",[736,44953,44955],{"id":44954},"founders","Founders",[11,44957,44958,44959,44964],{},"Let's take a look at the founders. I came across the ",[20,44960,44963],{"href":44961,"rel":44962},"https://pypi.org/project/deepface/",[24],"deepface"," PyPI project an was impressed at how accurately it can classify face data.",[11,44966,44967],{},"Here's a sample of YC Founder headshots:",[11,44969,44970],{},[2718,44971],{"alt":20386,"src":44972},"/static/waas/yc_founders_sample.png",[11,44974,44975],{},"Here's how I used the deepface library to add race, gender and age data for each of the headshot images:",[459,44977,44979],{"className":24401,"code":44978,"language":24403,"meta":464,"style":464},"IMG_DIR = 'data/waas_full_details_dump_files/'\n# headshots are all .jpg files, so we can get all headshots like this:\nfounder_headshots = [x for x in os.listdir(IMG_DIR) if x.endswith('.jpg')]\nfounder_count = len(founder_headshots)\n\nimg_paths = [IMG_DIR + x for x in founder_headshots]\nresults = {}\n\nfor idx, img in enumerate(img_paths):\n    print(f\"analyzing {idx}/{founder_count}\")\n    try:\n        img_key = img.split(\"/\")[-1]\n        obj = DeepFace.analyze(\n            img_path=img,\n            actions=['age', 'gender', 'race', 'emotion'],\n            enforce_detection=False\n        )\n        obj.update({\"img\": img_key})\n        results[img_key] = obj\n\n    except ValueError as e:\n        print(e)\n\nwith open(\"founder_images.json\", \"w+\") as f:\n    f.write(json.dumps(results))\n",[30,44980,44981,44990,44995,45026,45039,45043,45067,45076,45080,45094,45125,45131,45152,45162,45172,45201,45211,45215,45226,45236,45240,45251,45257,45261,45283],{"__ignoreMap":464},[151,44982,44983,44986,44988],{"class":469,"line":470},[151,44984,44985],{"class":477},"IMG_DIR",[151,44987,19865],{"class":1869},[151,44989,44536],{"class":481},[151,44991,44992],{"class":469,"line":488},[151,44993,44994],{"class":1527},"# headshots are all .jpg files, so we can get all headshots like this:\n",[151,44996,44997,45000,45002,45005,45007,45009,45011,45013,45015,45017,45019,45021,45024],{"class":469,"line":500},[151,44998,44999],{"class":503},"founder_headshots ",[151,45001,1876],{"class":1869},[151,45003,45004],{"class":503}," [x ",[151,45006,16732],{"class":1869},[151,45008,44552],{"class":503},[151,45010,16417],{"class":1869},[151,45012,44561],{"class":503},[151,45014,44985],{"class":477},[151,45016,16995],{"class":503},[151,45018,17218],{"class":1869},[151,45020,44570],{"class":503},[151,45022,45023],{"class":481},"'.jpg'",[151,45025,44576],{"class":503},[151,45027,45028,45031,45033,45036],{"class":469,"line":509},[151,45029,45030],{"class":503},"founder_count ",[151,45032,1876],{"class":1869},[151,45034,45035],{"class":2226}," len",[151,45037,45038],{"class":503},"(founder_headshots)\n",[151,45040,45041],{"class":469,"line":517},[151,45042,1090],{"emptyLinePlaceholder":609},[151,45044,45045,45048,45050,45052,45054,45056,45058,45060,45062,45064],{"class":469,"line":534},[151,45046,45047],{"class":503},"img_paths ",[151,45049,1876],{"class":1869},[151,45051,6604],{"class":503},[151,45053,44985],{"class":477},[151,45055,23378],{"class":1869},[151,45057,44552],{"class":503},[151,45059,16732],{"class":1869},[151,45061,44552],{"class":503},[151,45063,16417],{"class":1869},[151,45065,45066],{"class":503}," founder_headshots]\n",[151,45068,45069,45072,45074],{"class":469,"line":1413},[151,45070,45071],{"class":503},"results ",[151,45073,1876],{"class":1869},[151,45075,16634],{"class":503},[151,45077,45078],{"class":469,"line":1418},[151,45079,1090],{"emptyLinePlaceholder":609},[151,45081,45082,45084,45087,45089,45091],{"class":469,"line":2462},[151,45083,16732],{"class":1869},[151,45085,45086],{"class":503}," idx, img ",[151,45088,16417],{"class":1869},[151,45090,17042],{"class":2226},[151,45092,45093],{"class":503},"(img_paths):\n",[151,45095,45096,45098,45100,45102,45105,45107,45110,45112,45114,45116,45119,45121,45123],{"class":469,"line":2471},[151,45097,24285],{"class":2226},[151,45099,12386],{"class":503},[151,45101,13214],{"class":12347},[151,45103,45104],{"class":481},"\"analyzing ",[151,45106,5729],{"class":477},[151,45108,45109],{"class":503},"idx",[151,45111,2001],{"class":477},[151,45113,19883],{"class":481},[151,45115,5729],{"class":477},[151,45117,45118],{"class":503},"founder_count",[151,45120,2001],{"class":477},[151,45122,8592],{"class":481},[151,45124,3640],{"class":503},[151,45126,45127,45129],{"class":469,"line":2480},[151,45128,18280],{"class":1869},[151,45130,14372],{"class":503},[151,45132,45133,45136,45138,45141,45144,45146,45148,45150],{"class":469,"line":2489},[151,45134,45135],{"class":503},"        img_key ",[151,45137,1876],{"class":1869},[151,45139,45140],{"class":503}," img.split(",[151,45142,45143],{"class":481},"\"/\"",[151,45145,40832],{"class":503},[151,45147,12445],{"class":1869},[151,45149,6760],{"class":477},[151,45151,3691],{"class":503},[151,45153,45154,45157,45159],{"class":469,"line":2497},[151,45155,45156],{"class":503},"        obj ",[151,45158,1876],{"class":1869},[151,45160,45161],{"class":503}," DeepFace.analyze(\n",[151,45163,45164,45167,45169],{"class":469,"line":3140},[151,45165,45166],{"class":15210},"            img_path",[151,45168,1876],{"class":1869},[151,45170,45171],{"class":503},"img,\n",[151,45173,45174,45177,45179,45181,45184,45186,45189,45191,45194,45196,45199],{"class":469,"line":3149},[151,45175,45176],{"class":15210},"            actions",[151,45178,1876],{"class":1869},[151,45180,6698],{"class":503},[151,45182,45183],{"class":481},"'age'",[151,45185,106],{"class":503},[151,45187,45188],{"class":481},"'gender'",[151,45190,106],{"class":503},[151,45192,45193],{"class":481},"'race'",[151,45195,106],{"class":503},[151,45197,45198],{"class":481},"'emotion'",[151,45200,18746],{"class":503},[151,45202,45203,45206,45208],{"class":469,"line":3158},[151,45204,45205],{"class":15210},"            enforce_detection",[151,45207,1876],{"class":1869},[151,45209,45210],{"class":477},"False\n",[151,45212,45213],{"class":469,"line":3167},[151,45214,16824],{"class":503},[151,45216,45217,45220,45223],{"class":469,"line":3175},[151,45218,45219],{"class":503},"        obj.update({",[151,45221,45222],{"class":481},"\"img\"",[151,45224,45225],{"class":503},": img_key})\n",[151,45227,45228,45231,45233],{"class":469,"line":3184},[151,45229,45230],{"class":503},"        results[img_key] ",[151,45232,1876],{"class":1869},[151,45234,45235],{"class":503}," obj\n",[151,45237,45238],{"class":469,"line":3193},[151,45239,1090],{"emptyLinePlaceholder":609},[151,45241,45242,45244,45247,45249],{"class":469,"line":3720},[151,45243,18341],{"class":1869},[151,45245,45246],{"class":6205}," ValueError",[151,45248,18347],{"class":1869},[151,45250,18350],{"class":503},[151,45252,45253,45255],{"class":469,"line":3729},[151,45254,18355],{"class":2226},[151,45256,18358],{"class":503},[151,45258,45259],{"class":469,"line":3735},[151,45260,1090],{"emptyLinePlaceholder":609},[151,45262,45263,45265,45267,45269,45272,45274,45277,45279,45281],{"class":469,"line":3745},[151,45264,24959],{"class":1869},[151,45266,16970],{"class":2226},[151,45268,12386],{"class":503},[151,45270,45271],{"class":481},"\"founder_images.json\"",[151,45273,106],{"class":503},[151,45275,45276],{"class":481},"\"w+\"",[151,45278,16995],{"class":503},[151,45280,16998],{"class":1869},[151,45282,17001],{"class":503},[151,45284,45285],{"class":469,"line":3754},[151,45286,45287],{"class":503},"    f.write(json.dumps(results))\n",[11,45289,45290],{},"There are more steps needed to transform this data to make it compatible for use with a histogram showing age, race and gender. Check out the Jupyter notebook linked at the end of this article to see the code used to make this data transformation. Here's a simple way to count founders grouped by race and gender:",[459,45292,45295],{"className":45293,"code":45294,"language":997},[995],"race_and_gender_count = defaultdict(lambda: 0)\nfor result in list(results):\n    obj = results[result]\n    gender = obj[\"gender\"]\n    race = obj[\"dominant_race\"]\n    # use (race, gender) tuple as defaultdict key and increment\n    race_and_gender_count[(race, gender)] += 1\n\nsorted(race_and_gender_count.items(), key=lambda x: x[1], reverse=True)\n",[30,45296,45294],{"__ignoreMap":464},[459,45298,45300],{"className":24401,"code":45299,"language":24403,"meta":464,"style":464},"[(('white', 'Man'), 771),\n (('asian', 'Man'), 217),\n (('latino hispanic', 'Man'), 134),\n (('indian', 'Man'), 117),\n (('middle eastern', 'Man'), 111),\n (('black', 'Man'), 84),\n (('white', 'Woman'), 52),\n (('asian', 'Woman'), 17),\n (('latino hispanic', 'Woman'), 13),\n (('indian', 'Woman'), 4),\n (('black', 'Woman'), 1)]\n",[30,45301,45302,45322,45341,45359,45377,45395,45413,45431,45447,45463,45479],{"__ignoreMap":464},[151,45303,45304,45307,45310,45312,45315,45317,45320],{"class":469,"line":470},[151,45305,45306],{"class":503},"[((",[151,45308,45309],{"class":481},"'white'",[151,45311,106],{"class":503},[151,45313,45314],{"class":481},"'Man'",[151,45316,24817],{"class":503},[151,45318,45319],{"class":477},"771",[151,45321,37985],{"class":503},[151,45323,45324,45327,45330,45332,45334,45336,45339],{"class":469,"line":488},[151,45325,45326],{"class":503}," ((",[151,45328,45329],{"class":481},"'asian'",[151,45331,106],{"class":503},[151,45333,45314],{"class":481},[151,45335,24817],{"class":503},[151,45337,45338],{"class":477},"217",[151,45340,37985],{"class":503},[151,45342,45343,45345,45348,45350,45352,45354,45357],{"class":469,"line":500},[151,45344,45326],{"class":503},[151,45346,45347],{"class":481},"'latino hispanic'",[151,45349,106],{"class":503},[151,45351,45314],{"class":481},[151,45353,24817],{"class":503},[151,45355,45356],{"class":477},"134",[151,45358,37985],{"class":503},[151,45360,45361,45363,45366,45368,45370,45372,45375],{"class":469,"line":509},[151,45362,45326],{"class":503},[151,45364,45365],{"class":481},"'indian'",[151,45367,106],{"class":503},[151,45369,45314],{"class":481},[151,45371,24817],{"class":503},[151,45373,45374],{"class":477},"117",[151,45376,37985],{"class":503},[151,45378,45379,45381,45384,45386,45388,45390,45393],{"class":469,"line":517},[151,45380,45326],{"class":503},[151,45382,45383],{"class":481},"'middle eastern'",[151,45385,106],{"class":503},[151,45387,45314],{"class":481},[151,45389,24817],{"class":503},[151,45391,45392],{"class":477},"111",[151,45394,37985],{"class":503},[151,45396,45397,45399,45402,45404,45406,45408,45411],{"class":469,"line":534},[151,45398,45326],{"class":503},[151,45400,45401],{"class":481},"'black'",[151,45403,106],{"class":503},[151,45405,45314],{"class":481},[151,45407,24817],{"class":503},[151,45409,45410],{"class":477},"84",[151,45412,37985],{"class":503},[151,45414,45415,45417,45419,45421,45424,45426,45429],{"class":469,"line":1413},[151,45416,45326],{"class":503},[151,45418,45309],{"class":481},[151,45420,106],{"class":503},[151,45422,45423],{"class":481},"'Woman'",[151,45425,24817],{"class":503},[151,45427,45428],{"class":477},"52",[151,45430,37985],{"class":503},[151,45432,45433,45435,45437,45439,45441,45443,45445],{"class":469,"line":1418},[151,45434,45326],{"class":503},[151,45436,45329],{"class":481},[151,45438,106],{"class":503},[151,45440,45423],{"class":481},[151,45442,24817],{"class":503},[151,45444,42293],{"class":477},[151,45446,37985],{"class":503},[151,45448,45449,45451,45453,45455,45457,45459,45461],{"class":469,"line":2462},[151,45450,45326],{"class":503},[151,45452,45347],{"class":481},[151,45454,106],{"class":503},[151,45456,45423],{"class":481},[151,45458,24817],{"class":503},[151,45460,42327],{"class":477},[151,45462,37985],{"class":503},[151,45464,45465,45467,45469,45471,45473,45475,45477],{"class":469,"line":2471},[151,45466,45326],{"class":503},[151,45468,45365],{"class":481},[151,45470,106],{"class":503},[151,45472,45423],{"class":481},[151,45474,24817],{"class":503},[151,45476,9187],{"class":477},[151,45478,37985],{"class":503},[151,45480,45481,45483,45485,45487,45489,45491,45493],{"class":469,"line":2480},[151,45482,45326],{"class":503},[151,45484,45401],{"class":481},[151,45486,106],{"class":503},[151,45488,45423],{"class":481},[151,45490,24817],{"class":503},[151,45492,6760],{"class":477},[151,45494,44576],{"class":503},[142,45496,45497],{},[45498,45499],"race-gender-age-bar-chart",{},[736,45501,45503],{"id":45502},"founder-background-wordcloud","Founder background wordcloud",[11,45505,45506],{},"Here's a wordcloudsshowing founder background, education and experience. The code for this is similar to the wordcloud shown previously for company descriptions.",[11,45508,45509],{},[2718,45510],{"alt":20386,"src":45511},"/static/founders_wc.png",[56,45513,45515],{"id":45514},"generating-yc-startup-companies","Generating YC Startup Companies",[11,45517,45518],{},"Finally, I'll try to generate some plausible descriptions of YC companies based on the descriptions of companies scraped from WaaS. I have read about big advancements made in text generation with GPT-3, but otherwise I'm not familiar with text-generation or any other generative models.",[11,45520,45521,45522,45525,45526,45530],{},"My initial goal was to do this using a simple example that I could replicate locally using Tensorflow. Googling for ",[30,45523,45524],{},"text generation with python tensorflow"," led me to this tutorial: ",[20,45527,45528],{"href":45528,"rel":45529},"https://www.thepythoncode.com/article/text-generation-keras-python",[24],". I was able to run the example, but the results were not very good, at least not as good as the results used in the example of generating text from a model trained on the text of \"Alice in Wonderland\". This is probably because I ran 10 epochs instead of 30, but I didn't want to wait hours before getting results for each iteration.",[11,45532,45533,45534,45539,45540,187,45542,643],{},"Another Google search led me to ",[20,45535,45538],{"href":45536,"rel":45537},"https://minimaxir.com/2019/09/howto-gpt2/",[24],"this article on Max Woolf's Blog"," which I was able to get started with in just a few minutes. I combined the text from the 2 sections of the longer company descriptions: ",[30,45541,19656],{},[30,45543,45544],{},"technology",[11,45546,45547],{},"Here's the link to the Google Colab (anyone can view and comment):",[11,45549,45550],{},[20,45551,45552],{"href":45552,"rel":45553},"https://colab.research.google.com/drive/1u9b-FVGgUGcfifLy7bUXgoPe-pZ81rm3?usp=sharing",[24],[11,45555,45556],{},"Google Colab gives you access to an environment with a GPU suitable for working with GPT2. Here's an overview of the code used to train GPT2 on the company descriptions:",[459,45558,45560],{"className":24401,"code":45559,"language":24403,"meta":464,"style":464},"%tensorflow_version 1.x\n!pip install -q gpt-2-simple\nimport gpt_2_simple as gpt2\nfrom datetime import datetime\nfrom google.colab import files\n",[30,45561,45562,45574,45595,45607,45619],{"__ignoreMap":464},[151,45563,45564,45566,45569,45572],{"class":469,"line":470},[151,45565,44519],{"class":1869},[151,45567,45568],{"class":503},"tensorflow_version ",[151,45570,45571],{"class":477},"1.",[151,45573,19765],{"class":503},[151,45575,45576,45578,45581,45583,45586,45588,45590,45592],{"class":469,"line":488},[151,45577,12282],{"class":6607},[151,45579,45580],{"class":503},"pip install ",[151,45582,12445],{"class":1869},[151,45584,45585],{"class":503},"q gpt",[151,45587,12445],{"class":1869},[151,45589,6619],{"class":477},[151,45591,12445],{"class":1869},[151,45593,45594],{"class":503},"simple\n",[151,45596,45597,45599,45602,45604],{"class":469,"line":500},[151,45598,16859],{"class":1869},[151,45600,45601],{"class":503}," gpt_2_simple ",[151,45603,16998],{"class":1869},[151,45605,45606],{"class":503}," gpt2\n",[151,45608,45609,45611,45614,45616],{"class":469,"line":509},[151,45610,16853],{"class":1869},[151,45612,45613],{"class":503}," datetime ",[151,45615,16859],{"class":1869},[151,45617,45618],{"class":503}," datetime\n",[151,45620,45621,45623,45626,45628],{"class":469,"line":517},[151,45622,16853],{"class":1869},[151,45624,45625],{"class":503}," google.colab ",[151,45627,16859],{"class":1869},[151,45629,45630],{"class":503}," files\n",[11,45632,45633,45634,45641],{},"This installs ",[20,45635,45638],{"href":45636,"rel":45637},"https://github.com/minimaxir/gpt-2-simple",[24],[30,45639,45640],{},"gpt-2-simple"," and gives us access to the Google Drive connected to the Google account used to sign in to Google Colab.",[11,45643,45644],{},"The GPU can be inspected with:",[459,45646,45649],{"className":45647,"code":45648,"language":997},[995],"!nvidia-smi\n",[30,45650,45648],{"__ignoreMap":464},[459,45652,45655],{"className":45653,"code":45654,"language":997},[995],"+-----------------------------------------------------------------------------+\n| NVIDIA-SMI 460.32.03    Driver Version: 418.67       CUDA Version: 10.1     |\n|-------------------------------+----------------------+----------------------+\n| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n|                               |                      |               MIG M. |\n|===============================+======================+======================|\n|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |\n| N/A   51C    P8    10W /  70W |      0MiB / 15079MiB |      0%      Default |\n|                               |                      |                 ERR! |\n+-------------------------------+----------------------+----------------------+\n\n+-----------------------------------------------------------------------------+\n| Processes:                                                                  |\n|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\n|        ID   ID                                                   Usage      |\n|=============================================================================|\n|  No running processes found                                                 |\n+-----------------------------------------------------------------------------+\n\n+-----------------------------------------------------------------------------+\n| NVIDIA-SMI 460.32.03    Driver Version: 418.67       CUDA Version: 10.1     |\n|-------------------------------+----------------------+----------------------+\n| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n|                               |                      |               MIG M. |\n|===============================+======================+======================|\n|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |\n| N/A   51C    P8    10W /  70W |      0MiB / 15079MiB |      0%      Default |\n|                               |                      |                 ERR! |\n+-------------------------------+----------------------+----------------------+\n\n+-----------------------------------------------------------------------------+\n| Processes:                                                                  |\n|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\n|        ID   ID                                                   Usage      |\n|=============================================================================|\n|  No running processes found                                                 |\n+-----------------------------------------------------------------------------+\n",[30,45656,45654],{"__ignoreMap":464},[11,45658,45659],{},"Next we download the GPT-2 model:",[459,45661,45663],{"className":24401,"code":45662,"language":24403,"meta":464,"style":464},"gpt2.download_gpt2(model_name=\"124M\")\n",[30,45664,45665],{"__ignoreMap":464},[151,45666,45667,45670,45672,45674,45677],{"class":469,"line":470},[151,45668,45669],{"class":503},"gpt2.download_gpt2(",[151,45671,16891],{"class":15210},[151,45673,1876],{"class":1869},[151,45675,45676],{"class":481},"\"124M\"",[151,45678,3640],{"class":503},[11,45680,45681],{},"Then we mount Google Drive with the following command:",[459,45683,45685],{"className":24401,"code":45684,"language":24403,"meta":464,"style":464},"gpt2.mount_gdrive()\n",[30,45686,45687],{"__ignoreMap":464},[151,45688,45689],{"class":469,"line":470},[151,45690,45684],{"class":503},[11,45692,45693,45694,45697],{},"With our file (",[30,45695,45696],{},"waas.txt",") uploaded to Google Drive, the following",[459,45699,45701],{"className":24401,"code":45700,"language":24403,"meta":464,"style":464},"file_name = \"waas.txt\"\ngpt2.copy_file_from_gdrive(file_name)\n",[30,45702,45703,45713],{"__ignoreMap":464},[151,45704,45705,45708,45710],{"class":469,"line":470},[151,45706,45707],{"class":503},"file_name ",[151,45709,1876],{"class":1869},[151,45711,45712],{"class":481}," \"waas.txt\"\n",[151,45714,45715],{"class":469,"line":488},[151,45716,45717],{"class":503},"gpt2.copy_file_from_gdrive(file_name)\n",[11,45719,45720],{},"Now the model needs to be fine-tuned:",[459,45722,45724],{"className":24401,"code":45723,"language":24403,"meta":464,"style":464},"sess = gpt2.start_tf_sess()\n\ngpt2.finetune(\n    sess,\n    dataset=file_name,\n    model_name='124M',\n    steps=1000,\n    restore_from='fresh',\n    run_name='run1',\n    print_every=10,\n    sample_every=200,\n    save_every=500\n)\n",[30,45725,45726,45736,45740,45745,45750,45760,45771,45782,45794,45806,45817,45828,45838],{"__ignoreMap":464},[151,45727,45728,45731,45733],{"class":469,"line":470},[151,45729,45730],{"class":503},"sess ",[151,45732,1876],{"class":1869},[151,45734,45735],{"class":503}," gpt2.start_tf_sess()\n",[151,45737,45738],{"class":469,"line":488},[151,45739,1090],{"emptyLinePlaceholder":609},[151,45741,45742],{"class":469,"line":500},[151,45743,45744],{"class":503},"gpt2.finetune(\n",[151,45746,45747],{"class":469,"line":509},[151,45748,45749],{"class":503},"    sess,\n",[151,45751,45752,45755,45757],{"class":469,"line":517},[151,45753,45754],{"class":15210},"    dataset",[151,45756,1876],{"class":1869},[151,45758,45759],{"class":503},"file_name,\n",[151,45761,45762,45764,45766,45769],{"class":469,"line":534},[151,45763,14760],{"class":15210},[151,45765,1876],{"class":1869},[151,45767,45768],{"class":481},"'124M'",[151,45770,9417],{"class":503},[151,45772,45773,45775,45777,45780],{"class":469,"line":1413},[151,45774,20492],{"class":15210},[151,45776,1876],{"class":1869},[151,45778,45779],{"class":477},"1000",[151,45781,9417],{"class":503},[151,45783,45784,45787,45789,45792],{"class":469,"line":1418},[151,45785,45786],{"class":15210},"    restore_from",[151,45788,1876],{"class":1869},[151,45790,45791],{"class":481},"'fresh'",[151,45793,9417],{"class":503},[151,45795,45796,45799,45801,45804],{"class":469,"line":2462},[151,45797,45798],{"class":15210},"    run_name",[151,45800,1876],{"class":1869},[151,45802,45803],{"class":481},"'run1'",[151,45805,9417],{"class":503},[151,45807,45808,45811,45813,45815],{"class":469,"line":2471},[151,45809,45810],{"class":15210},"    print_every",[151,45812,1876],{"class":1869},[151,45814,12423],{"class":477},[151,45816,9417],{"class":503},[151,45818,45819,45822,45824,45826],{"class":469,"line":2480},[151,45820,45821],{"class":15210},"    sample_every",[151,45823,1876],{"class":1869},[151,45825,41624],{"class":477},[151,45827,9417],{"class":503},[151,45829,45830,45833,45835],{"class":469,"line":2489},[151,45831,45832],{"class":15210},"    save_every",[151,45834,1876],{"class":1869},[151,45836,45837],{"class":477},"500\n",[151,45839,45840],{"class":469,"line":2497},[151,45841,3640],{"class":503},[11,45843,45844],{},"Here are some results from my first attempt at using Google Colab. Training the model will output a sample after every 200 steps. I have included only the first sample from the training, but you can see the output from each step in the Colab notebook.",[459,45846,45849],{"className":45847,"code":45848,"language":997},[995],"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/gpt_2_simple/src/sample.py:17: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\nInstructions for updating:\nUse tf.where in 2.0, which has the same broadcast rule as np.where\nLoading checkpoint models/124M/model.ckpt\nINFO:tensorflow:Restoring parameters from models/124M/model.ckpt\n  0%|          | 0/1 [00:00\u003C?, ?it/s]Loading dataset...\n100%|██████████| 1/1 [00:00\u003C00:00,  1.00it/s]\ndataset has 128333 tokens\nTraining...\n[10 | 28.79] loss=3.60 avg=3.60\n[20 | 50.75] loss=3.30 avg=3.45\n[30 | 73.39] loss=3.25 avg=3.38\n[40 | 96.80] loss=3.18 avg=3.33\n[50 | 121.40] loss=3.12 avg=3.29\n[60 | 145.52] loss=3.10 avg=3.26\n[70 | 169.15] loss=2.56 avg=3.16\n[80 | 193.06] loss=2.46 avg=3.07\n[90 | 217.28] loss=2.59 avg=3.01\n[100 | 241.38] loss=2.49 avg=2.96\n[110 | 265.33] loss=2.34 avg=2.90\n[120 | 289.30] loss=2.52 avg=2.86\n[130 | 313.46] loss=2.37 avg=2.82\n[140 | 337.69] loss=2.38 avg=2.79\n[150 | 361.84] loss=1.92 avg=2.73\n[160 | 385.94] loss=2.25 avg=2.70\n[170 | 410.01] loss=1.73 avg=2.63\n[180 | 434.10] loss=1.66 avg=2.58\n[190 | 458.18] loss=1.25 avg=2.50\n[200 | 482.22] loss=1.47 avg=2.44\n======== SAMPLE 1 ========\n\n to improve the usability of our Services and to provide better\n offline features through offline messaging. Our Platform\n connects users from across the country to collect data on\n non-US citizens abroad, and then gives US citizens a secure and\n authentic online presence while keeping their identities\n private. Our mission is to enable Americans to be more secure\n in online world.  We do this by putting a damper on online\n threats while preventing online attacks and slowing down the\n spread of cyberthreats. Backed by a world class Series A and B\n investments from leading technology firms, including Shas\n Ventures and Y Combinator, we\\'re an organization that believes\n in doing what is right. We\\'re built to last and provide an\n experience that\\'s built to last.\\xa0We\\'ve partnered with some\n of the world\\'s top law enforcement institutions including the\n Washington DC Metropolitan Detention Center, National Domestic\n Relations Task Force, Lambda Legal, Lambda Legal National Labs,\n NIMBYs Angels of San Francisco and Rally Labs provide legal\n technology & community education, as well as legal assistance/\n cure treatments. We\\'re a tech firm that\\'s built to last ‘lots\n of years 🙏. We\\'ve solved some really really hard problems in\n the past, and haven\\'t forgotten.We\\'re a team of passionate\n engineers, machine learning researchers, and business\n executives from Y Combinator and Naval Institute. We\\'ve just\n closed our Series B and C funding, and\\'s back at it with two\n more large-scale brothels and a bona fide law firm dedicated to\n bringing brothels to the people. Join us on this next chapter\n in our journey, and help us build the next Clark County jail.\n We\\'re looking for guys who can quickly learn leadership roles\n and fill in for leaders, not executives. We like to take what\n we learn and apply it to real problems. ReactJS, Node, Django,\n Postgres, and a sprinkle of AWS. ReactInfrastructure serves\n Express-like functions in the cloud. Our customers are law\n firms and other third-party service providers, largely\n comprised of tech startups. Our customers are generating\n billions of dollars in revenue through our platform every\n year.  Come join us and help us transform this dynamic into a\n platform-as-a-works. Rippling uses the following\n technologies:React Native JavaScriptFast, clean,\n nativeCSS3Deterministic executionReact NativeScriptWeb appLync,\n typescriptArchitecture-as-a-systemResponsible for maintaining\n and expanding the Ethereum and Bitcoin infrastructure. We\\'re\n based in New York City, and are backed by the VC-leading\n venture firms in the space. Our flagship product, Rippling, is\n an intuitive phone-in-a-person phone banking solution that\n integrates an RFID reader and password manager, as well as\n password managers dedicated to managing assets and managing\n cards. Users can create portfolios and trade on the Rippling\n marketplace, or access the technology to create their own\n portfolios on the Bitcoin and Ethereum markets. Simply put,\n Rippling is 21+ year old inventors. We\\'ve reinvented banking\n by making money available today through an open API. Banks now\n have an easy path to secure assets, including a digital wallet\n and code backed by an AllinAir’s digital asset portfolio. We\n are a small team of computer scientists building tools that\n improve customer experience, reduce wait times for businesses,\n and increase transparency in banking. Our platform runs on very\n little electricity, minimal mechanical power, and is 100x more\n robust 20 than traditional DPG/PGD systems. The system stores 1\n watt of energy and 8 watts of power per watt signal in a\n standard APS-C (Advanced Plasmon Super Capacitor Stacker).\n Cells in our cell-resistant cell-glide-based sprays secrete a\n protein that is targeted to trigger cell death, called a T\n cell. This triggered death is what causes breast cancer.\n Stacking together our diverse array of biological and chemical\n biology and chemical pathologies, and building in new energy\n efficiency technology to power our smart consumer products,\n we\\'re taking our users on an exciting journey toward modern\n energy efficiency. We are a team of computer scientists located\n in Barcelona, Spain, and we are focusing on the extremely first\n and foremost research and development applications in renewable\n energy. Current RCP is the largest R&D and R&E net for the 21st\n century, automating substantial paperwork and providing\n essential modern services. With the exception of emergency\n services and work, we are unaffected by the most common types\n of CO2 emissions including man-made nitrous oxide, human-made\n CO2, and volatile organic compounds. Current RCP delivers the\n highest quality, highest efficiency possible, including\n equipment, software, waste management, and notification\n systems, all in a modern, integrated and convenient way. By\n using RCP, consumers, businesses, civil society, researchers\n and others around the world can save money on power bills,\n reduce emissions and waste, receive timely assistance from the\n energy sector and be\n\n[210 | 517.61] loss=1.62 avg=2.40\n\n[...]\n\n[1000 | 2448.17] loss=0.07 avg=0.47\nSaving checkpoint/run1/model-1000\nWARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tensorflow_core/python/training/saver.py:963: remove_checkpoint (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\nInstructions for updating:\nUse standard file APIs to delete files with this prefix.\n",[30,45850,45848],{"__ignoreMap":464},[11,45852,45853],{},"Here's a sample generated using the trained model:",[210,45855,45856],{},[11,45857,45858],{},"We are a small team of MIT-trained researchers and entrepreneurs based in Palo Alto, California. We're creating a new form of transportation for people and gear. Based in the heart of downtown LA, our downtown L.A. location will be the last remaining obstacle preventing people from making, selling, and visiting the Smithsonian. We're building a disruptive technology that impacts the way people interact in the world in a positive way. Token Transit is digitizing much of the way we travel for real. We are building it to work for everyone, digitizing billions of things we buy and sell every day. Our platform connects real users and non-users in a global network. Our first product, UNITY, made it from scratch and is in mixed state. We are helping to develop the first truly international distributed logistics network powered by an AI and a IoT. Our mission is to revolutionize how companies move containers, trucks, and other loads across borders. Our technology is being architected and being implemented by experts across the globe. Our annual revenue:$150M/year. How we work We are a small, fast growing company with a core team based in San Francisco, CA.  Our software is being used by thousands of customers worldwide. We are building our technology platform in two key areas: (1) automated supply chain forecasting for companies and (2) internal analysis of company data to help leaders analyze and improve production. Our YearEnd blog is currently looking for Senior Electricians & Candidates. If you're excited about this exciting challenge, please apply. Burrow is a daily driver app for mobile that works on any device. With Burrow, your driver is on your phone to show your friends where you are going, which destinations you are on, and more. You can also add your phone to a list and get in-depth insights into the driving itself. We love to tie in-stream data, visualised by React Native, to the driving experience. We also take a data-driven approach to managing our customer list, managing our teams and our operations framework. Our API serves a large amount of native API functions, and using Cockroach to access these functions through microservices allowed us to interface with the relevant services (e.g. DynamoDB, Picnic and our own API). We also inserted a lot of debugging and monitoring functionality into the ecosystem. Our API serves a large amount of data - including relational and non-relational data - about where you are and what you've wanted to do the longest, what you've got, and where you are now. We also manage a lot of the operations of the platform and provide some of the data integrations with your existing relational and non-relational data. Technologies we use NodeJS, React, Apollo, Postgres, Heroku, Docker, and Tensorflow: Fluttertable, Datapoint, Pandas, Keras, Numpy, Scipy, Scikit, SciPy, scikit-learn AI-ron at MERN Automation is a Google-backed NARability company that is committed to being the leading platform for accurately measuring and diagnosing agricultural production. We have built proprietary software and software automation systems to make it easy for any government to track agricultural production and prices. Our primary technologies today are:Tables, Entry Points into Global Food Production, Pipeline and Grain Market, and GenomicData.utility.Upstream is revolutionizing how U.S. agriculture is produced and sold. We are serving the agronomists, farmers, market researchers, and data scientists in the USA. WWrendy is on a mission to enable breakthrough scientific research in agricultural technology. We are growing cassava in our bioreactors, improving crop health and producing more durables than we need to feed ourselves. We use machine learning to identify cancer-causing microbes in seeds and in consumer durables to make food more accessible and more sustainable. We are building an entirely new technical infrastructure for agricultural diagnostics: tractors, biores, and tractors-all with a singular goal of improving yields and feed the world. We have humble beginnings as a grocery delivery service and have since grown to serve millions of people every year. We are a tight-knit team of MIT-trained researchers, technologists, and businesspeople, who have built an industry-leading technology platform that uses sensors and software to analyze crop production data to create intelligent and efficient food products. Our mission is to help all farmers have a more sustainable crop, by using microbial discoveries to improve the health and prosperity of their communities. We're a well-funded, well-funded, yuletide startup. We're looking to grow and affordably upgrade our tech stack every year. We use less expensive tools and techniques to analyze and build comprehensive datasets on crop production and food safety to help farmers grow more food and avoid over-production. Rails / React / AWS We build, manage, and cybersecurity insurance through an on-premise platform. We",[11,45860,45861],{},"From this first attempt there are already a few ideas for YC startups:",[210,45863,45864],{},[11,45865,45866],{},"We are helping to develop the first truly international distributed logistics network powered by an AI and a IoT. Our mission is to revolutionize how companies move containers, trucks, and other loads across borders. Our technology is being architected and being implemented by experts across the globe.",[210,45868,45869],{},[11,45870,45871],{},"We have built proprietary software and software automation systems to make it easy for any government to track agricultural production and prices.",[11,45873,45874,45875,45877],{},"Next we can try improving the output by using some additional features of ",[30,45876,45640],{},". One important setting is the size of the model. There are three released sizes of GPT-2:",[76,45879,45880,45886,45892,45898],{},[79,45881,45882,45885],{},[30,45883,45884],{},"124M"," (default): the \"small\" model, 500MB on disk. This is the one used on my first try",[79,45887,45888,45891],{},[30,45889,45890],{},"355M",": the \"medium\" model, 1.5GB on disk.",[79,45893,45894,45897],{},[30,45895,45896],{},"774M",": the \"large\" model, cannot currently be finetuned with Colaboratory but can be used to generate text from the pretrained model (see later in Notebook)",[79,45899,45900,45903,45904,45906],{},[30,45901,45902],{},"1558M",": the \"extra large\", true model. Will not work if a K80 GPU is attached to the notebook. (like ",[30,45905,45896],{},", it cannot be finetuned).",[11,45908,45909,45910,187,45913,45916],{},"We can try again using the 355M model. The ",[30,45911,45912],{},"large",[30,45914,45915],{},"extra large"," models won't work for our use case of finetuning the model on our sample text.",[11,45918,45919],{},"This would be a great time to plug my startup, but I don't have one. Instead, here are 10,000 startup ideas I generated with GPT-2 trained on the 335M model using the YC company descriptions for finetuning:",[459,45921,45923],{"className":24401,"code":45922,"language":24403,"meta":464,"style":464},"gpt2.generate_to_file(sess,\n                      destination_path=gen_file,\n                      length=150,\n                      prefix=\"we are building the world's first\",\n                      temperature=0.7,\n                      nsamples=10000,\n                      batch_size=20,\n                      )\n",[30,45924,45925,45930,45940,45952,45964,45975,45987,45998],{"__ignoreMap":464},[151,45926,45927],{"class":469,"line":470},[151,45928,45929],{"class":503},"gpt2.generate_to_file(sess,\n",[151,45931,45932,45935,45937],{"class":469,"line":488},[151,45933,45934],{"class":15210},"                      destination_path",[151,45936,1876],{"class":1869},[151,45938,45939],{"class":503},"gen_file,\n",[151,45941,45942,45945,45947,45950],{"class":469,"line":500},[151,45943,45944],{"class":15210},"                      length",[151,45946,1876],{"class":1869},[151,45948,45949],{"class":477},"150",[151,45951,9417],{"class":503},[151,45953,45954,45957,45959,45962],{"class":469,"line":509},[151,45955,45956],{"class":15210},"                      prefix",[151,45958,1876],{"class":1869},[151,45960,45961],{"class":481},"\"we are building the world's first\"",[151,45963,9417],{"class":503},[151,45965,45966,45969,45971,45973],{"class":469,"line":517},[151,45967,45968],{"class":15210},"                      temperature",[151,45970,1876],{"class":1869},[151,45972,16281],{"class":477},[151,45974,9417],{"class":503},[151,45976,45977,45980,45982,45985],{"class":469,"line":534},[151,45978,45979],{"class":15210},"                      nsamples",[151,45981,1876],{"class":1869},[151,45983,45984],{"class":477},"10000",[151,45986,9417],{"class":503},[151,45988,45989,45992,45994,45996],{"class":469,"line":1413},[151,45990,45991],{"class":15210},"                      batch_size",[151,45993,1876],{"class":1869},[151,45995,9097],{"class":477},[151,45997,9417],{"class":503},[151,45999,46000],{"class":469,"line":1418},[151,46001,46002],{"class":503},"                      )\n",[142,46004,46005],{},[46006,46007],"generated-ideas",{},[736,46009,46011],{"id":46010},"credits-links-and-learning-resources","Credits, Links and Learning Resources",[11,46013,46014,46015,46017],{},"Here's a link to Max Woolf's blog which has a lot of helpful resources on using GPT-2. Max is the author of ",[30,46016,45640],{}," which is the Python package used in the Google Colab (Max is the author of that Google Colab as well):",[11,46019,46020],{},[20,46021,46022],{"href":46022,"rel":46023},"https://minimaxir.com/",[24],[11,46025,46026],{},"Here are some resources for learning more about text generation. I came across Jay Alammar's blog which has a lot of great visualizations:",[11,46028,46029],{},[20,46030,46031],{"href":46031,"rel":46032},"https://jalammar.github.io/",[24],[11,46034,46035],{},"Jay also has a great YouTube channel:",[23881,46037],{"width":23883,"height":46038,"src":46039,"frameBorder":9181,"allow":46040,"allowFullScreen":609},400,"https://www.youtube.com/embed/MQnJZuBGmSQ","accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture",[11,46042,46043,46044],{},"Here's the link to the source markdown file for this article: ",[20,46045,46046],{"href":46046,"rel":46047},"https://github.com/briancaffey/briancaffey.github.io/tree/master/content/2021/01/16",[24],[11,46049,46050,46051],{},"Here's a link to the repo containing all of the scraped data and Jupyter notebooks used for exploring the data: ",[20,46052,46053],{"href":46053,"rel":46054},"https://gitlab.com/briancaffey/yc-waas-data",[24],[11,46056,46057,46058],{},"Here's the link to the Google Colab used for generating company descriptions with GPT-2 and gpt-2-simple: ",[20,46059,46060],{"href":46060,"rel":46061},"https://colab.research.google.com/drive/1u9b-FVGgUGcfifLy7bUXgoPe-pZ81rm3?usp=sharing#scrollTo=8DKMc0fiej4N",[24],[11,46063,46064],{},"Thank you for reading!",[589,46066,46067],{},"html pre.shiki code .s8-w5, html code.shiki .s8-w5{--shiki-default:#6A737D;--shiki-dark:#6A737D;--shiki-sepia:#88846F}html pre.shiki code .sq6CD, html code.shiki .sq6CD{--shiki-default:#D73A49;--shiki-default-font-style:inherit;--shiki-dark:#F97583;--shiki-dark-font-style:inherit;--shiki-sepia:#66D9EF;--shiki-sepia-font-style:italic}html pre.shiki code .sXSQT, html code.shiki .sXSQT{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#F8F8F2}html pre.shiki code .sC2Qs, html code.shiki .sC2Qs{--shiki-default:#D73A49;--shiki-dark:#F97583;--shiki-sepia:#F92672}html pre.shiki code .sMOD_, html code.shiki .sMOD_{--shiki-default:#24292E;--shiki-dark:#E1E4E8;--shiki-sepia:#F8F8F2}html pre.shiki code .srTi1, html code.shiki .srTi1{--shiki-default:#6F42C1;--shiki-dark:#B392F0;--shiki-sepia:#A6E22E}html pre.shiki code .sstjo, html code.shiki .sstjo{--shiki-default:#032F62;--shiki-dark:#9ECBFF;--shiki-sepia:#E6DB74}html pre.shiki code .s7F3e, html code.shiki .s7F3e{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#AE81FF}html pre.shiki code .s-m8C, html code.shiki .s-m8C{--shiki-default:#005CC5;--shiki-default-font-style:inherit;--shiki-dark:#79B8FF;--shiki-dark-font-style:inherit;--shiki-sepia:#66D9EF;--shiki-sepia-font-style:italic}html pre.shiki code .st05x, html code.shiki .st05x{--shiki-default:#B31D28;--shiki-default-font-style:italic;--shiki-dark:#FDAEB7;--shiki-dark-font-style:italic;--shiki-sepia:#F44747;--shiki-sepia-font-style:inherit}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html .sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html.sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html pre.shiki code .sTrkL, html code.shiki .sTrkL{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#66D9EF}html pre.shiki code .sCZoN, html code.shiki .sCZoN{--shiki-default:#032F62;--shiki-dark:#9ECBFF;--shiki-sepia:#CFCFC2}html pre.shiki code .sTHNf, html code.shiki .sTHNf{--shiki-default:#E36209;--shiki-default-font-style:inherit;--shiki-dark:#FFAB70;--shiki-dark-font-style:inherit;--shiki-sepia:#FD971F;--shiki-sepia-font-style:italic}html pre.shiki code .so59x, html code.shiki .so59x{--shiki-default:#24292E;--shiki-default-font-style:inherit;--shiki-dark:#E1E4E8;--shiki-dark-font-style:inherit;--shiki-sepia:#FD971F;--shiki-sepia-font-style:italic}",{"title":464,"searchDepth":488,"depth":488,"links":46069},[46070,46071,46072,46080],{"id":40791,"depth":488,"text":40792},{"id":41030,"depth":488,"text":41031},{"id":41408,"depth":488,"text":41409,"children":46073},[46074,46075,46076,46077,46078,46079],{"id":41415,"depth":500,"text":41416},{"id":44007,"depth":500,"text":44008},{"id":44426,"depth":500,"text":44427},{"id":44438,"depth":500,"text":44439},{"id":44954,"depth":500,"text":44955},{"id":45502,"depth":500,"text":45503},{"id":45514,"depth":488,"text":45515,"children":46081},[46082],{"id":46010,"depth":500,"text":46011},"2021-01-16",{},"/2021/01/16/i-scraped-analyzed-and-generated-yc-companies-founders-and-work-at-a-startup-job-postings",{"title":40771,"description":40771},"2021/01/16/i-scraped-analyzed-and-generated-yc-companies-founders-and-work-at-a-startup-job-postings",[12886,12355,46089,46090,46091],"scraping","startups","yc","3YzOLLFBH4EHv9fWOhHupoibeKE55cnfHQywRtzzs7g",{"id":46094,"title":46095,"body":46096,"comments":602,"date":46118,"description":46119,"draft":602,"extension":605,"external":606,"image":46120,"meta":46121,"navigation":609,"path":46122,"seo":46123,"stem":46124,"tags":46125,"__hash__":46126},"blog/2021/01/15/adding-internationalization-to-a-statically-generated-nuxt-site.md","Adding internationalization to a statically generated Nuxt site",{"type":8,"value":46097,"toc":46116},[46098,46101,46104,46110,46113],[11,46099,46100],{},"This article will give an overview of my experience adding internationalization to my statically-generated Nuxt website.",[11,46102,46103],{},"Since I migrated my site from Jekyll to Nuxt, I have had issues with my blog post URLs. My Jekyll site automatically generated URLs that included the date:",[459,46105,46108],{"className":46106,"code":46107,"language":997},[995],"https://my-site.com/2021/01/01/my-first-blog-post-of-2021\n",[30,46109,46107],{"__ignoreMap":464},[11,46111,46112],{},"To replicate this URL pattern with Nuxt, I have had to use some advanced features of the content API and Nuxt configuration options, as well as an awkward folder structure.",[11,46114,46115],{},"Because of this, I will be implementing internationalization in to phases: first I'll work on translating the core pages, next I'll work on setting up translations for individual blog posts. I don't plan on translating most of my blog posts, and in some cases I might only be providing translations for some of the locales that my site supports, falling back to the English version where a given locale is not available.",{"title":464,"searchDepth":488,"depth":488,"links":46117},[],"2021-01-15","This article will summarize my experience implementing internationalization (i18n) for a statically-generated Nuxt.js site","/static/emoji_flags.png",{},"/2021/01/15/adding-internationalization-to-a-statically-generated-nuxt-site",{"title":46095,"description":46119},"2021/01/15/adding-internationalization-to-a-statically-generated-nuxt-site",[12268,12646,11803],"L7tDGvkvzZOtHqEPv6j8UJ09lysVEeVJf1izeHiMSmY",{"id":46128,"title":46129,"body":46130,"comments":602,"date":46587,"description":46588,"draft":602,"extension":605,"external":606,"image":46589,"meta":46590,"navigation":609,"path":46591,"seo":46592,"stem":46593,"tags":46594,"__hash__":46596},"blog/2021/01/02/using-the-stripe-api-for-recurring-monthly-saas-subscription-payments-in-django-and-vue-application.md","Using Stripe for recurring monthly SaaS subscriptions in a Django + Vue application",{"type":8,"value":46131,"toc":46577},[46132,46136,46139,46142,46145,46153,46160,46164,46176,46181,46185,46188,46192,46312,46316,46319,46482,46486,46489,46520,46524],[14063,46133,46135],{"id":46134},"using-stripe-for-recurring-monthly-payments-to-a-paid-saas-subscription-in-a-django-vuejs-application","Using Stripe for recurring monthly payments to a paid SaaS subscription in a Django + Vue.js application",[11,46137,46138],{},"This diagram shows the flow of data for the lifecycle of a paid customer subscription in a Django application with a Vue.js client. There are four stages:",[11,46140,46141],{},"I. Account setup, configuration, model and object creation\nII. Logic and data flow for starting a customer's premium monthly subscription\nIII. Automatic subscription renewal\nIV. Cancelling a premium subscription",[56,46143,46144],{"id":39524},"Context",[11,46146,46147,46148,46152],{},"This is my first attempt at using Stripe, or any other online payment service API. Most of what I have diagramed here comes from this article from the Stripe documentation: ",[20,46149,46150],{"href":46150,"rel":46151},"https://stripe.com/docs/billing/subscriptions/fixed-price",[24],". Knowing almost nothing about what is needed to create a SaaS subscription, I found this article very helpful. It was a lot to read at once, but each call to to the Stripe API is very clear and straightforward.",[11,46154,46155,46156,13576],{},"I made some modifications and additions to this walk-through for my use case, which is an API service called Open SEC Data, an open source project that I'm working on (",[20,46157,46158],{"href":46158,"rel":46159},"https://gitlab.com/briancaffey/sec-filings-app",[24],[56,46161,46163],{"id":46162},"diagram","Diagram",[11,46165,46166,46167,46171,46172,643],{},"Here's a read-only link to the diagram: ",[20,46168,46169],{"href":46169,"rel":46170},"https://drive.google.com/file/d/1oH2b0W-c-dI5oXzc_jvCGvXx9sJagr4a/view?usp=sharing",[24],". This diagram is made with ",[20,46173,46174],{"href":46174,"rel":46175},"https://www.diagrams.net/",[24],[11,46177,46178],{},[2718,46179],{"alt":20386,"src":46180},"/static/django_vue_stripe_diagram.png",[56,46182,46184],{"id":46183},"legend","Legend",[11,46186,46187],{},"Here's a detailed description of each part of the diagram, starting with the first section.",[736,46189,46191],{"id":46190},"i-account-setup-configuration-model-and-object-creation","I. Account setup, configuration, model and object creation",[700,46193,46194,46201,46207,46220,46227,46237,46240,46255,46268,46281,46292,46298],{},[79,46195,46196,46197,46200],{},"Setup a Stripe account. For local development, make sure you turn on ",[30,46198,46199],{},"View test data",". On your local machine, install the stripe CLI and authenticate with your Stripe account",[79,46202,46203,46204,748],{},"Create a Product in Stripe (mine is called ",[30,46205,46206],{},"Open SEC Data Premium Subscription",[79,46208,46209,46210,46212,46213,46216,46217,643],{},"Create a Price in Stripe that references the Product.",[1205,46211],{},"Instead of creating these objects in the Stripe Dashboard, you can also create them with the Stripe CLI or the Python SDK. I created a Django management command called ",[30,46214,46215],{},"create_stripe_data"," that will create a Product and related Price in Stripe. We will need the id of the Price, it looks like this: ",[30,46218,46219],{},"price_1Hx0goL67dRDwyuDh9yEWsBo",[79,46221,46222,46223,46226],{},"Add the Price ID as an environment variable ",[30,46224,46225],{},"SUBSCRIPTION_PRICE_ID"," to the backend. This will be used later when we make API calls to Stripe from inside of Django views.",[79,46228,46229,46230,46232,46233,46236],{},"For production environments, you will need to a register a Stripe webhook. This is an endpoint in Django that Stripe will POST to in order to inform the Django application of events that have happened in Stripe.",[1205,46231],{},"For local development we need to run ",[30,46234,46235],{},"stripe listen --forward-to localhost/api/stripe-webhooks/"," in order to forward webhook events to the local Django application. This works really well for local development.",[79,46238,46239],{},"In both local and production environments we need to add an environment variables to the Django application that will be used to validate the webhook event.",[79,46241,46242,46243,46246,46247,46250,46251,46254],{},"You will need to create a ",[30,46244,46245],{},"Subscription"," model or similar in your Django models. This model should be related to your user model in some way. At a minimum it should have the ",[30,46248,46249],{},"subscription_id"," (the ID of the Stripe Subscription) and ",[30,46252,46253],{},"current_period_end"," (also from the Stripe Subscription object). We will use this model in the next sections.",[79,46256,46257,46260,46261,46264,46265,13576],{},[30,46258,46259],{},"/api/stripe-webhooks/"," is the endpoint in the Django application that Stripe will send POST requests to in order to inform the Django application of events that happen in Stripe. The URL can be called anything you want, as long as you register it with that URL. In local development, you need to specify this URL in the ",[30,46262,46263],{},"stipe listen"," command (for example, ",[30,46266,46267],{},"stripe listen forward-to localhost/api/stripe-webhooks/",[79,46269,46270,46273,46274,46277,46278,643],{},[30,46271,46272],{},"STRIPE_SECRET_KEY"," is the name of the secret API key that should only be accessible by the backend. In local development, this key looks like ",[30,46275,46276],{},"sk_test_Abc123",". In production, this key will look like ",[30,46279,46280],{},"sk_Abc123",[79,46282,46283,46286,46287,129,46289,13576],{},[30,46284,46285],{},"stripe"," is the name of the PyPI package that we need to add to ",[30,46288,38577],{},[30,46290,46291],{},"requirements/base.txt",[79,46293,46294,46297],{},[30,46295,46296],{},"STRIPE_PUBLISHABLE_KEY"," is the value of the Stripe API key that can be made public and is used in the Vue application to instantiate Stripe.",[79,46299,46300,46301,46304,46305],{},"The Stripe library is included in ",[30,46302,46303],{},"index.html"," via CDN so that it is accessible anywhere in the Vue application. Stripe object is instantiated in the Vue application with:",[210,46306,46307],{},[11,46308,46309],{},[30,46310,46311],{},"let stripe = Stripe(process.env.STRIPE_PUBLISHABLE_KEY)",[736,46313,46315],{"id":46314},"ii-logic-and-data-flow-for-starting-a-customers-premium-monthly-subscription","II. Logic and data flow for starting a customer's premium monthly subscription",[11,46317,46318],{},"With everything setup and configured properly in Stripe, the backend Django application and the frontend Vue application, customers can now start paying for monthly subscriptions. In my application, a user can sign up for an account first without having a premium subscription. In other scenarios, having an active account may require a premium subscription.",[700,46320,46321,46341,46364,46370,46382,46391,46398,46401,46404,46407,46427,46436,46441,46446,46455,46458,46461,46464,46479],{"start":2497},[79,46322,46323,46324,46327,46328,46331,46332,46337,46338,643],{},"When a logged-in user visits their ",[30,46325,46326],{},"/account"," page, they will see the status of their account: Basic (free) or Premium (paid subscription). Users on a Basic plan will see the option to upgrade to Premium. They will be redirected to a ",[30,46329,46330],{},"/premium"," page where they will be presented with a credit card form. This credit card form is generated by ",[20,46333,46336],{"href":46334,"rel":46335},"https://stripe.com/payments/elements",[24],"Stripe Elements",". The user fills out their credit card, expiration date, card security code and billing ZIP code and then clicks ",[30,46339,46340],{},"Purchase",[79,46342,46343,46344,46346,46347,46350,46351,26802,46354,46357,46358,46360,46361,643],{},"Clicking on ",[30,46345,46340],{}," calls a method ",[30,46348,46349],{},"purchase"," that calls ",[30,46352,46353],{},"stripe.CreatePaymentMethod",[30,46355,46356],{},"paymentMethodId"," token returned from ",[30,46359,46353],{}," is then passed to the method called ",[30,46362,46363],{},"createSubscription",[79,46365,46366,46367,46369],{},"Stripe creates this object and returns a response that contains a ",[30,46368,46356],{}," token.",[79,46371,46372,46374,46375,46378,46379,46381],{},[30,46373,46363],{}," sends a POST request to ",[30,46376,46377],{},"/api/stripe/create-subscription/"," in the Django application with the ",[30,46380,46356],{}," that we generated in the previous step.",[79,46383,46384,46386,46387,46390],{},[30,46385,46377],{}," calls a view called ",[30,46388,46389],{},"create_subscription"," which makes a number of API calls to Stripe and then finally saves some data in the application's Postgres database.",[79,46392,46393,46394,46397],{},"The first API call creates the Customer object in Stripe if it does not exist. ",[30,46395,46396],{},"email=request.user.email"," is used in the API call to associate the Stripe customer with the user's email.",[79,46399,46400],{},"Next the payment method is attached to Stripe Customer model.",[79,46402,46403],{},"Next the Stripe payment method is set as the default payment method for Stripe customer for future billing.",[79,46405,46406],{},"The Stripe subscription model is created with the customer ID that was created in the earlier and the price ID corresponding to the premium subscription (added in the setup stage).",[79,46408,46409,46410,106,46413,187,46416,46419,46420,46422,46423,46426],{},"Once these Stripe API calls have finished, a new Subscription is saved in the Postgres database. ",[30,46411,46412],{},"stripe_subscription_id",[30,46414,46415],{},"stripe_customer_id",[30,46417,46418],{},"valid_through"," (DateTimeField that keeps track of the date through which the user's subscription has been paid for) are saved to the ",[30,46421,46245],{}," model and then the subscription model is saved to the user model's ",[30,46424,46425],{},"subscription"," field.",[79,46428,46429,46430,46432,46433],{},"When the ",[30,46431,46363],{}," method's POST requests returns successfully, the user's account is fetched again from ",[30,46434,46435],{},"/api/account/",[79,46437,46438,46439,643],{},"The browser makes a request to ",[30,46440,46435],{},[79,46442,46443,46445],{},[30,46444,46435],{}," returns information on the user and their subscription.",[79,46447,46448,46449,46451,46452,46454],{},"Data from ",[30,46450,46435],{}," is updated in Vuex ",[30,46453,8387],{}," store.",[79,46456,46457],{},"The user is now able to make requests to resources for premium features.",[79,46459,46460],{},"In this application, one such example is the ability to request an API key for making for making API calls.",[79,46462,46463],{},"A user makes a request to an API endpoint for a premium feature.",[79,46465,46466,46467,313,46470,46473,46474,46476,46477,643],{},"When determining permissions for resources that should only be accessible to customers with valid subscriptions, we need to compare ",[30,46468,46469],{},"request.user.subscription.valid_through",[30,46471,46472],{},"timezone.now()"," and make sure that ",[30,46475,46418],{}," is greater than ",[30,46478,46472],{},[79,46480,46481],{},"Requests for protected resources are successfully returned to the browser.",[736,46483,46485],{"id":46484},"iii-automatic-subscription-renewal","III. Automatic subscription renewal",[11,46487,46488],{},"The customer's credit card is charged once each month that they are subscribed to the service. This action happens in Stripe. This section assumes that the customer's primary payment method is still valid (it has not been canceled expired or not able to be charged for some other reason).",[700,46490,46491,46494,46505],{"start":3820},[79,46492,46493],{},"The customer's card is charged in Stripe and an event is sent to the Django backend via a webhook that we registered in the setup stage.",[79,46495,46496,46497,46500,46501,46504],{},"The webhook view checks ",[30,46498,46499],{},"event.type"," and if the event is of type ",[30,46502,46503],{},"invoice.paid"," we extend the user's subscription by one month.",[79,46506,46507,46508,46511,46512,46514,46515,46517,46518,643],{},"To extend the user's subscription, we modify the ",[30,46509,46510],{},"DateTimeField"," field on the ",[30,46513,46245],{}," that tracks the ",[30,46516,46253],{}," which is included in the webhook data object. The model field in my code is called ",[30,46519,46418],{},[736,46521,46523],{"id":46522},"iv-cancelling-a-premium-subscription","IV. Cancelling a premium subscription",[700,46525,46526,46532,46558,46565,46571],{"start":7211},[79,46527,46528,46529,36701],{},"When a user decides to cancel their payed subscription service, they click on the ",[30,46530,46531],{},"Cancel My Subscription",[79,46533,46534,46535,46538,46539,46542,46543,46546,46547,46550,46551,46554,46555,46557],{},"This makes a POST request to ",[30,46536,46537],{},"/api/stripe/cancel-subscription"," which calls the ",[30,46540,46541],{},"cancel_subscription"," view. This view calls ",[30,46544,46545],{},"stripe.Subscription.delete(subscriptionId)",", where the ",[30,46548,46549],{},"subscriptionId"," is retrieved from ",[30,46552,46553],{},"request.user.subscription"," (the ",[30,46556,46245],{}," model created in the setup section).",[79,46559,46560,46561,46564],{},"The subscription is deleted in Stripe through the ",[30,46562,46563],{},"stripe.Subscription.delete"," API call.",[79,46566,46567,46568,643],{},"The user's subscription is deleted from the user model with ",[30,46569,46570],{},"request.user.subscription.delete()",[79,46572,46573,46574,46576],{},"The frontend responds to the deleted subscription by fetching ",[30,46575,46435],{}," again, refresh, or redirecting and the user no longer has access to their premium subscription.",{"title":464,"searchDepth":488,"depth":488,"links":46578},[46579,46580,46581],{"id":39524,"depth":488,"text":46144},{"id":46162,"depth":488,"text":46163},{"id":46183,"depth":488,"text":46184,"children":46582},[46583,46584,46585,46586],{"id":46190,"depth":500,"text":46191},{"id":46314,"depth":500,"text":46315},{"id":46484,"depth":500,"text":46485},{"id":46522,"depth":500,"text":46523},"2021-01-02","This article shares my experience learning and implementing the Stripe API for recurring monthly SaaS subscription payments in an application using Django and Vue.js","/static/django_vue_stripe.png",{},"/2021/01/02/using-the-stripe-api-for-recurring-monthly-saas-subscription-payments-in-django-and-vue-application",{"title":46129,"description":46588},"2021/01/02/using-the-stripe-api-for-recurring-monthly-saas-subscription-payments-in-django-and-vue-application",[30122,12646,46595,46285],"drf","p5NxbOdUNY3TgcpBF747knZCKNoT9TntABqHUWs-eZ8",{"id":46598,"title":46599,"body":46600,"comments":602,"date":47199,"description":47200,"draft":602,"extension":605,"external":606,"image":47201,"meta":47202,"navigation":609,"path":47203,"seo":47204,"stem":47205,"tags":47206,"__hash__":47207},"blog/2021/01/01/session-authentication-with-django-django-rest-framework-and-nuxt.md","Session Authentication with Django, Django REST Framework and Nuxt",{"type":8,"value":46601,"toc":47174},[46602,46605,46608,46614,46617,46620,46623,46626,46628,46631,46634,46638,46641,46645,46648,46662,46665,46669,46672,46696,46700,46703,46706,46710,46713,46716,46719,46721,46727,46730,46738,46741,46758,46761,46775,46789,46874,46880,46912,46941,46967,46973,47004,47014,47024,47030,47111,47115,47119,47122,47126,47133,47136,47140,47143,47146,47150,47153,47157,47160,47164,47167,47171],[11,46603,46604],{},"This will be a continuation of the discussion about how data flows in Django + Nuxt applications, looking specifically at session authentication.",[11,46606,46607],{},"Here's a GitLab repo that where you can find the source code and other diagrams related to this project:",[11,46609,46610],{},[20,46611,46612],{"href":46612,"rel":46613},"https://gitlab.com/briancaffey/django-nuxt-starter",[24],[11,46615,46616],{},"This diagram focuses on the interactions between:",[11,46618,46619],{},"I. The browser",[11,46621,46622],{},"II. The Nuxt server (Node process)",[11,46624,46625],{},"III. The Django backend API server (gunicorn process)",[56,46627,46144],{"id":39524},[11,46629,46630],{},"For illustration purposes, I'm using a simple CRUD application that has two models: Users and (blog) Posts. Users can log in with email and password credentials and create, read, update and delete blog posts (CRUD). Currently I'm only doing the R (read) of CRUD: listing and viewing blog posts. Creating, updating and delete will be added later. For now, users must be logged in to see posts.",[11,46632,46633],{},"I'm still learning a lot about Nuxt and how it can be used with Django and Django REST Framework. This project is an effort at documenting my learning process, learning in public and learning from mistakes, so any feedback or guidance on what I have written here would be highly appreciated!",[56,46635,46637],{"id":46636},"why-nuxt","Why Nuxt?",[11,46639,46640],{},"Using Nuxt (with Server Side Rendering, or SSR) is one of many ways to use Vue.js with Django. Vue is a progressive framework, which means that it can be gradually adopted into a project--you don't have to go all-in on the framework or rewrite the application from scratch to fit with how Vue works.",[736,46642,46644],{"id":46643},"different-ways-to-use-vue-with-django","Different ways to use Vue with Django",[11,46646,46647],{},"In terms of Django, here are some ways that you can use Vue:",[76,46649,46650,46653,46656,46659],{},[79,46651,46652],{},"Vue as a jQuery replacement for adding basic interactivity in views served by Django templates",[79,46654,46655],{},"Build a static Vue application and serve it as a set of static assets in a Django project alongside routes that are served by other normal Django templates views.",[79,46657,46658],{},"Build a Vue SPA which consumes a Django API (usually built with Django REST Framework or similar), and serve it over a content delivery network (CDN).",[79,46660,46661],{},"Use Vue to build an Electron desktop app that uses Django as an API",[11,46663,46664],{},"In these scenarios, Vue is served as either static assets (such as in the case of serving a SPA over a CDN), or Vue code is included in an HTML response from a server (where the view library, not your application, is served over a CDN), similar to how jQuery is used.",[736,46666,46668],{"id":46667},"different-ways-to-use-nuxt","Different ways to use Nuxt",[11,46670,46671],{},"Nuxt is a Framework that can be used in a few different ways, I'll briefly discus three ways in which Nuxt can be used. Common to all three of these ways of using Nuxt is the directory structure. No matter how you use Nuxt, it provides a great way to organize Vue code.",[700,46673,46674,46680,46683],{},[79,46675,46676,46677,13576],{},"Static mode: this mode allows you to write Vue code which is built into a static HTML, and then that HTML is deployed to a CDN or webserver like NGINX. The developer (or CI/CD process) runs a command to generate HTML files for each page in the application, and these pages are served as-is when accessed by a user. I recently migrated my personal blog from Jekyll to Nuxt with full-static mode. Check it out at (",[20,46678,662],{"href":19426,"rel":46679},[24],[79,46681,46682],{},"SPA mode: This is similar to what you might use if you started a Vue project with Vue CLI. The project is also generated as in Static Mode, but what is generated is primarily Javascript code that is executed on the browser.",[79,46684,46685,46686,46689,46690,46692,46693,46695],{},"SSR mode: Server Side Rendering is the mode that I'll be focusing on here. Unlike the other ways of using Vue that have already been discussed, this mode involves a Node.js server that will handle our requests. For example, a web request for ",[30,46687,46688],{},"/posts"," is sent to our Nuxt Server (a Node.js server process) and Node.js is responsible for returning HTML that contains all of the blog Posts that we want to show (or a paginated selection of all blog posts, which is how my example blog app is built). So the Nuxt app has to make a request to our Django API server before returning fully rendered HTML page for the ",[30,46691,46688],{}," page. The user then gets the page from Nuxt, reads all of the blog posts and then decides to check out the blog posts on the second page of posts. When the user clicks on page 2, we request the second page of data from our Django API directly, not from Nuxt. The user then sees a short loading animation followed by the second page of blog posts that are loaded in using AJAX (usually with ",[30,46694,19943],{}," or axios).",[736,46697,46699],{"id":46698},"nuxt-benefits","Nuxt Benefits",[11,46701,46702],{},"The main reason for using Nuxt is to render the first page loads on the server, returning a complete HTML response that can be beneficial for SEO, social sharing, and other scenarios where you need control over how a website's pages are delivered (specifically, the initial request made to the server).",[11,46704,46705],{},"This type of control is not possible for applications that serve Vue over CDN since they can only request backend API data once the JS client has been requested from a CDN.",[736,46707,46709],{"id":46708},"nuxt-downsides-and-tradeoffs","Nuxt Downsides and Tradeoffs",[11,46711,46712],{},"Using Nuxt for SSR introduces quite a bit of complexity in both the application deployment and our Vue code. The backend API won't have to change at all when moving to Nuxt from a static Vue SPA.",[11,46714,46715],{},"Django alone is capable of returning fully generated, SEO-optimized HTML for each request, but applications built with Vue and Django templates may be difficult to work on as the project grows larger and larger. The Django/DRF + Nuxt approach may be more appropriate for projects with dedicated backend and frontend teams.",[11,46717,46718],{},"One other potential downside is added latency because of the \"double request\". If the Nuxt server and the Django server are on the same machine, then this latency will probably be a non-issue.",[56,46720,46163],{"id":46162},[11,46722,46723],{},[2718,46724],{"alt":46725,"src":46726},"Nuxt Django Auth","/static/django_nuxt_auth.png",[11,46728,46729],{},"This diagram looks at session authentication with a focus on the browser, the Nuxt server and the Django server. It looks at two simple user stories, ordered from top to bottom in the diagram.",[11,46731,46732,46733,46735,46736,21027],{},"I. An existing application user visits the site in a new browser, navigates to the Login page, logs in with credentials and then visits a protected page: ",[30,46734,46688],{},".\nII. The user closes the browser and then comes back directly to the ",[30,46737,46688],{},[11,46739,46740],{},"These two user stories sound simple, but they touch on a lot of the features of Nuxt that make it powerful, and complicated at first (for Vue users). These include:",[76,46742,46743,46747,46752,46755],{},[79,46744,46745],{},[30,46746,19808],{},[79,46748,46749],{},[30,46750,46751],{},"nuxtServerInit",[79,46753,46754],{},"Vuex on the client and server",[79,46756,46757],{},"Custom plugin for axios",[11,46759,46760],{},"Some important parts of Nuxt that this diagram does not (yet) touch on are:",[76,46762,46763,46766,46772],{},[79,46764,46765],{},"Nuxt auth module (I don't know if this is relevant for my use case)",[79,46767,46768,46769,46771],{},"Nuxt fetch property (different from the ",[30,46770,19943],{}," web API)",[79,46773,46774],{},"Nuxt middleware (I'm also not sure if this would be helpful for anything I am doing in this example project)",[736,46776,46778,46779,46781,46782,46785,46786,46788],{"id":46777},"user-story-i-a-user-tries-to-open-posts-is-redirected-to-login-logs-in-then-navigate-to-posts-and-sees-blog-posts","User story I.: A user tries to open ",[30,46780,46688],{},", is redirected to ",[30,46783,46784],{},"/login",", logs in, then navigate to ",[30,46787,46688],{}," and sees blog posts",[700,46790,46791,46798,46813,46828,46837,46840,46843,46863,46866],{},[79,46792,46793,46794,46797],{},"User navigates to ",[30,46795,46796],{},"http://domain.com/",". This request is handled by the Nuxt server.",[79,46799,19225,46800,46802,46803,46808,46809,46812],{},[30,46801,46751],{}," action is called (",[20,46804,46807],{"href":46805,"rel":46806},"https://nuxtjs.org/docs/2.x/directory-structure/store#the-nuxtserverinit-action",[24],"read more on nuxtServerInit","). This is a special Vuex action that, if defined in ",[30,46810,46811],{},"store/index.js",", will be called once per request to the Nuxt Server (when a page is initially visited or refreshed in the browser).",[79,46814,46815,46817,46818,46820,46821,46824,46825,46827],{},[30,46816,46751],{}," dispatches a Vuex action in the ",[30,46819,8387],{}," module called ",[30,46822,46823],{},"fetchData",". This action makes an GET request to ",[30,46826,46435],{}," in the Django application.",[79,46829,46830,46831,46833,46834,748],{},"An API call to ",[30,46832,46435],{}," is made to the Django backend directly from the Nuxt container over the docker network (",[30,46835,46836],{},"backend:8000",[79,46838,46839],{},"If the request is made by an anonymous user (no user is logged in), a 403 response is returned to the Nuxt server and no account data is set in the Vuex store (on the server).",[79,46841,46842],{},"Since the user is currently not logged in, the request returns a 403 response.",[79,46844,46845,46848,46849,46851,46852,46855,46856,46858,46859,46862],{},[30,46846,46847],{},"authMiddleware"," (on the Nuxt server) redirects the user to ",[30,46850,46784],{}," based on the value of ",[30,46853,46854],{},"authenticated"," in the Vuex store. The Original request for ",[30,46857,46688],{}," returns a fully-rendered ",[30,46860,46861],{},"/login/"," page instead.",[79,46864,46865],{},"User is now on the Login page",[79,46867,19225,46868,46870,46871,643],{},[30,46869,2953],{}," hook for the Login page makes a GET request to ",[30,46872,46873],{},"/api/login-set-cookie/",[11,46875,46876,46877,643],{},"10, 11. This endpoint calls a simple view that is decorated with ",[30,46878,46879],{},"@ensure_csrf_token",[700,46881,46882,46889,46907],{"start":2489},[79,46883,46884,46885,46888],{},"When the response returns to the browser, the ",[30,46886,46887],{},"csrftoken"," is set in the browser.",[79,46890,46891,46892,46895,46896,46898,46899,46902,46903,46906],{},"The $apiCall function is defined in ",[30,46893,46894],{},"plugins/axios.js",", and it adds the ",[30,46897,46887],{}," cookie to the ",[30,46900,46901],{},"X-CSRFToken"," header of API requests. This is important for POST request where the CSRF token is required. When the user fills out their email and password in the login form, the $apiCall function is called with ",[30,46904,46905],{},"/api/login/"," and the email/password as credentials.",[79,46908,46909,46910,643],{},"The email and password are sent as data in the POST request to ",[30,46911,46905],{},[11,46913,46914,46915,46917,46918,46921,46922,6208,46925,187,46928,18952,46931,46933,46934,46936,46937,46940],{},"15, 16. The ",[30,46916,46905],{}," URL calls the ",[30,46919,46920],{},"login_view"," which makes use of two functions from ",[30,46923,46924],{},"django.contrib.auth",[30,46926,46927],{},"authenticate",[30,46929,46930],{},"login",[30,46932,46927],{}," gets a user from the provided email/password, and the ",[30,46935,46930],{}," function sets an HttpOnly ",[30,46938,46939],{},"sessionid"," session cookie on the response.",[700,46942,46943,46952,46962],{"start":3167},[79,46944,46945,46946,46948,46949,46951],{},"The HttpOnly ",[30,46947,46939],{}," cookie is automatically set on the browser when the ",[30,46950,46905],{}," request returns successfully.",[79,46953,46954,46955,46957,46958,46961],{},"When this ",[30,46956,46905],{}," request returns successfully, a value in the ",[30,46959,46960],{},"auth"," Vuex module is set to keep track of the current user's logged in state.",[79,46963,46964,46965,643],{},"Next, a GET request is made to ",[30,46966,46435],{},[11,46968,46969,46970,46972],{},"20, 21. Since the ",[30,46971,46939],{}," cookie is set and sent along with the request automatically, this request will succeed.",[700,46974,46975,46983,46989,46998],{"start":3729},[79,46976,46429,46977,46979,46980,46982],{},[30,46978,46435],{}," request returns, the user's account information is saved to the ",[30,46981,8387],{}," Vuex module. At this point, the client may redirect automatically to the home page, or user account page, dashboard, etc.",[79,46984,46985,46986,46988],{},"Now logged in, the user navigates (again via Vue router) to ",[30,46987,46688],{},", a page that shows a paginated view of all blog posts.",[79,46990,46991,46992,46994,46995,643],{},"This page has an ",[30,46993,19808],{}," method which is called when the page component is created and it dispatches a Vuex action ",[30,46996,46997],{},"posts/fetchData",[79,46999,47000,47001,643],{},"This Vuex action makes a GET request to ",[30,47002,47003],{},"/api/posts/",[11,47005,47006,47007,47009,47010,47013],{},"26, 27. ",[30,47008,47003],{}," uses a ",[30,47011,47012],{},"ModelViewSet"," and returns a paginated list of blog posts",[700,47015,47016],{"start":3782},[79,47017,46429,47018,47020,47021,47023],{},[30,47019,47003],{}," request returns successfully, the blog post data is saved to the ",[30,47022,1825],{}," Vuex module.",[736,47025,47027,47028],{"id":47026},"user-story-ii-logged-in-user-opens-new-browser-window-and-revisits-posts","User story II.: Logged in user opens new browser window and revisits ",[30,47029,46688],{},[700,47031,47032,47037,47042,47050,47064,47072,47079,47086,47098,47108],{"start":3791},[79,47033,47034,47035,643],{},"The user closes their browser and then opens a new browser window and navigates to ",[30,47036,46688],{},[79,47038,47039,47041],{},[30,47040,46751],{}," is called as usual,",[79,47043,19225,47044,47047,47048,643],{},[30,47045,47046],{},"user/fetchData"," action is called. This action makes a GET request to ",[30,47049,46435],{},[79,47051,19225,47052,47054,47055,47057,47058,47060,47061,47063],{},[30,47053,46435],{}," request returns successfully. The ",[30,47056,46939],{}," cookie is passed along from the browser to the API request that is made from the Nuxt server to the backend API (",[30,47059,46435],{},").  User account data is then set on the Vuex ",[30,47062,8387],{}," module.",[79,47065,19225,47066,47068,47069,47071],{},[30,47067,19808],{}," method for the ",[30,47070,46688],{}," pages is called.",[79,47073,47074,47076,47077],{},[30,47075,19808],{}," dispatches a Vuex action ",[30,47078,46997],{},[79,47080,47081,47083,47084,643],{},[30,47082,46997],{}," makes an API request to ",[30,47085,47003],{},[79,47087,19225,47088,47090,47091,47093,47094,47097],{},[30,47089,47003],{}," request is handled by a ",[30,47092,47012],{}," for the ",[30,47095,47096],{},"Post"," model that gets blog posts and then sets them to the Vuex store (on the server) when the request returns a response (to the Nuxt server).",[79,47099,47100,47101,187,47103,47093,47105,47107],{},"Once the async data fetching is compete (",[30,47102,46751],{},[30,47104,19808],{},[30,47106,46688],{}," page), the page HTML is rendered using the Vuex store data stored on the server. The Vuex data is sent back with the rendered HTML (I think this is how it works).",[79,47109,47110],{},"Finally, the user sees the list of blog posts. The page is loaded \"at once\"; there is no waiting for data to load after loading the page initially.",[56,47112,47114],{"id":47113},"discussion","Discussion",[736,47116,47118],{"id":47117},"complexity","Complexity",[11,47120,47121],{},"Is this authentication process overly complicated? When I make these diagrams, I try to make simple concept as detailed as possible, but there are a lot of distinct actions being taken in many different parts of the application and getting them all into one diagram was tricky.",[736,47123,47125],{"id":47124},"httponly-session-cookies","HttpOnly Session Cookies",[11,47127,47128,47129,47132],{},"Session authentication is the officially recommended way to do authentication with Django REST Framework for clients that run in the browser. However, there seem to be lots of people using JWT with DRF and Javascript clients that run in the browser. The main argument against doing this is that the JWT must be stored in a Javascript-accessible store (localStorage or Cookies) so it can be passed with each request. Many people are also interested in trying to store JWT for authentication in HttpOnly cookies to harden client-side security. I'm very curious to know if anyone is actually doing this, and what the implementation looks like. While ",[30,47130,47131],{},"djangorestframework_simplejwt"," doesn't support HttpOnly, there seems to be lots of interest in doing this. I think it might be possible with a special middleware, so let me know if anyone is interested in proof-of-concept/diagram for that.",[11,47134,47135],{},"Some use cases for JWT and other token authentication methods with DRF might include native mobile apps or Desktop apps. For most cases, I think session authentication with Django's built in session cookies for DRF authentication is the best option. JWTs also have no clear solution for logging out, which may be important for some security considerations. The concept of stateless authentication is interesting, but for most use cases I would argue that it is not worth doing. Let me know if anyone has thoughts on this, I'm curious to see what everyone thinks.",[736,47137,47139],{"id":47138},"next-steps","Next Steps",[11,47141,47142],{},"My next steps for this project/repo are to deploy this to a production environment as soon as I have time to do so. My local setup has been working well, and I think it should work well for a simple DigitalOcean docker swarm deployment like I have done with other Django + Vue projects.",[11,47144,47145],{},"I also want to add the create, update and delete functionality for posts, improve error handling with API calls, add form validation, and maybe write some tests with Jest.",[56,47147,47149],{"id":47148},"questions","Questions",[11,47151,47152],{},"Here are some questions and areas that I still need to investigate.",[736,47154,47156],{"id":47155},"nuxt-composition-api","Nuxt Composition API",[11,47158,47159],{},"I have seen that there is a Composition API module for Nuxt. I have only just now started looking at Composition API examples and documentation for \"vanilla\" Vue, but I have heard that the Nuxt Composition API module has some additional features specifically for use with Nuxt, so I'm curious to learn what these are.",[736,47161,47163],{"id":47162},"nuxt-v3-and-vue-3","Nuxt v3 and Vue 3",[11,47165,47166],{},"Nuxt looks like it has plans to support Vue 3, so I am interested to learn more about Vue 3 as it is adopted by Vue frameworks such as Nuxt and Quasar.",[736,47168,47170],{"id":47169},"nuxts-fetch-method-server-middleware-nuxt-auth-module","Nuxt's fetch method, server middleware, Nuxt auth module",[11,47172,47173],{},"I think I am using server middleware correctly, it can be improved by redirecting to the initial requested route after successful login. I'm not sure if I should use the Nuxt auth module in this application, I have read that it doesn't support HttpOnly cookie use cases, but I could be wrong.",{"title":464,"searchDepth":488,"depth":488,"links":47175},[47176,47177,47183,47189,47194],{"id":39524,"depth":488,"text":46144},{"id":46636,"depth":488,"text":46637,"children":47178},[47179,47180,47181,47182],{"id":46643,"depth":500,"text":46644},{"id":46667,"depth":500,"text":46668},{"id":46698,"depth":500,"text":46699},{"id":46708,"depth":500,"text":46709},{"id":46162,"depth":488,"text":46163,"children":47184},[47185,47187],{"id":46777,"depth":500,"text":47186},"User story I.: A user tries to open /posts, is redirected to /login, logs in, then navigate to /posts and sees blog posts",{"id":47026,"depth":500,"text":47188},"User story II.: Logged in user opens new browser window and revisits /posts",{"id":47113,"depth":488,"text":47114,"children":47190},[47191,47192,47193],{"id":47117,"depth":500,"text":47118},{"id":47124,"depth":500,"text":47125},{"id":47138,"depth":500,"text":47139},{"id":47148,"depth":488,"text":47149,"children":47195},[47196,47197,47198],{"id":47155,"depth":500,"text":47156},{"id":47162,"depth":500,"text":47163},{"id":47169,"depth":500,"text":47170},"2021-01-01","This article shows how to use session authentication with Django + Nuxt.js applications","/static/django_nuxt_auth_og.png",{},"/2021/01/01/session-authentication-with-django-django-rest-framework-and-nuxt",{"title":46599,"description":47200},"2021/01/01/session-authentication-with-django-django-rest-framework-and-nuxt",[30122,12646,11803,46595,36041],"QNhKMyUFZxiHwo7hdnfz0PzsZ9ICARDle0xhC4U0Duc",{"id":47209,"title":47210,"body":47211,"comments":602,"date":47525,"description":47526,"draft":602,"extension":605,"external":606,"image":47247,"meta":47527,"navigation":609,"path":47528,"seo":47529,"stem":47530,"tags":47531,"__hash__":47532},"blog/2020/12/27/building-web-applications-with-django-drf-and-nuxt.md","Building web applications with Django, Django REST Framework, Nuxt.js and docker",{"type":8,"value":47212,"toc":47518},[47213,47216,47234,47240,47243,47248,47252,47255,47268,47271,47274,47277,47283,47286,47289,47293,47310,47320,47467,47469,47472,47475,47479,47499,47502,47505,47512,47515],[11,47214,47215],{},"Over the holidays between lots of big meals and many naps, I tried to tackle one more goal of mine before this year come to an end: building an application with Django and Nuxt.js.",[11,47217,47218,47219,47223,47224,47227,47228,47233],{},"This year I rebuilt my personal blog (",[20,47220,662],{"href":47221,"rel":47222},"https://briancaffey.github.io/",[24],") with Nuxt.js, the ",[30,47225,47226],{},"@nuxt/content"," headless git-based CMS and TailwindCSS. It is statically generated with Nuxt's full-static mode and has been really enjoyable to work with. I have also learned a lot more about SEO and how Nuxt helps improve Vue applications' SEO. I have also been working a lot with Django and Vue.js applications where Django serves as an API to a Vue.js SPA. This combination of technologies works well for a lot of use cases, but it falls short in SEO. Nuxt also provides a great way to organize large Vue.js projects which I have been finding very helpful. For these reasons, combining Django and Nuxt has been something that I have wanted to try for a while, so this article will share some of my experiences in recent efforts to build with these two frameworks. I took ",[20,47229,47232],{"href":47230,"rel":47231},"https://gitlab.com/briancaffey/django-nuxt-starter/-/blob/develop/STEP_BY_STEP.md",[24],"detailed notes of each step of the project setup"," starting from an empty repository, and I put together a diagram of my understanding of how data flows in the application.",[11,47235,47236,47237],{},"Here's the link to the project repository that I'll be referencing: ",[20,47238,46612],{"href":46612,"rel":47239},[24],[11,47241,47242],{},"This article will focus on explaining the project through the diagram shown below. I added two types of labels: letters and numbers. The letters will introduce each component of the application and its role in the application as a whole. The numbers summarize how data flows through the different components in my sample blog application.",[11,47244,47245],{},[2718,47246],{"alt":46163,"src":47247},"/static/django_nuxt_app_diagram.png",[56,47249,47251],{"id":47250},"diagram-components","Diagram components",[11,47253,47254],{},"A. Your computer - Possibly also your development machine which is running the application in docker containers with docker-compose.",[11,47256,47257,47258,106,47261,106,47264,47267],{},"B. NGINX - This is the \"front desk\" of the application that does a few different things. It is the first component that web requests come to. It serves as a reverse proxy which does path-based routing. It looks at the URL request and determines where to send it. For example: ",[30,47259,47260],{},"/api/posts/1",[30,47262,47263],{},"/dashboard/",[30,47265,47266],{},"/admin/"," could all be routed differently depending on the NGINX configuration file. We will look at this again in the next section. This  component, like most of the other things in the diagram, runs in a container. NGINX can also serve static files for our Django app and do TLS termination to make our application available over a secure HTTPS connection.",[11,47269,47270],{},"C. Nuxt.JS server - The first \"S\" in SSR (server side rendering). It is a Node.js process that renders HTML from Vue components that we define in our Nuxt app, as well as data fetched from other servers/APIs before returning HTML back to the client.",[11,47272,47273],{},"D. Django server - This runs the WSGI application with a gunicorn process in a container.",[11,47275,47276],{},"E. Django REST Framework is a Django package the facilitates the creation of REST API endpoints. This is part of the Django application, it primarily takes care of data serialization (which can be thought of as translating between JSON and Python objects that represent rows of data in our Postgres database)",[11,47278,47279,47280,643],{},"F. This is the Postgres database, also a containerized service. It is on the same docker network as the Django/gunicorn application, so the Django application can connect to the Postgres database using the hostname ",[30,47281,47282],{},"postgres",[11,47284,47285],{},"G. docker-compose is used to orchestrate the docker network, containers and volumes that make up the application.",[11,47287,47288],{},"H. This box represents the docker network that allows for easy networking between services. We will come back to this the significance of this in the next section.",[56,47290,47292],{"id":47291},"data-flow-in-the-application","Data flow in the application",[11,47294,47295,47296,47299,47300,47302,47303,47306,47307,643],{},"The simple application I have built for this demonstration is a blog. There is only a list view and a detail view for simple blog post model with three fields: title, body and created date. For the list view, the frontend (Nuxt) route is ",[30,47297,47298],{},"/posts/"," and the backend route is ",[30,47301,47003],{}," for the detail view the frontend route is ",[30,47304,47305],{},"/posts/_id"," and the API route is ",[30,47308,47309],{},"/api/posts/_id/",[11,47311,47312,47313,47316,47317,13576],{},"The data flow shown here will walk through what happens when a user visits ",[30,47314,47315],{},"http://localhost/posts/",", and then show what happens when the user clicks on one of the listed posts to see the detail view of the post (",[30,47318,47319],{},"http://localhost/posts/2",[700,47321,47322,47328,47334,47342,47348,47371,47384,47393,47396,47399,47404,47449],{"start":40612},[79,47323,47324,47327],{},[30,47325,47326],{},"docker-compose up"," is one command that is used to start the entire application in local development. This exposes the NGINX process on port 80 of the host machine (your laptop).",[79,47329,47330,47331,47333],{},"When the application is running on your machine and you navigate to ",[30,47332,47315],{},", the request is first handled by NGINX.",[79,47335,47336,47337,30583,47339,47341],{},"As we mentioned earlier, NGINX's path-based routing sends all requests that do not start with ",[30,47338,27524],{},[30,47340,27527],{}," to the Nuxt.js server.",[79,47343,47344,47345,47347],{},"When the request gets to the Nuxt server, the Nuxt lifecycle methods start. The important one that I'm using so far is ",[30,47346,19808],{},". This property is used to request data that will be used in the rendering of our HTML response.",[79,47349,47350,47351,47353,47354,47356,47357,36611,47359,47362,47363,47366,47367,47370],{},"Inside of ",[30,47352,19808],{},", the application uses axios to make a request to ",[30,47355,47003],{}," (for example). In ",[30,47358,19456],{},[30,47360,47361],{},"privateRuntimeConfig"," sets a baseUrl value for axios to ",[30,47364,47365],{},"http://backend:8000",". Since the Nuxt server is on the same docker network as the backend Django/gunicorn server, the Nuxt server is able to resolve ",[30,47368,47369],{},"http://backend"," to the address of the backend server.",[79,47372,47373,47374,47377,47378,10744,47381,643],{},"Django processes this endpoint, using the ",[30,47375,47376],{},"PostViewSet",", the views of which have been added to ",[30,47379,47380],{},"urlpatterns",[30,47382,47383],{},"blog/urls.py",[79,47385,19225,47386,47388,47389,47392],{},[30,47387,47376],{}," makes a database query on the ",[30,47390,47391],{},"posts"," which is used to serialize the data.",[79,47394,47395],{},"The Django server returns the response to the original axios call.",[79,47397,47398],{},"The data returned from Django is used to render the HTML response.",[79,47400,47401,47402,643],{},"The HTML response from the Nuxt server is sent back to the browser that originally navigated to ",[30,47403,47315],{},[79,47405,47406,47407,47410,47411,47414,47415,47418,47419,47422,47423,47426,47427,47429,47430,47433,47434,47437,47438,47441,47442,47445,47446,47448],{},"The user is presented with page that lists blog posts. Each blog posts lists to a detail view. When a blog post (let's say the post with ",[30,47408,47409],{},"id"," of 2) is clicked on, a request for ",[30,47412,47413],{},"/posts/2/"," is made directly to the Django backend. The ",[30,47416,47417],{},"browserBaseURL"," value in the ",[30,47420,47421],{},"axios"," settings under ",[30,47424,47425],{},"publicRuntimeConfig"," defined in ",[30,47428,19456],{}," is set to ",[30,47431,47432],{},"http://localhost",", so the request is made to ",[30,47435,47436],{},"http://localhost/api/posts/2/",". To clarify, since we are making this request using axios in the browser, we can't make a request to ",[30,47439,47440],{},"http://backend:8000/api/posts/2/"," like we did in step 4 (",[30,47443,47444],{},"http://backend:8000/api/posts/",") because the browser doesn't know how to resolve the ",[30,47447,26811],{}," hostname.",[79,47450,47451,47452,47454,47455,47458,47459,47462,47463,47466],{},"This request to ",[30,47453,47436],{},", like all others, first goes to NGINX which sends it to the backend since the path starts with ",[30,47456,47457],{},"/api/",". At this point the application functions like a regular Vue SPA making axios calls to a backend service. This is because we used ",[30,47460,47461],{},"\u003Cnuxt-link>"," for the posts listed in the posts list view. If we used ",[30,47464,47465],{},"\u003Ca>"," tags, we would go through the same process as in step 4 where the HTML is rendered on the Nuxt server and sent back to the browser all at once.",[56,47468,47114],{"id":47113},[11,47470,47471],{},"My main takeaway is that using Nuxt and Django together can give you good SEO and a great SPA experience at the same time. Using Django alone, or Django with traditional non SSR Vue makes this harder to do. Being a progressive framework, there are a lot of ways to use Vue with any other backend. From what I have heard, most people use Vue via CDN similar to how jQuery was and still is delivered for use in the browser.",[11,47473,47474],{},"There is additional work in setting up 3 servers for a single application (Nuxt, Django and NGINX), but the tradeoff is that I am (at least I feel) very productive writing frontend logic in Vue and backend logic with DRF. I have never liked working with Django templates and I used to know a lot more about them than I do now.",[56,47476,47478],{"id":47477},"spotlight-for-baserowios-awesome-open-source-django-nuxt-application","Spotlight for baserow.io's awesome open-source Django Nuxt application",[11,47480,47481,47482,47487,47488,47493,47494,47498],{},"Lastly I want to mention that there are some great resources in the ",[20,47483,47486],{"href":47484,"rel":47485},"https://github.com/nuxt-community/awesome-nuxt",[24],"nuxt-community/awesome-nuxt"," GitHub repo. There's one project that really stood out to me when I searched for \"django\" projects in the README, and that project is called ",[20,47489,47492],{"href":47490,"rel":47491},"https://baserow.io/",[24],"baserow.io"," (repo: ",[20,47495,47496],{"href":47496,"rel":47497},"https://gitlab.com/bramw/baserow",[24],"). Please check this repo our if you are interested in Django and Nuxt. This company is building an open source no-code database, similar to Airtable which I have worked with before.",[11,47500,47501],{},"Their entire product is open source and I have been very impressed with what I have seen. Please go give that project a star or consider becoming a Github sponsor if you are interested. I'm not affiliated with that project in any way, but I'll be referencing how they use Django and Nuxt to build their application.",[56,47503,47504],{"id":47138},"Next steps",[11,47506,47507,47508,47511],{},"There is a still a lot I have to learn about Nuxt. I'm still very new to the Framework and this is my first time using Nuxt's SSR mode. Nuxt seems to have its own way of doing lots of things that I'm used to doing in Vue. There is a very supportive community and well-maintained official packages to help with lots of things, like the ",[30,47509,47510],{},"@nuxt/axios"," package that I'm using.",[11,47513,47514],{},"My next step is to keep expanding my blog application. One thing I didn't mention is authentication. I plan on using Django session authentication for authenticating request to Django. It seems that it already works correctly in my application (logging in through Django admin and then navigating to Nuxt routes that make Django requests are working only when I'm logged in.) I think I have an idea about how Vuex, authentication and route guards will work together, but I haven't gotten there yet. If anyone has some good reference projects or recommendations on how to expand on what I already have, please let me know!",[11,47516,47517],{},"I know that Nuxt has an auth module, so I need to see if that is relevant for what I want need in my application. I also need to continue reading the Nuxt documentation. I still don't know what I don't know about Nuxt and the plugins and modules that it makes available. I also noticed that Nuxt has it's own version of the Vue 3 Composition API, something I am just now starting to learn more about, so that it another area I'll need to dig into eventually.",{"title":464,"searchDepth":488,"depth":488,"links":47519},[47520,47521,47522,47523,47524],{"id":47250,"depth":488,"text":47251},{"id":47291,"depth":488,"text":47292},{"id":47113,"depth":488,"text":47114},{"id":47477,"depth":488,"text":47478},{"id":47138,"depth":488,"text":47504},"2020-12-27","This article documents my progress combining the Django web framework with Nuxt JS to build applications that have both great SEO and a smooth SPA user experience.",{},"/2020/12/27/building-web-applications-with-django-drf-and-nuxt",{"title":47210,"description":47526},"2020/12/27/building-web-applications-with-django-drf-and-nuxt",[30122,12646,11803,46595,30129],"2rmd0ET1499gf_ZwBljrP4P6gk87WHP0wa3cvCPGUbo",{"id":47534,"title":47535,"body":47536,"comments":602,"date":47663,"description":47664,"draft":602,"extension":605,"external":606,"image":47665,"meta":47666,"navigation":609,"path":47667,"seo":47668,"stem":47669,"tags":47670,"__hash__":47673},"blog/2020/11/29/weekend-project-update-open-sec-data.md","Weekend project update: Open SEC Data",{"type":8,"value":47537,"toc":47661},[47538,47541,47563,47572,47575,47621,47624],[11,47539,47540],{},"Here's an early look at a project I have been working on to practice some Django and Vue.js concepts: Open SEC Data.",[76,47542,47543,47550,47556],{},[79,47544,47545,47549],{},[20,47546,47547],{"href":47547,"rel":47548},"https://opensecdata.ga",[24]," (project staging website, deployed to docker swarm cluster running on DigitalOcean)",[79,47551,47552,47555],{},[20,47553,46158],{"href":46158,"rel":47554},[24]," (main repository, requires GitLab account)",[79,47557,47558,47562],{},[20,47559,47560],{"href":47560,"rel":47561},"https://github.com/briancaffey/sec-filings-app",[24]," (mirror, no account required to view)",[11,47564,47565,47566,47571],{},"This project uses Django, DRF and Celery to read public SEC filings from ",[20,47567,47570],{"href":47568,"rel":47569},"https://www.sec.gov/Archives/edgar/full-index/",[24],"sec.gov",", build it into an API which is consumed through a Vue.js application. I'm currently focused on 13F filings which are required for large US investment funds managing over $100 million USD. There is data dating back to 1993 and it is published quarterly.",[11,47573,47574],{},"Here are some of the things I'm focusing on in this project in no particular order:",[76,47576,47577,47580,47583,47586,47589,47592,47595,47606,47615,47618],{},[79,47578,47579],{},"Getting better at Django REST Framework. This project has been helping me apply some of the parts of DRF that I have found difficult. I'm currently using ViewSets which feels function-based views inside of class-based views. They are flexible, but I would like to add more abstraction with filtering",[79,47581,47582],{},"Django admin. While this project primarily uses Django as a REST API with Django REST Framework, I have tried to take advantage of the Django admin to build out helpful views that can be used to spot check the data I'm creating. Most of my API is read-only, this makes things pretty simple.",[79,47584,47585],{},"Moderately complex paginated data tables with Vue. I work with lots of paginated table data, and I think there is a better way to do abstract some of the repeated logic that I use (getting and setting current page, rows per page). I'm using Vuex, and I have heard of module factories, but I'm thinking that there will be a better way to do this when Vue 3 officially comes to Quasar Framework (Quasar is a Vue.js framework).",[79,47587,47588],{},"Session authentication with DRF. There are a lot of guides showing how to use JWT and Token Authentication for DRF with Javascript frontends. The DRF recommends using Session Authentication for such use cases as a web-base Javascript client, so I hope I can promote some best practices around how to use Django's built-in session authentication for use with the Django REST Framework using an HttpOnly session cookie. I also understand that all security decisions have trade-offs, and I'm trying to understand what trade-offs come with handling authentication in this way.",[79,47590,47591],{},"Social authentication. I have previously setup social authentication with Google, Facebook and GitHub using Python Social Auth. I think it is a great package, and it adds a lot of flexibility with it's concept of pipelines, but I haven't done much with these yet, so I'm hoping to dig in further and better understand how I can make better use of social authentication in my app. This app uses Linkedin 0Auth2 with a custom user model. Logging in with Linkedin account gives you the ability to request an API Token (Django REST Framework's Token) to access the public API.",[79,47593,47594],{},"Automatic API documentation with OpenAPI. Swagger/OpenAPI seems like nice way to document and API, so I'm hoping to build best practices around how to document a DRF API automatically with OpenAPI and Swagger UI.",[79,47596,47597,47598,47601,47602,47605],{},"CI/CD with GitLab and docker swarm. I will admit that I am huge GitLab fan. I love how flexible their CI/CD pipelines are. Being a docker fan as well, I chose to use docker swarm for this project to keep things simple and straightforward. I think one under-appreciate feature of docker is being able to set ",[30,47599,47600],{},"DOCKER_HOST"," to an SSH connection, such as ",[30,47603,47604],{},"ssh://root@123.456.789.10",". This let's you control the remote docker host without needing to SSH to it first, and it is also how I'm able to deploy and run management commands \"manually\" through the GitLab UI.",[79,47607,47608,47609,313,47612,47614],{},"Productive development environment. To start the project, you only need to run docker-compose up (after copying ",[30,47610,47611],{},".env.template",[30,47613,11004],{}," in the root directory for storing sensitive data outside of git such as LinkedIn OAuth2 keys). The development environment is very similar to how this project runs in production with some additional utilities for monitoring and debugging such as pgadmin4, flower (for celery), redis commander (a GUI for viewing redis databases), Django debug toolbar (a must have for any Django project, I believe), runserver_plus with Werkzeug, and others. Also, the backend and frontend hot reload automatically with the help of webpack for Vue and watchdog for Django and Celery.",[79,47616,47617],{},"Automatic TLS certificate generation with Traefik. For a simple project in docker swarm, I'm really happy with how simple it is to request TLS certificates from Let's Encrypt automatically with Traefik. There are no scripts, cron jobs or one-time setup jobs, it just seems to work out of the box if configured correctly.",[79,47619,47620],{},"Testing with pytest. I have only been trying to test most of my API views so far. I really like using factory with pytest, so I use that in most of my tests.",[11,47622,47623],{},"That's all I have for now. I have a long list of questions, things I want to improve, add and experiment with, here are just a few that come to mind:",[76,47625,47626,47629,47632,47640,47643,47646,47649,47652],{},[79,47627,47628],{},"Frontend testing. I don't have any component testing or e2d tests, so this would be good to add eventually. Since I'm using a component library and my app uses these components directly, I'm not exactly sure how much testing I should be doing.",[79,47630,47631],{},"Data verification/validation. There are a lot of site that do provide similar data, WhaleWisdom is the biggest one that I know of. Once I get more data built on the site it would be good to spot check some of the values. There are some nuances to the filing data that I haven't addressed, such as Amendment filings and additions.",[79,47633,47634,47635,643],{},"Calculating period changes. One of the features that I'm not sure how best to implement is the ability to sort holdings for a filer in a given period on the percent increase from the last period. One way would be to add these as additional fields to the Holding model and then calculate these values as I process the data in celery. If I process the data from recent periods to later periods, I will have to update these values once the previous period has been processed, so it would be an additional check to do. I'll probably post this question here in more detail later. Here's ",[20,47636,47639],{"href":47637,"rel":47638},"https://whalewisdom.com/filer/ubs-ag#tabholdings_tab_link",[24],"an example of what this means from WhaleWisdom",[79,47641,47642],{},"Accessing LinkedIn profile data to populate fields on my CustomUser model.",[79,47644,47645],{},"Scaling? I have a lot more experience with deploying projects to AWS which is built around the ability to scale. I don't know a project on DigitalOcean would be scaled automatically. A single node docker swarm cluster while take some time to process all of the data. I would probably be better of scaling vertically with much bigger droplets and higher celery concurrency.",[79,47647,47648],{},"Docker swarm secrets. I'm currently using environment variables to pass secrets stored in GitLab CI when I build images and deploy to docker swarm. I would like to learn how to properly use swarm secrets and work them into my CI/CD pipeline.",[79,47650,47651],{},"As I mentioned above, I'm also interested in updating this project to Vue3 and to apply some of its new features to this project.",[79,47653,47654,47655,47657,47658,643],{},"Use pipenv, poetry or some other way of pinning secondary python dependencies. Does anyone have a recommendation on how best to do this with docker. I have always thought that docker ",[51,47656,40448],{}," the virtual environment, but I realize that some versions of indirect dependencies may change when pip installing without using a lockfile similar to ",[30,47659,47660],{},"package-lock.json",{"title":464,"searchDepth":488,"depth":488,"links":47662},[],"2020-11-29","This project uses Django, DRF and Celery to read public SEC filings from sec.gov, build it into an API which is consumed through a Vue.js application.","/static/sec-update.jpg",{},"/2020/11/29/weekend-project-update-open-sec-data",{"title":47535,"description":47664},"2020/11/29/weekend-project-update-open-sec-data",[30122,12646,12355,47671,47672,30129,27556],"api","gitlab","08eEPj3uympIqf4j5KknXLeDHR1mN86R_NY81yB6_U8",{"id":47675,"title":47676,"body":47677,"comments":609,"date":48042,"description":47681,"draft":602,"extension":605,"external":606,"image":48043,"meta":48044,"navigation":609,"path":48046,"seo":48047,"stem":48048,"tags":48049,"__hash__":48050},"blog/2020/11/27/how-to-authenticate-django-rest-framework-from-vue-app-with-session-authentication-httponly-cookies.md","How to authenticate Django REST Framework API calls from a (Vue) JS client using Session Authentication and HttpOnly cookies",{"type":8,"value":47678,"toc":48036},[47679,47682,47689,47706,47715,47718,47723,47736,47740,47743,47966,47968,47971,47974,47977,47980,47983,47985,47988,48002,48005],[11,47680,47681],{},"This article will describe an authentication strategy using Django REST Framework with a Javascript frontend application. I'll be demonstrating this with Vue.js (Qusar Framework, using Vue 2), but the concepts should transfer to any other Javascript framework.",[11,47683,47684,47685,643],{},"Here's a GitLab repository for a project that I will be referencing throughout this article: ",[20,47686,47687],{"href":47687,"rel":47688},"https://gitlab.com/verbose-equals-true/django-postgres-vue-gitlab-ecs",[24],[11,47690,47691,47692,47697,47698,47705],{},"When I first started learning about how to do authentication from a Vue client, I found ",[20,47693,47696],{"href":47694,"rel":47695},"https://blog.sqreen.com/authentication-best-practices-vue/",[24],"this article from Sqreen"," which describes how to use JWT to authenticate a Vue application. The example uses a mocked backend, but it is a good proxy for what you would have if you were to use a library like ",[20,47699,47702],{"href":47700,"rel":47701},"https://github.com/SimpleJWT/django-rest-framework-simplejwt",[24],[30,47703,47704],{},"django-rest-framework-simplejwt",", which I have previously used with success in Django projects.",[11,47707,47708,47709,47714],{},"JWT is an option for doing authentication with DRF ",[20,47710,47713],{"href":47711,"rel":47712},"https://www.django-rest-framework.org/api-guide/authentication/#json-web-token-authentication",[24],"listed in the authentication documentation",", but the documentation doesn't recommend when or how to use JWT authentication.",[11,47716,47717],{},"Session authentication is mentioned as well:",[210,47719,47720],{},[11,47721,47722],{},"This authentication scheme uses Django's default session backend for authentication. Session authentication is appropriate for AJAX clients that are running in the same session context as your website.",[11,47724,47725,47726,47728,47729,187,47732,47735],{},"In my project, both in local development and in production environments, I serve the API and the Javascript clients on the same domain. ",[30,47727,27524],{}," requests go to the API, and all other request paths route to the frontend client. In other scenarios such as using ",[30,47730,47731],{},"https://mysite.com",[30,47733,47734],{},"https://api.mysite.com"," for hosting a frontend and API an different subdomains, there would need to be additional considerations for CORS, but since I have the frontend and the backend being served on the same domain (and same subdomain), this isn't a concern. You might need to watch out for this if your requirements are different.",[56,47737,47739],{"id":47738},"the-authentication-flow","The authentication flow",[11,47741,47742],{},"Now let's describe the login process at a high level.",[700,47744,47745,47751,47760,47778,47797,47820,47845,47874,47883,47903,47915,47933],{},[79,47746,47747,47748,47750],{},"A user navigates to your site. There is currently nothing in the browser's ",[30,47749,33952],{}," or cookies related to authentication.",[79,47752,47753,47754,47757,47758,643],{},"The user navigates to the ",[30,47755,47756],{},"Login"," page at ",[30,47759,46784],{},[79,47761,47762,47763,47766,47767,47769,47770,47773,47774,47777],{},"Loading this Vue component makes a ",[30,47764,47765],{},"GET"," request to a special endpoint in our Django backend ",[30,47768,46873],{},". This request returns a simple JSON message: ",[30,47771,47772],{},"\"CSRF cookie set\"",", and as the message says, the response sets a ",[30,47775,47776],{},"csrf"," cookie on our browser.",[79,47779,47780,47781,47783,47784,47787,47788,47790,47791,47793,47794,47796],{},"Once the CSRF cookie is set by the response from ",[30,47782,46873],{},", the user is presented with a login form and enters account credentials (email and password in my example, where email is the ",[30,47785,47786],{},"USERNAME_FIELD"," on my custom user model). Clicking \"Login\" dispatches a Vuex action that uses Axios to send a send a request to ",[30,47789,46905],{}," with the ",[30,47792,47776],{}," cookie set in a ",[30,47795,46901],{}," header.",[79,47798,47799,47801,47802,47804,47805,6208,47807,187,47809,18952,47811,47813,47814,47816,47817,47819],{},[30,47800,46905],{}," is handled by the ",[30,47803,46920],{}," view which uses two important functions from ",[30,47806,46924],{},[30,47808,46927],{},[30,47810,46930],{},[30,47812,46927],{}," gets the user from the provided credentials, and ",[30,47815,46930],{}," sets a ",[30,47818,46939],{}," HttpOnly cookie on the response.",[79,47821,47822,47823,47825,47826,47828,47829,313,47831,47834,47835,47838,47839,47841,47842,748],{},"When the response from ",[30,47824,46905],{}," comes back, two things happen: first the ",[30,47827,46939],{}," HttpOnly cookie is set on our browser. Second, we set a value in both Vuex and localStorage named ",[30,47830,46854],{},[30,47832,47833],{},"success",". We are not storing any sensitive information in this value. Instead, we are using this value to signal to the rest of our Vue application that the user has authenticated. Storing this in Vuex allows us to use global Vuex ",[30,47836,47837],{},"getters"," so that we can change component state and other logic where authentication is concerned, such as route guards (for Vue router). We store it in localStorage so that when a new browser tab is opened, we can set the value of ",[30,47840,46854],{}," in Vuex based on the value in localStorage. (",[30,47843,47844],{},"authenticated: localStorage.getItem(\"authenticated\") || \"\",",[79,47846,47847,47848,47850,47851,47854,47855,47858,47859,47861,47862,47865,47866,47868,47869,313,47871,47873],{},"Since the ",[30,47849,46939],{}," cookie is HttpOnly, we can't use Javascript to interact with it, so when we want to logout the user we can't just delete the cookie. To logout the user, we make a request to ",[30,47852,47853],{},"/api/logout/"," when the user clicks on the logout button. The view for this endpoint does ",[30,47856,47857],{},"logout(request)",". This returns a response with a new value for the ",[30,47860,46939],{}," cookie: ",[30,47863,47864],{},"Set-Cookie: sessionid=\"\"; expires=Thu, 01 Jan 1970 00:00:00 GMT; Max-Age=0; Path=/; SameSite=Lax",". Since the cookie's expiration is in the past, it is removed entirely. We also remove the ",[30,47867,46854],{}," localStorage item and set the Vuex store value of ",[30,47870,46854],{},[30,47872,2301],{}," (which is falsey). This lets the Vue application know that user has been logged out. One consideration for this is that your user can't logout while offline.",[79,47875,47876,47877,47882],{},"This repo also implements social authentication with the fantastic Python Social Auth library. I used Facebook, Google and GitHub, but there are lots of other providers you can choose from depending on what you need. This is a lengthy topic, and I recommend that you read ",[20,47878,47881],{"href":47879,"rel":47880},"https://www.toptal.com/django/integrate-oauth-2-into-django-drf-back-end",[24],"How to Integrate OAuth 2 Into Your Django/DRF Back-end Without Going Insane"," on which I have based my implementation. Here's the short story of how this works. First, a user clicks on one of the social sign-in links. These links are the same for sign-in and sign-up.",[79,47884,47885,47886,26802,47889,47892,47893,47896,47897,47899,47900,47902],{},"The link has a few parts, here's an example: ",[30,47887,47888],{},"https://github.com/login/oauth/authorize?client_id=r66bdfgsfsbferfef4&redirect_uri=http:%2F%2Flocalhost%2Fauth%2Fgithub%2Fcallback&login=&scope=user:email&state=ewori4t95k3vdzem",[30,47890,47891],{},"client_id"," is the app we created to allow our users to sign in. When you create this app, you specify the ",[30,47894,47895],{},"redirect_uri"," in the configuration, and you reference that here as well. ",[30,47898,37849],{}," specifies scope of access we are requesting from the user's social account. ",[30,47901,34576],{}," is used for security.",[79,47904,47905,47906,6208,47908,47911,47912,47914],{},"When you click on the link above, you are redirected to GitHub and asked if you want to grant my GitHub application access to your account's associated email address. Clicking on \"Authorize\" then redirects you back to the ",[30,47907,47895],{},[30,47909,47910],{},"http://localhost/auth/github/callback?code=veroi3409e203ej&state=ewori4t95k3vdzem",". Notice the ",[30,47913,30],{}," parameter.",[79,47916,47917,47918,47921,47922,47925,47926,47929,47930,643],{},"When you navigate to ",[30,47919,47920],{},"/auth/github/callback"," on the Vue application, you see a message: \"Logging in with GitHub...\". On this page's ",[30,47923,47924],{},"mounted"," method we call ",[30,47927,47928],{},"handleOauthCallback"," which makes a request to our Django application: ",[30,47931,47932],{},"/api/social/github/?code=veroi3409e203ej",[79,47934,47935,47936,47939,47940,47943,47944,47946,47947,47950,47951,47954,47955,47958,47959,47961,47962,47965],{},"This API endpoint uses the ",[30,47937,47938],{},"exchange_token"," view which is where Python Social Auth starts to do the heavy lifting. First, we need to make an API request to GitHub (",[30,47941,47942],{},"https://github.com/login/oauth/access_token",") with the ",[30,47945,30],{}," as a URL parameter, then we pass the access code that this API call returns into Python Social Auth's ",[30,47948,47949],{},"do_auth"," function to get our Django user. Finally, similar to the email/password login approach described above, we call ",[30,47952,47953],{},"login(request, user)"," and return a simple JSON response: ",[30,47956,47957],{},"{\"detail\": \"success\"}",". This will set the ",[30,47960,46939],{}," automatically when the response returns, and we can dispatch the same Vuex action ",[30,47963,47964],{},"AUTH_SUCCESS"," to tell Vuex that a user has been logged in.",[56,47967,47114],{"id":47113},[11,47969,47970],{},"Using the default Django session authentication mechanism has some nice advantages. It allows us to easily navigate between our Javascript SPA which uses Django REST Framework, regular Django admin views that you may also be using, as well as the Django admin.",[11,47972,47973],{},"Using DRF's token authentication is still possible if you choose to use Session authentication for your JS frontend. For example, you may wish to allow users to make authenticated API requests to your public API using DRF Token Authentication.",[11,47975,47976],{},"JWT is a really interesting concept and important to know about, but it doesn't seem like a practical solution for any of my use cases with Django APIs or frontends. You also can't really \"logout\" a user if you are using this solution for authentication.",[11,47978,47979],{},"What I have described here is pretty simple scenario. It assumes that there is only one type of user and that there are no additional steps needed to make your account \"active\". Doing this would require additional logic on the Vue/Vuex side as well as the backend logic, including the User model. I also don't make use of any data from the social providers except for the user's email address.",[11,47981,47982],{},"Another thing to be aware of with this scenario is that a user can register with social authentication first, and then reset their password and login with email (I haven't implemented this client-side on this project yet). Or, you can login with an email account that was created through the Django admin and then login with a social account tied to that email. There are a lot of options for each backend in Python social auth, making it a very flexible library for handling social authentication.",[56,47984,47139],{"id":47138},[11,47986,47987],{},"There is a lot more work to do on this example project regarding authentication, but I hope it can help point some people in the right direction. Here are some areas that I would like to work on next:",[76,47989,47990,47993,47996,47999],{},[79,47991,47992],{},"Error handling for a bad authentication attempt",[79,47994,47995],{},"A signup form with email confirmation, handling cases where the user trying to signup may already have signed in with social authentication",[79,47997,47998],{},"Password reset with email",[79,48000,48001],{},"Making use of Python Social Auth settings and options, including pipelines.",[56,48003,26869],{"id":48004},"resources",[76,48006,48007,48012,48018,48024,48030],{},[79,48008,48009],{},[20,48010,47879],{"href":47879,"rel":48011},[24],[79,48013,48014],{},[20,48015,48016],{"href":48016,"rel":48017},"https://yoongkang.com/blog/cookie-based-authentication-spa-django/",[24],[79,48019,48020],{},[20,48021,48022],{"href":48022,"rel":48023},"https://github.com/encode/django-rest-framework/issues/7273",[24],[79,48025,48026],{},[20,48027,48028],{"href":48028,"rel":48029},"https://github.com/SimpleJWT/django-rest-framework-simplejwt/issues/71",[24],[79,48031,48032],{},[20,48033,48034],{"href":48034,"rel":48035},"http://cryto.net/~joepie91/blog/2016/06/13/stop-using-jwt-for-sessions/",[24],{"title":464,"searchDepth":488,"depth":488,"links":48037},[48038,48039,48040,48041],{"id":47738,"depth":488,"text":47739},{"id":47113,"depth":488,"text":47114},{"id":47138,"depth":488,"text":47139},{"id":48004,"depth":488,"text":26869},"2020-11-27","/static/padlocks.jpg",{"layout":48045},"post","/2020/11/27/how-to-authenticate-django-rest-framework-from-vue-app-with-session-authentication-httponly-cookies",{"title":47676,"description":47681},"2020/11/27/how-to-authenticate-django-rest-framework-from-vue-app-with-session-authentication-httponly-cookies",[30122,12646,36041,47671],"GvTaAGutMol1B0i9VJwzVjzU5A3eb8KoWwGqhzkLUBk",{"id":48052,"title":48053,"body":48054,"comments":609,"date":48842,"description":48058,"draft":602,"extension":605,"external":606,"image":48843,"meta":48844,"navigation":609,"path":48845,"seo":48846,"stem":48847,"tags":48848,"__hash__":48849},"blog/2020/10/10/how-to-add-email-signup-form-to-nuxt-site-with-mailchimp.md","How to add an email signup form to a Nuxt site with MailChimp",{"type":8,"value":48055,"toc":48833},[48056,48059,48063,48066,48077,48079,48082,48121,48124,48128,48139,48145,48148,48158,48164,48185,48192,48731,48734,48738,48744,48748,48751,48754,48759,48768,48771,48780,48787,48801,48803,48817,48819,48830],[11,48057,48058],{},"This is a guide for setting up a MailChimp-powered newsletter signup form on a static Nuxt site hosted on GitHub Pages. I wanted to implement this on my personal blog to grow a mailing list so I can update readers when I puslish a new blog article. I haven't done too much work with MailChimp before, but I've definitely subscribed to plenty of MailChimp mailing lists.",[56,48060,48062],{"id":48061},"goals","Goals",[11,48064,48065],{},"I'm hoping to acheive some of the following:",[76,48067,48068,48071,48074],{},[79,48069,48070],{},"Build a list of emails from people who visit my site and want get",[79,48072,48073],{},"Include a simple newletter signup form (Vue) component in the Footer of my site",[79,48075,48076],{},"Send customized emails (campaigns) to a mailing list that I can track",[56,48078,47149],{"id":47148},[11,48080,48081],{},"Here are some of the questions (and answers) that I had going into this:",[76,48083,48084,48089,48102,48107,48110,48116],{},[79,48085,48086,48087,748],{},"Do I need to use the MailChimp API or MailChimp API keys? (",[15,48088,13404],{},[79,48090,48091,48092,48095,48096,48099,48100,748],{},"Is ",[15,48093,48094],{},"Double Opt-In"," possible with a static GitHub pages site hosted on a ",[30,48097,48098],{},"\u003Cusernme>.github.io"," subdomain? (",[15,48101,13436],{},[79,48103,48104,48105,748],{},"What is the signup flow? Will a new subscriber be sent to a MailChimp page first, and then back to my site? (",[15,48106,13436],{},[79,48108,48109],{},"How do I setup a \"Thank you\" page to show users after they subscribe?",[79,48111,48112,48113,748],{},"How much will this cost? (",[15,48114,48115],{},"Free up to 2,000 subscribers",[79,48117,48118,48119,748],{},"Should I create a new campaign for each new blog post email update I send out? (",[15,48120,13436],{},[11,48122,48123],{},"I'll touch on these questions as I describe how to set things up.",[56,48125,48127],{"id":48126},"creating-the-form","Creating the form",[11,48129,48130,48131,48134,48135,48138],{},"Under the ",[30,48132,48133],{},"Audience > Signup forms"," menu, I selected the ",[30,48136,48137],{},"Embedded Forms"," option:",[459,48140,48143],{"className":48141,"code":48142,"language":997},[995],"Embedded forms\nGenerate HTML code to embed in your site or blog to collect signups.\n",[30,48144,48142],{"__ignoreMap":464},[11,48146,48147],{},"Most of the MailChimp Admin that I have been using seems to have 4 different menus: 2 vertical menus and 2 horizontal.",[11,48149,48130,48150,48153,48154,48157],{},[30,48151,48152],{},"Embedded forms"," menu, I selected ",[30,48155,48156],{},"Unstyled"," since I want to add my own Tailwind CSS classes to keep the style of my signup form consistent with the different color schemes available on my site.",[11,48159,48160,48161,208],{},"I'll start with an Unstyled Embedded form. Let's go through the ",[30,48162,48163],{},"Form options",[76,48165,48166,48172,48182],{},[79,48167,48168,48171],{},[15,48169,48170],{},"Include form title"," (no, I'll add this myself)",[79,48173,48174,48177,48178,48181],{},[15,48175,48176],{},"Show only required fields"," (For now, ",[30,48179,48180],{},"email"," is the only field I will be capturing. It can be helpful to include a First and/or Last name to avoid having your emails go to users' spam folders.)",[79,48183,48184],{},"Unselect everything else",[11,48186,48187,48188,48191],{},"Here's the HTML that we can use in our ",[30,48189,48190],{},"Subscribe.vue"," component:",[459,48193,48195],{"className":19811,"code":48194,"language":19813,"meta":464,"style":464},"    \u003C!-- Begin Mailchimp Signup Form -->\n    \u003Cdiv id=\"mc_embed_signup\">\n      \u003Cform\n        action=\"https://github.us2.list-manage.com/subscribe/post?u=43a795784ca963e25903a0da6&id=9937fe4fc5\"\n        method=\"post\"\n        id=\"mc-embedded-subscribe-form\"\n        name=\"mc-embedded-subscribe-form\"\n        class=\"validate\"\n        target=\"_blank\"\n        novalidate\n      >\n        \u003Cdiv id=\"mc_embed_signup_scroll\">\n          \u003Cdiv class=\"mc-field-group\">\n            \u003C!-- \u003Clabel for=\"mce-EMAIL\">Email Address \u003C/label> -->\n            \u003C!-- Added placeholder -->\n            \u003Cinput\n              type=\"email\"\n              value=\"\"\n              name=\"EMAIL\"\n              class=\"required email\"\n              id=\"mce-EMAIL\"\n              placeholder=\"Enter your email address\"\n            />\n          \u003C/div>\n          \u003Cdiv id=\"mce-responses\" class=\"clear\">\n            \u003Cdiv\n              class=\"response\"\n              id=\"mce-error-response\"\n              style=\"display: none\"\n            >\u003C/div>\n            \u003Cdiv\n              class=\"response\"\n              id=\"mce-success-response\"\n              style=\"display: none\"\n            >\u003C/div>\n          \u003C/div>\n          \u003C!-- real people should not fill this in and expect good things - do not remove this or risk form bot signups-->\n          \u003Cdiv style=\"position: absolute; left: -5000px\" aria-hidden=\"true\">\n            \u003Cinput\n              type=\"text\"\n              name=\"b_43a795784ca963e25903a0da6_9937fe4fc5\"\n              tabindex=\"-1\"\n              value=\"\"\n            />\n          \u003C/div>\n          \u003Cdiv class=\"clear\">\n            \u003Cinput\n              type=\"submit\"\n              value=\"Subscribe\"\n              name=\"subscribe\"\n              id=\"mc-embedded-subscribe\"\n              class=\"button\"\n            />\n          \u003C/div>\n        \u003C/div>\n      \u003C/form>\n    \u003C/div>\n\n    \u003C!--End mc_embed_signup-->\n",[30,48196,48197,48202,48218,48226,48242,48251,48260,48269,48279,48289,48294,48299,48314,48331,48336,48341,48348,48358,48368,48378,48388,48398,48408,48413,48422,48444,48451,48460,48469,48479,48488,48494,48502,48511,48519,48527,48535,48540,48564,48570,48579,48588,48598,48606,48610,48618,48632,48638,48647,48656,48665,48674,48683,48687,48695,48703,48713,48722,48726],{"__ignoreMap":464},[151,48198,48199],{"class":469,"line":470},[151,48200,48201],{"class":1527},"    \u003C!-- Begin Mailchimp Signup Form -->\n",[151,48203,48204,48206,48208,48211,48213,48216],{"class":469,"line":488},[151,48205,34669],{"class":503},[151,48207,23950],{"class":14368},[151,48209,48210],{"class":473}," id",[151,48212,1876],{"class":503},[151,48214,48215],{"class":481},"\"mc_embed_signup\"",[151,48217,3742],{"class":503},[151,48219,48220,48223],{"class":469,"line":500},[151,48221,48222],{"class":503},"      \u003C",[151,48224,48225],{"class":14368},"form\n",[151,48227,48228,48231,48233,48236,48239],{"class":469,"line":509},[151,48229,48230],{"class":473},"        action",[151,48232,1876],{"class":503},[151,48234,48235],{"class":481},"\"https://github.us2.list-manage.com/subscribe/post?u=43a795784ca963e25903a0da6",[151,48237,48238],{"class":477},"&",[151,48240,48241],{"class":481},"id=9937fe4fc5\"\n",[151,48243,48244,48246,48248],{"class":469,"line":517},[151,48245,14472],{"class":473},[151,48247,1876],{"class":503},[151,48249,48250],{"class":481},"\"post\"\n",[151,48252,48253,48255,48257],{"class":469,"line":534},[151,48254,27874],{"class":473},[151,48256,1876],{"class":503},[151,48258,48259],{"class":481},"\"mc-embedded-subscribe-form\"\n",[151,48261,48262,48265,48267],{"class":469,"line":1413},[151,48263,48264],{"class":473},"        name",[151,48266,1876],{"class":503},[151,48268,48259],{"class":481},[151,48270,48271,48274,48276],{"class":469,"line":1418},[151,48272,48273],{"class":473},"        class",[151,48275,1876],{"class":503},[151,48277,48278],{"class":481},"\"validate\"\n",[151,48280,48281,48284,48286],{"class":469,"line":2462},[151,48282,48283],{"class":473},"        target",[151,48285,1876],{"class":503},[151,48287,48288],{"class":481},"\"_blank\"\n",[151,48290,48291],{"class":469,"line":2471},[151,48292,48293],{"class":473},"        novalidate\n",[151,48295,48296],{"class":469,"line":2480},[151,48297,48298],{"class":503},"      >\n",[151,48300,48301,48303,48305,48307,48309,48312],{"class":469,"line":2489},[151,48302,21070],{"class":503},[151,48304,23950],{"class":14368},[151,48306,48210],{"class":473},[151,48308,1876],{"class":503},[151,48310,48311],{"class":481},"\"mc_embed_signup_scroll\"",[151,48313,3742],{"class":503},[151,48315,48316,48319,48321,48324,48326,48329],{"class":469,"line":2497},[151,48317,48318],{"class":503},"          \u003C",[151,48320,23950],{"class":14368},[151,48322,48323],{"class":473}," class",[151,48325,1876],{"class":503},[151,48327,48328],{"class":481},"\"mc-field-group\"",[151,48330,3742],{"class":503},[151,48332,48333],{"class":469,"line":3140},[151,48334,48335],{"class":1527},"            \u003C!-- \u003Clabel for=\"mce-EMAIL\">Email Address \u003C/label> -->\n",[151,48337,48338],{"class":469,"line":3149},[151,48339,48340],{"class":1527},"            \u003C!-- Added placeholder -->\n",[151,48342,48343,48345],{"class":469,"line":3158},[151,48344,21079],{"class":503},[151,48346,48347],{"class":14368},"input\n",[151,48349,48350,48353,48355],{"class":469,"line":3167},[151,48351,48352],{"class":473},"              type",[151,48354,1876],{"class":503},[151,48356,48357],{"class":481},"\"email\"\n",[151,48359,48360,48363,48365],{"class":469,"line":3175},[151,48361,48362],{"class":473},"              value",[151,48364,1876],{"class":503},[151,48366,48367],{"class":481},"\"\"\n",[151,48369,48370,48373,48375],{"class":469,"line":3184},[151,48371,48372],{"class":473},"              name",[151,48374,1876],{"class":503},[151,48376,48377],{"class":481},"\"EMAIL\"\n",[151,48379,48380,48383,48385],{"class":469,"line":3193},[151,48381,48382],{"class":473},"              class",[151,48384,1876],{"class":503},[151,48386,48387],{"class":481},"\"required email\"\n",[151,48389,48390,48393,48395],{"class":469,"line":3720},[151,48391,48392],{"class":473},"              id",[151,48394,1876],{"class":503},[151,48396,48397],{"class":481},"\"mce-EMAIL\"\n",[151,48399,48400,48403,48405],{"class":469,"line":3729},[151,48401,48402],{"class":473},"              placeholder",[151,48404,1876],{"class":503},[151,48406,48407],{"class":481},"\"Enter your email address\"\n",[151,48409,48410],{"class":469,"line":3735},[151,48411,48412],{"class":503},"            />\n",[151,48414,48415,48418,48420],{"class":469,"line":3745},[151,48416,48417],{"class":503},"          \u003C/",[151,48419,23950],{"class":14368},[151,48421,3742],{"class":503},[151,48423,48424,48426,48428,48430,48432,48435,48437,48439,48442],{"class":469,"line":3754},[151,48425,48318],{"class":503},[151,48427,23950],{"class":14368},[151,48429,48210],{"class":473},[151,48431,1876],{"class":503},[151,48433,48434],{"class":481},"\"mce-responses\"",[151,48436,48323],{"class":473},[151,48438,1876],{"class":503},[151,48440,48441],{"class":481},"\"clear\"",[151,48443,3742],{"class":503},[151,48445,48446,48448],{"class":469,"line":3760},[151,48447,21079],{"class":503},[151,48449,48450],{"class":14368},"div\n",[151,48452,48453,48455,48457],{"class":469,"line":3773},[151,48454,48382],{"class":473},[151,48456,1876],{"class":503},[151,48458,48459],{"class":481},"\"response\"\n",[151,48461,48462,48464,48466],{"class":469,"line":3782},[151,48463,48392],{"class":473},[151,48465,1876],{"class":503},[151,48467,48468],{"class":481},"\"mce-error-response\"\n",[151,48470,48471,48474,48476],{"class":469,"line":3791},[151,48472,48473],{"class":473},"              style",[151,48475,1876],{"class":503},[151,48477,48478],{"class":481},"\"display: none\"\n",[151,48480,48481,48484,48486],{"class":469,"line":3803},[151,48482,48483],{"class":503},"            >\u003C/",[151,48485,23950],{"class":14368},[151,48487,3742],{"class":503},[151,48489,48490,48492],{"class":469,"line":3811},[151,48491,21079],{"class":503},[151,48493,48450],{"class":14368},[151,48495,48496,48498,48500],{"class":469,"line":3820},[151,48497,48382],{"class":473},[151,48499,1876],{"class":503},[151,48501,48459],{"class":481},[151,48503,48504,48506,48508],{"class":469,"line":7084},[151,48505,48392],{"class":473},[151,48507,1876],{"class":503},[151,48509,48510],{"class":481},"\"mce-success-response\"\n",[151,48512,48513,48515,48517],{"class":469,"line":7148},[151,48514,48473],{"class":473},[151,48516,1876],{"class":503},[151,48518,48478],{"class":481},[151,48520,48521,48523,48525],{"class":469,"line":7211},[151,48522,48483],{"class":503},[151,48524,23950],{"class":14368},[151,48526,3742],{"class":503},[151,48528,48529,48531,48533],{"class":469,"line":7273},[151,48530,48417],{"class":503},[151,48532,23950],{"class":14368},[151,48534,3742],{"class":503},[151,48536,48537],{"class":469,"line":7335},[151,48538,48539],{"class":1527},"          \u003C!-- real people should not fill this in and expect good things - do not remove this or risk form bot signups-->\n",[151,48541,48542,48544,48546,48549,48551,48554,48557,48559,48562],{"class":469,"line":7398},[151,48543,48318],{"class":503},[151,48545,23950],{"class":14368},[151,48547,48548],{"class":473}," style",[151,48550,1876],{"class":503},[151,48552,48553],{"class":481},"\"position: absolute; left: -5000px\"",[151,48555,48556],{"class":473}," aria-hidden",[151,48558,1876],{"class":503},[151,48560,48561],{"class":481},"\"true\"",[151,48563,3742],{"class":503},[151,48565,48566,48568],{"class":469,"line":7462},[151,48567,21079],{"class":503},[151,48569,48347],{"class":14368},[151,48571,48572,48574,48576],{"class":469,"line":7467},[151,48573,48352],{"class":473},[151,48575,1876],{"class":503},[151,48577,48578],{"class":481},"\"text\"\n",[151,48580,48581,48583,48585],{"class":469,"line":7532},[151,48582,48372],{"class":473},[151,48584,1876],{"class":503},[151,48586,48587],{"class":481},"\"b_43a795784ca963e25903a0da6_9937fe4fc5\"\n",[151,48589,48590,48593,48595],{"class":469,"line":7537},[151,48591,48592],{"class":473},"              tabindex",[151,48594,1876],{"class":503},[151,48596,48597],{"class":481},"\"-1\"\n",[151,48599,48600,48602,48604],{"class":469,"line":7603},[151,48601,48362],{"class":473},[151,48603,1876],{"class":503},[151,48605,48367],{"class":481},[151,48607,48608],{"class":469,"line":7608},[151,48609,48412],{"class":503},[151,48611,48612,48614,48616],{"class":469,"line":7673},[151,48613,48417],{"class":503},[151,48615,23950],{"class":14368},[151,48617,3742],{"class":503},[151,48619,48620,48622,48624,48626,48628,48630],{"class":469,"line":7678},[151,48621,48318],{"class":503},[151,48623,23950],{"class":14368},[151,48625,48323],{"class":473},[151,48627,1876],{"class":503},[151,48629,48441],{"class":481},[151,48631,3742],{"class":503},[151,48633,48634,48636],{"class":469,"line":7708},[151,48635,21079],{"class":503},[151,48637,48347],{"class":14368},[151,48639,48640,48642,48644],{"class":469,"line":7713},[151,48641,48352],{"class":473},[151,48643,1876],{"class":503},[151,48645,48646],{"class":481},"\"submit\"\n",[151,48648,48649,48651,48653],{"class":469,"line":7746},[151,48650,48362],{"class":473},[151,48652,1876],{"class":503},[151,48654,48655],{"class":481},"\"Subscribe\"\n",[151,48657,48658,48660,48662],{"class":469,"line":7751},[151,48659,48372],{"class":473},[151,48661,1876],{"class":503},[151,48663,48664],{"class":481},"\"subscribe\"\n",[151,48666,48667,48669,48671],{"class":469,"line":7816},[151,48668,48392],{"class":473},[151,48670,1876],{"class":503},[151,48672,48673],{"class":481},"\"mc-embedded-subscribe\"\n",[151,48675,48676,48678,48680],{"class":469,"line":7821},[151,48677,48382],{"class":473},[151,48679,1876],{"class":503},[151,48681,48682],{"class":481},"\"button\"\n",[151,48684,48685],{"class":469,"line":7847},[151,48686,48412],{"class":503},[151,48688,48689,48691,48693],{"class":469,"line":7852},[151,48690,48417],{"class":503},[151,48692,23950],{"class":14368},[151,48694,3742],{"class":503},[151,48696,48697,48699,48701],{"class":469,"line":7887},[151,48698,21175],{"class":503},[151,48700,23950],{"class":14368},[151,48702,3742],{"class":503},[151,48704,48705,48708,48711],{"class":469,"line":7892},[151,48706,48707],{"class":503},"      \u003C/",[151,48709,48710],{"class":14368},"form",[151,48712,3742],{"class":503},[151,48714,48715,48718,48720],{"class":469,"line":7924},[151,48716,48717],{"class":503},"    \u003C/",[151,48719,23950],{"class":14368},[151,48721,3742],{"class":503},[151,48723,48724],{"class":469,"line":7929},[151,48725,1090],{"emptyLinePlaceholder":609},[151,48727,48728],{"class":469,"line":7991},[151,48729,48730],{"class":1527},"    \u003C!--End mc_embed_signup-->\n",[11,48732,48733],{},"Now we can create a Vue component that contains the HTML form generated by the MailChimp admin. For now, just put the embed HTML in the template of a Vue component.",[56,48735,48737],{"id":48736},"styling-the-form","Styling the form",[11,48739,48740,48741,48743],{},"Next you can add styles to the form. You can reference the ",[30,48742,48190],{}," file in the repo for this site to see how I have added styles using TailwindCSS.",[56,48745,48747],{"id":48746},"settings","Settings",[11,48749,48750],{},"Next let's look at some settings around Double Opt-in.",[11,48752,48753],{},"To navigate to the page where these settings can be set, go to:",[11,48755,48756],{},[30,48757,48758],{},"Audience > Signup forms > Settings > Audience name and defaults",[11,48760,27190,48761,48764,48765],{},[30,48762,48763],{},"Form Settings"," of this menu, you can select ",[30,48766,48767],{},"Enable double opt-in",[11,48769,48770],{},"Next, let's configure the redirect to a custom \"Thank you for subscribing page\" on our static site. Go to",[11,48772,48773,48776,48777],{},[30,48774,48775],{},"Audience > Signup forms > Signup forms"," and select ",[30,48778,48779],{},"Form builder",[11,48781,48782,48783,48786],{},"Select ",[15,48784,48785],{},"Signup thank you page"," from the dropdown menu and add a custom URL that you will show users when they first submit their email to the form.",[11,48788,48789,48790,48793,48794,48797,48798],{},"Next, also on the ",[30,48791,48792],{},"Form Builder"," menu, select ",[15,48795,48796],{},"Confirmation thank you page"," from the dropdown menu and enter the URL of the page that you want to redirect to after a user confirms their subscription where it says: ",[30,48799,48800],{},"Instead of showing this thank you page, send subscribers to another URL",[56,48802,47139],{"id":47138},[76,48804,48805,48811,48814],{},[79,48806,48807,48810],{},[15,48808,48809],{},"Form validation",": we can validate that the user has entered a valid email address",[79,48812,48813],{},"GDPR considerations",[79,48815,48816],{},"Captcha",[56,48818,26869],{"id":48004},[76,48820,48821],{},[79,48822,48823,6208,48826],{},[15,48824,48825],{},"Single vs Double Opt-in",[20,48827,48828],{"href":48828,"rel":48829},"https://www.sendinblue.com/blog/double-opt-in/",[24],[589,48831,48832],{},"html pre.shiki code .s8-w5, html code.shiki .s8-w5{--shiki-default:#6A737D;--shiki-dark:#6A737D;--shiki-sepia:#88846F}html pre.shiki code .sMOD_, html code.shiki .sMOD_{--shiki-default:#24292E;--shiki-dark:#E1E4E8;--shiki-sepia:#F8F8F2}html pre.shiki code .s5clZ, html code.shiki .s5clZ{--shiki-default:#22863A;--shiki-dark:#85E89D;--shiki-sepia:#F92672}html pre.shiki code .srTi1, html code.shiki .srTi1{--shiki-default:#6F42C1;--shiki-dark:#B392F0;--shiki-sepia:#A6E22E}html pre.shiki code .sstjo, html code.shiki .sstjo{--shiki-default:#032F62;--shiki-dark:#9ECBFF;--shiki-sepia:#E6DB74}html pre.shiki code .s7F3e, html code.shiki .s7F3e{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#AE81FF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html .sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html.sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}",{"title":464,"searchDepth":488,"depth":488,"links":48834},[48835,48836,48837,48838,48839,48840,48841],{"id":48061,"depth":488,"text":48062},{"id":47148,"depth":488,"text":47149},{"id":48126,"depth":488,"text":48127},{"id":48736,"depth":488,"text":48737},{"id":48746,"depth":488,"text":48747},{"id":47138,"depth":488,"text":47139},{"id":48004,"depth":488,"text":26869},"2020-10-11","/static/chimps.webp",{"layout":48045},"/2020/10/10/how-to-add-email-signup-form-to-nuxt-site-with-mailchimp",{"title":48053,"description":48058},"2020/10/10/how-to-add-email-signup-form-to-nuxt-site-with-mailchimp",[11803,35582,12646],"bKJURMF8UTq1rN598T2fpx1IxdSuuUIrSwtKwXlsZ8s",{"id":48851,"title":48852,"body":48853,"comments":609,"date":49190,"description":48857,"draft":602,"extension":605,"external":606,"image":49051,"meta":49191,"navigation":609,"path":49192,"seo":49193,"stem":49194,"tags":49195,"__hash__":49198},"blog/2020/09/17/migrating-from-jekyll-to-nuxt-static-site.md","Migrating my personal GitHub pages blog from Jekyll to Nuxt",{"type":8,"value":48854,"toc":49183},[48855,48858,48861,48872,48876,48885,49031,49034,49040,49046,49052,49056,49059,49102,49106,49116,49150,49154,49173,49175,49180],[11,48856,48857],{},"Jekyll has served me well for a long time, but I've decided to switch the static site generator I use for my personal blog from Jekyll to Nuxt. This article will hopefully be the first new article in my Nuxt blog.",[11,48859,48860],{},"Here are some goals for what I want to do with this new site:",[76,48862,48863,48866,48869],{},[79,48864,48865],{},"Learn more about Nuxt and Nuxt Content, JAM Stack and static site generation",[79,48867,48868],{},"Master TailwindCSS for building responsive layouts",[79,48870,48871],{},"Better understand and measure SEO for my site and the content I publish on it",[56,48873,48875],{"id":48874},"creating-a-nuxt-project","Creating a nuxt project",[11,48877,48878,48879,48882,48883,208],{},"I'm going to start off with a ",[30,48880,48881],{},"feature-nuxt"," feature branch. Since I can't create the nuxt project in a non-empty directory, I'll create a new nuxt project in a folder called ",[30,48884,11803],{},[459,48886,48890],{"className":48887,"code":48888,"language":48889,"meta":464,"style":464},"language-txt shiki shiki-themes github-light github-dark monokai","brian@x1:~/github/briancaffey.github.io/nuxt$ npx create-nuxt-app .\n\ncreate-nuxt-app v3.2.0\n✨  Generating Nuxt.js project in .\n(node:8073) ExperimentalWarning: Conditional exports is an experimental feature. This feature could change at any time\n? Project name: brian-caffey\n? Programming language: JavaScript\n? Package manager: Yarn\n? UI framework: Tailwind CSS\n? Nuxt.js modules: Axios, Content\n? Linting tools: ESLint, Prettier\n? Testing framework: None\n? Rendering mode: Universal (SSR / SSG)\n? Deployment target: Static (Static/JAMStack hosting)\n? Development tools: jsconfig.json (Recommended for VS Code if you're not using typescript)\nyarn run v1.22.5\n$ eslint --ext .js,.vue --ignore-path .gitignore . --fix\nDone in 1.57s.\n\n🎉  Successfully created project brian-caffey\n\n  To get started:\n\n        yarn dev\n\n  To build & start for production:\n\n        yarn build\n        yarn start\n","txt",[30,48891,48892,48897,48901,48906,48911,48916,48921,48926,48931,48936,48941,48946,48951,48956,48961,48966,48971,48976,48981,48985,48990,48994,48999,49003,49008,49012,49017,49021,49026],{"__ignoreMap":464},[151,48893,48894],{"class":469,"line":470},[151,48895,48896],{},"brian@x1:~/github/briancaffey.github.io/nuxt$ npx create-nuxt-app .\n",[151,48898,48899],{"class":469,"line":488},[151,48900,1090],{"emptyLinePlaceholder":609},[151,48902,48903],{"class":469,"line":500},[151,48904,48905],{},"create-nuxt-app v3.2.0\n",[151,48907,48908],{"class":469,"line":509},[151,48909,48910],{},"✨  Generating Nuxt.js project in .\n",[151,48912,48913],{"class":469,"line":517},[151,48914,48915],{},"(node:8073) ExperimentalWarning: Conditional exports is an experimental feature. This feature could change at any time\n",[151,48917,48918],{"class":469,"line":534},[151,48919,48920],{},"? Project name: brian-caffey\n",[151,48922,48923],{"class":469,"line":1413},[151,48924,48925],{},"? Programming language: JavaScript\n",[151,48927,48928],{"class":469,"line":1418},[151,48929,48930],{},"? Package manager: Yarn\n",[151,48932,48933],{"class":469,"line":2462},[151,48934,48935],{},"? UI framework: Tailwind CSS\n",[151,48937,48938],{"class":469,"line":2471},[151,48939,48940],{},"? Nuxt.js modules: Axios, Content\n",[151,48942,48943],{"class":469,"line":2480},[151,48944,48945],{},"? Linting tools: ESLint, Prettier\n",[151,48947,48948],{"class":469,"line":2489},[151,48949,48950],{},"? Testing framework: None\n",[151,48952,48953],{"class":469,"line":2497},[151,48954,48955],{},"? Rendering mode: Universal (SSR / SSG)\n",[151,48957,48958],{"class":469,"line":3140},[151,48959,48960],{},"? Deployment target: Static (Static/JAMStack hosting)\n",[151,48962,48963],{"class":469,"line":3149},[151,48964,48965],{},"? Development tools: jsconfig.json (Recommended for VS Code if you're not using typescript)\n",[151,48967,48968],{"class":469,"line":3158},[151,48969,48970],{},"yarn run v1.22.5\n",[151,48972,48973],{"class":469,"line":3167},[151,48974,48975],{},"$ eslint --ext .js,.vue --ignore-path .gitignore . --fix\n",[151,48977,48978],{"class":469,"line":3175},[151,48979,48980],{},"Done in 1.57s.\n",[151,48982,48983],{"class":469,"line":3184},[151,48984,1090],{"emptyLinePlaceholder":609},[151,48986,48987],{"class":469,"line":3193},[151,48988,48989],{},"🎉  Successfully created project brian-caffey\n",[151,48991,48992],{"class":469,"line":3720},[151,48993,1090],{"emptyLinePlaceholder":609},[151,48995,48996],{"class":469,"line":3729},[151,48997,48998],{},"  To get started:\n",[151,49000,49001],{"class":469,"line":3735},[151,49002,1090],{"emptyLinePlaceholder":609},[151,49004,49005],{"class":469,"line":3745},[151,49006,49007],{},"        yarn dev\n",[151,49009,49010],{"class":469,"line":3754},[151,49011,1090],{"emptyLinePlaceholder":609},[151,49013,49014],{"class":469,"line":3760},[151,49015,49016],{},"  To build & start for production:\n",[151,49018,49019],{"class":469,"line":3773},[151,49020,1090],{"emptyLinePlaceholder":609},[151,49022,49023],{"class":469,"line":3782},[151,49024,49025],{},"        yarn build\n",[151,49027,49028],{"class":469,"line":3791},[151,49029,49030],{},"        yarn start\n",[11,49032,49033],{},"Running the following:",[459,49035,49038],{"className":49036,"code":49037,"language":997},[995],"yarn generate\nyarn start\n",[30,49039,49037],{"__ignoreMap":464},[11,49041,49042,49043,208],{},"Starts my project on ",[30,49044,49045],{},"localhost:3000",[11,49047,49048],{},[2718,49049],{"alt":49050,"src":49051},"Nuxt app","/static/nuxt-app.png",[56,49053,49055],{"id":49054},"tricky-parts","Tricky parts",[11,49057,49058],{},"Here are some of the parts of migrating that I need to think about:",[76,49060,49061,49081],{},[79,49062,49063,49066,49067,49070,49071,49074,49075,49077,49078,49080],{},[15,49064,49065],{},"Disqus comments",": I previously based the ",[30,49068,49069],{},"disqus_identifier"," used to link Disqus threads to specific pages on Jekyll's page URLs (of the form ",[30,49072,49073],{},"/YYYY/MM/DD/name-of-article.html","). I could transform the slug value with a hook in ",[30,49076,19456],{},", or manually add a ",[30,49079,49069],{}," value to the frontmatter of markdown files.",[79,49082,49083,49086,49087,49090,49091,49093,49094,49097,49098,49101],{},[15,49084,49085],{},"Static Content",": In Jekyll I had static content in a few different places. With Nuxt, all static content will live under the top level folder ",[30,49088,49089],{},"/static"," that is mapped to the root URL where my site is hosted. Since I had lots of content under ",[30,49092,49089],{}," in my Jekyll site, such as ",[30,49095,49096],{},"/static/my-image.png",", I ended up putting conent in ",[30,49099,49100],{},"/static/static",", hopefully this won't be too confusing.",[56,49103,49105],{"id":49104},"nuxt-community","Nuxt Community",[11,49107,49108,49109,49115],{},"There are a lot of great packages from the ",[20,49110,49113],{"href":49111,"rel":49112},"https://github.com/nuxt-community",[24],[30,49114,49104],{}," GitHub organization that are widely used in many Nuxt projects. Here are a few of the Nuxt community plugins and other Nuxt extensions that I have used so far:",[76,49117,49118,49126,49131,49136,49141,49146],{},[79,49119,49120,49125],{},[20,49121,49124],{"href":49122,"rel":49123},"https://content.nuxtjs.org/",[24],"nuxt/content",": an official Nuxt project that provides a Git-based Headless CMS",[79,49127,49128],{},[30,49129,49130],{},"@nuxtjs/tailwindcss",[79,49132,49133],{},[30,49134,49135],{},"@nuxtjs/color-mode",[79,49137,49138],{},[30,49139,49140],{},"@nuxtjs/google-analytics",[79,49142,49143],{},[30,49144,49145],{},"@nuxtjs/sitemap",[79,49147,49148],{},[30,49149,35548],{},[56,49151,49153],{"id":49152},"issues","Issues",[76,49155,49156,49163,49166],{},[79,49157,49158,49159,748],{},"Tailwind removes markdown styles in blog articles from nuxt/content (solved by adding some extra css with ",[20,49160,23252],{"href":49161,"rel":49162},"https://github.com/iandinwoodie/github-markdown-tailwindcss/blob/master/markdown.css",[24],[79,49164,49165],{},"No support for automatically adding markdown anchors (GitHub Flavored Markdown supports this)",[79,49167,49168,49169],{},"Not able to do true server-side redirects, but you can do something like this: ",[20,49170,49171],{"href":49171,"rel":49172},"https://github.com/nuxt-community/redirect-module/issues/1#issuecomment-615070920",[24],[56,49174,21038],{"id":21037},[76,49176,49177],{},[79,49178,49179],{},"Add tags, categories, search to blog",[589,49181,49182],{},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html .sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html.sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}",{"title":464,"searchDepth":488,"depth":488,"links":49184},[49185,49186,49187,49188,49189],{"id":48874,"depth":488,"text":48875},{"id":49054,"depth":488,"text":49055},{"id":49104,"depth":488,"text":49105},{"id":49152,"depth":488,"text":49153},{"id":21037,"depth":488,"text":21038},"2020-09-17",{"layout":48045},"/2020/09/17/migrating-from-jekyll-to-nuxt-static-site",{"title":48852,"description":48857},"2020/09/17/migrating-from-jekyll-to-nuxt-static-site",[11803,12646,8398,49196,49197],"static-site","web","q0L2YEQVHly8ZbIA9Bhi89pfzXNhfNlc3YpO0zrAwkQ",{"id":49200,"title":49201,"body":49202,"comments":609,"date":51669,"description":51670,"draft":602,"extension":605,"external":606,"image":51671,"meta":51672,"navigation":609,"path":51673,"seo":51674,"stem":51675,"tags":51676,"__hash__":51680},"blog/2020/08/09/digital-ocean-docker-swarm-django-traefik-nginx.md","Deploying Django applications with docker swarm on DigitalOcean using GitLab CI, Traefik, NGINX and REX-Ray",{"type":8,"value":49203,"toc":51643},[49204,49225,49228,49233,49240,49243,49245,49248,49286,49307,49311,49339,49343,49363,49366,49369,49372,49376,49382,49422,49426,49439,49443,49446,49452,49458,49462,49469,49472,49478,49489,49492,49498,49506,49511,49514,49519,49536,49557,49566,49571,49665,49678,49697,49730,49736,49748,49788,49794,49804,49843,49851,49855,49873,49879,49894,49913,49917,49920,49926,49931,49951,49954,49960,49967,49976,49982,49992,49998,50007,50012,50024,50154,50187,50213,50232,50238,50249,50252,50257,50265,50271,50276,50287,50293,50308,50310,50429,50437,50442,50448,50467,50492,50498,50520,50529,50565,50687,50702,50705,50708,50714,50725,50730,50839,50849,50864,50870,50900,50906,50916,50928,50934,50970,50976,51094,51097,51107,51117,51123,51126,51132,51148,51164,51167,51172,51181,51184,51195,51198,51203,51206,51220,51229,51246,51249,51433,51438,51449,51497,51503,51508,51514,51519,51525,51531,51540,51546,51549,51555,51558,51561,51567,51576,51580,51583,51592,51597,51600,51602,51605,51609,51612,51616,51619,51623,51626,51630,51633,51637,51640],[11,49205,49206,49207,49212,49213,49218,49219,49224],{},"I recently wrote two articles about deploying Django applications to AWS serverless environments: one on ",[20,49208,49211],{"href":49209,"rel":49210},"https://briancaffey.github.io/2020/06/02/django-postgres-vue-gitlab-ecs.html",[24],"AWS Fargate"," (CloudFront, ALB and ECS Fargate containers) and one on ",[20,49214,49217],{"href":49215,"rel":49216},"https://briancaffey.github.io/2020/08/01/django-and-lambda-with-cdk-and-api-gateway.html",[24],"AWS Lambda"," (Lambda + API Gateway, without using Zappa or Serverless Framework). Both projects focused on automating as much of the setup and operation as possible using DevOps patterns: Infrastructure as Code, GitOps, CI/CD and docker containers. I used AWS resources exclusively (with the exception of GitLab) with the help of ",[20,49220,49223],{"href":49221,"rel":49222},"https://aws.amazon.com/cdk/",[24],"AWS Cloud Development Kit (CDK)",", an awesome tool that I have really come to like. In general I really like AWS, and the more I use it I start to think about what I would do without it. Also, a lot of the feedback I got on these projects recommended to \"just use a VPS\" instead of bothering with AWS because it is complicated, expensive, overkill, etc. This got me thinking about how far I could get in deploying a Django application on a server with little or no external services that AWS has spoiled me with. After a little bit of discomfort and confusion, I was able to check off most of what I was hoping to and came away with a few questions as well. If you are interested to know how I got things setup and and hear some of my thoughts on running Django applications in production, continue reading!",[11,49226,49227],{},"In this article, I'm going to go over my approach to deploying and running Django applications using DigitalOcean Droplets (Linux-based virtual machine that runs on top of virtualized hardware) and block storage volumes (network-based block devices that provide additional data storage for Droplets).",[210,49229,49230],{},[11,49231,49232],{},"I'll also touch on trade-offs between DigitalOcean and AWS and emphasize aspects of the project that confuse/d me with these block quotes.",[11,49234,49235,49236,643],{},"Here's a link to my project that I'll be referencing: ",[20,49237,49238],{"href":49238,"rel":49239},"https://gitlab.com/briancaffey/digital-ocean-docker-swarm-django-traefik",[24],[11,49241,49242],{},"The project setup is a combination of some of the best practices I have picked up along the way as well as some very helpful guides, repositories and blog posts that I'll do my best to reference throughout this article.",[56,49244,40602],{"id":40601},[11,49246,49247],{},"Here are some of the key parts of the project that I'll go over:",[76,49249,49250,49253,49256,49259,49262,49265,49271,49274,49277,49280,49283],{},[79,49251,49252],{},"DigitalOcean and GitLab setup",[79,49254,49255],{},"Creating an A Record that points to our Droplet IP",[79,49257,49258],{},"Using a prebuilt VM image that ships with docker and docker-compose",[79,49260,49261],{},"Setting up the REX-Ray storage driver to automatically provision Digital Ocean block storage volumes",[79,49263,49264],{},"Setting up a docker swarm cluster",[79,49266,49267,49268,49270],{},"Setting up a ",[30,49269,38128],{}," file to build images and push them to a private GitLab CI project registry",[79,49272,49273],{},"Writing a docker-compose file to configure the services (containers) that will support the application",[79,49275,49276],{},"Deploying a stack to the docker swarm cluster on DigitalOcean from our GitLab CI environment over SSH",[79,49278,49279],{},"Django project settings and management commands for our Postgres database and static files",[79,49281,49282],{},"Monitoring, logging and debugging",[79,49284,49285],{},"Destroying the environment + cleanup",[11,49287,49288,49289,49294,49295,49300,49301,49306],{},"Before I dig into all of this, I recommend that you check out ",[20,49290,49293],{"href":49291,"rel":49292},"https://mattsegal.dev/django-prod-architectures.html",[24],"this article about Django production architectures by Matt Segal",". This is a great primer for a lot of what I'll be talking about and it includes some great visualizations. ",[20,49296,49299],{"href":49297,"rel":49298},"https://mattsegal.dev",[24],"mattsegal.dev"," has lots of good content related to Django, I also recommend checking out ",[20,49302,49305],{"href":49303,"rel":49304},"https://mattsegal.dev/nginx-django-reverse-proxy-config.html",[24],"this article about how NGINX is used with Django",". Thanks for the great resources, Matt!",[56,49308,49310],{"id":49309},"digitalocean-setup","DigitalOcean setup",[76,49312,49313,49316,49323,49330],{},[79,49314,49315],{},"Sign up for a new DigitalOcean account if you don't already have one",[79,49317,49318,49319],{},"Create a DigitalOcean project ",[20,49320,49321],{"href":49321,"rel":49322},"https://cloud.digitalocean.com/projects/new",[24],[79,49324,49325,49326],{},"Create a personal access token (we will use this to configure a docker addon that will provision block storage volumes automatically) ",[20,49327,49328],{"href":49328,"rel":49329},"https://cloud.digitalocean.com/account/api/tokens",[24],[79,49331,49332,49333,49338],{},"Create and add an SSH key to your account. This is a pretty simple step, but DigitalOcean still has really thorough documentation on how to do this (see ",[20,49334,49337],{"href":49335,"rel":49336},"https://www.digitalocean.com/docs/droplets/how-to/add-ssh-keys/",[24],"this article"," for more information)",[56,49340,49342],{"id":49341},"prebuilt-docker-vm-image","Prebuilt Docker VM image",[11,49344,49345,49346,49351,49352,49355,49356,49358,49359,49362],{},"From the ",[20,49347,49350],{"href":49348,"rel":49349},"https://cloud.digitalocean.com/droplets/new",[24],"Create Droplets"," page, select ",[30,49353,49354],{},"Marketplace"," and search for ",[30,49357,30129],{},". Select the ",[30,49360,49361],{},"Docker 5:19.03.1~3 18.04"," image. Note that this VM is Ubuntun 18.04 with Docker Community Edition and docker-compose pre-installed.",[11,49364,49365],{},"Select the basic plan, and then scroll to the left to choose the $5.00/month option. Select a datacenter region. Most of these regions should be OK, but you should verify that the region you have selected supports volumes (they may all support volumes, but there are some DO features that are not supported accross all regiongs, similar to AWS). Do not select a VPC or any of the additional options.",[11,49367,49368],{},"For Authentication, select the SSH key that you created earlier.",[11,49370,49371],{},"Take note of the Droplet's IP address; we will use this in the next step.",[56,49373,49375],{"id":49374},"gitlab-setup","GitLab setup",[11,49377,49378,49379,49381],{},"Create a new GitLab project and clone it locally. You can also clone or fork my project and use that as a starting point. Go to ",[30,49380,38283],{}," in your GitLab project and add the following environment variables:",[76,49383,49384,49396,49402,49408,49414],{},[79,49385,49386,49388,49389,49392,49393],{},[30,49387,33109],{},": the value should start with ",[30,49390,49391],{},"-----BEGIN RSA PRIVATE KEY-----"," and end with ",[30,49394,49395],{},"-----END RSA PRIVATE KEY-----",[79,49397,49398,49401],{},[30,49399,49400],{},"DROPLET_IP",": the IP address of the droplet you just created",[79,49403,49404,49407],{},[30,49405,49406],{},"POSTGRES_PASSWORD",": a secure password that we will use for our Postgres database (we will share this with our Django application later on)",[79,49409,49410,49413],{},[30,49411,49412],{},"SECRET_KEY",": a random secret key to use for our Django application.",[79,49415,49416,49419,49420],{},[30,49417,49418],{},"DEBUG",": the number ",[30,49421,9181],{},[56,49423,49425],{"id":49424},"a-record","A Record",[11,49427,49428,49429,49432,49433,49438],{},"By the end of this project you will be able to deploy your Django application to a live domain name provided that you have one. All you need to do is create an A Record that points to the Droplet IP. There are no DNS configuration changes to make inside of DigitalOcean. I'm using a domain that I purchased through Route53. You can get a free ",[30,49430,49431],{},".tk"," domain from ",[20,49434,49437],{"href":49435,"rel":49436},"https://www.freenom.com/en/freeandpaiddomains.html",[24],"freenom",". I have used this before and it is a great option for testing things out.",[56,49440,49442],{"id":49441},"ssh-into-your-digitalocean-droplet","SSH into your DigitalOcean Droplet",[11,49444,49445],{},"You can do this with the following command:",[459,49447,49450],{"className":49448,"code":49449,"language":997},[995],"ssh -i ~/.ssh/a1_rsa root@123.45.578.91\n",[30,49451,49449],{"__ignoreMap":464},[11,49453,49454,49457],{},[30,49455,49456],{},"a1_rsa"," is the private key I added to GitHub. You can logout for now, but keep this command handy, because we will be coming back to our Droplet via SSH shortly.",[56,49459,49461],{"id":49460},"add-the-rex-ray-docker-plugin","Add the REX-Ray docker plugin",[11,49463,49464,49465,643],{},"This step is very simple, you can follow along with this short guide: ",[20,49466,49467],{"href":49467,"rel":49468},"https://www.digitalocean.com/community/questions/how-to-attach-digitalocean-block-storage-to-docker-container",[24],[11,49470,49471],{},"There is basically one command to run:",[459,49473,49476],{"className":49474,"code":49475,"language":997},[995],"docker plugin install rexray/dobs DOBS_TOKEN=YOUR_DIGITALOCEAN_TOKEN DOBS_REGION=nyc1 LINUX_VOLUME_FILEMODE=0775\n",[30,49477,49475],{"__ignoreMap":464},[11,49479,49480,49481,49484,49485,49488],{},"You will need to make sure that you replace ",[30,49482,49483],{},"YOUR_DIGITALOCEAN_TOKEN"," with the personal access token you added earlier. Also, ",[30,49486,49487],{},"DOBS_REGION"," should be the region you selected for your Droplet earlier.",[11,49490,49491],{},"Check that the plugin was installed correctly with:",[459,49493,49496],{"className":49494,"code":49495,"language":997},[995],"docker plugin ls\n",[30,49497,49495],{"__ignoreMap":464},[11,49499,49500,49501,208],{},"Here's a quick intro to REX-Ray from ",[20,49502,49505],{"href":49503,"rel":49504},"https://rexray.readthedocs.io/en/stable/",[24],"rexray.readthedocs.io",[210,49507,49508],{},[11,49509,49510],{},"REX-Ray is an open source, storage management solution designed to support container runtimes such as Docker and Mesos. REX-Ray enables stateful applications, such as databases, to persist and maintain its data after the life cycle of the container has ended. Built-in high availability enables orchestrators such as Docker Swarm, Kubernetes, and Mesos Frameworks like Marathon to automatically orchestrate storage tasks between hosts in a cluster.",[11,49512,49513],{},"In the context of this project, REX-Ray will automate the creation of DigitalOcean Block Storage Volumes. We will talk about volumes and how they are used later on in this article.",[56,49515,49517],{"id":49516},"gitlab-ciyml",[30,49518,38128],{},[11,49520,49521,49523,49524,49529,49530,49535],{},[30,49522,38128],{}," is a file that configures pipelines when code is pushed to GitLab, similar to how GitHub Actions work with GitHub. This single file is a huge topic, if you are unfamiliar with GitLab CI, you might want to have a look over ",[20,49525,49528],{"href":49526,"rel":49527},"https://docs.gitlab.com/ee/ci/yaml/",[24],"this page from the GitLab documentation"," which goes over all of the configuration options with many examples. Also, ",[20,49531,49534],{"href":49532,"rel":49533},"https://docs.gitlab.com/ee/ci/variables/predefined_variables.html",[24],"this documentation page"," covers the predefined environment variables that are made available to GitLab CI pipelines. I'm using these in a few different places as we will see shortly.",[11,49537,49538,49539,49541,49542,106,49545,187,49548,49551,49552,187,49554,49556],{},"CI/CD pipelines that I define with ",[30,49540,38128],{}," typically contain three stages: ",[30,49543,49544],{},"test",[30,49546,49547],{},"build",[30,49549,49550],{},"deploy",". We will focus on the ",[30,49553,49547],{},[30,49555,49550],{}," stages for now (reference the article on my Fargate project linked above for reference on setting up unit tests with pytest).",[11,49558,49559,49562,49563,49565],{},[30,49560,49561],{},"build-backend"," is the name of a GitLab CI job that builds and tags a docker image from the source code in the ",[30,49564,26811],{}," directory of this project and pushes the tagged container image to a private image registry on gitlab.com that we will use later.",[11,49567,49568,49569,208],{},"Here's the YAML code for ",[30,49570,49561],{},[459,49572,49574],{"className":21928,"code":49573,"language":21930,"meta":464,"style":464},"build-backend:\n  stage: build\n  image: docker:19.03.1\n  services:\n    - docker:19.03.5-dind\n  before_script:\n    - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY\n  script:\n    - |\n      docker build \\\n        -t $CI_REGISTRY_IMAGE/backend:$CI_COMMIT_SHORT_SHA \\\n        -f backend/docker/Dockerfile.prod \\\n        ./backend/\n    - docker push $CI_REGISTRY_IMAGE/backend:$CI_COMMIT_SHORT_SHA\n",[30,49575,49576,49582,49590,49599,49606,49613,49619,49626,49632,49638,49643,49648,49653,49658],{"__ignoreMap":464},[151,49577,49578,49580],{"class":469,"line":470},[151,49579,49561],{"class":14368},[151,49581,14372],{"class":503},[151,49583,49584,49586,49588],{"class":469,"line":488},[151,49585,38176],{"class":14368},[151,49587,6208],{"class":503},[151,49589,40159],{"class":481},[151,49591,49592,49594,49596],{"class":469,"line":500},[151,49593,22226],{"class":14368},[151,49595,6208],{"class":503},[151,49597,49598],{"class":481},"docker:19.03.1\n",[151,49600,49601,49604],{"class":469,"line":509},[151,49602,49603],{"class":14368},"  services",[151,49605,14372],{"class":503},[151,49607,49608,49610],{"class":469,"line":517},[151,49609,29541],{"class":503},[151,49611,49612],{"class":481},"docker:19.03.5-dind\n",[151,49614,49615,49617],{"class":469,"line":534},[151,49616,38213],{"class":14368},[151,49618,14372],{"class":503},[151,49620,49621,49623],{"class":469,"line":1413},[151,49622,29541],{"class":503},[151,49624,49625],{"class":481},"docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY\n",[151,49627,49628,49630],{"class":469,"line":1418},[151,49629,38240],{"class":14368},[151,49631,14372],{"class":503},[151,49633,49634,49636],{"class":469,"line":2462},[151,49635,29541],{"class":503},[151,49637,20607],{"class":1869},[151,49639,49640],{"class":469,"line":2471},[151,49641,49642],{"class":481},"      docker build \\\n",[151,49644,49645],{"class":469,"line":2480},[151,49646,49647],{"class":481},"        -t $CI_REGISTRY_IMAGE/backend:$CI_COMMIT_SHORT_SHA \\\n",[151,49649,49650],{"class":469,"line":2489},[151,49651,49652],{"class":481},"        -f backend/docker/Dockerfile.prod \\\n",[151,49654,49655],{"class":469,"line":2497},[151,49656,49657],{"class":481},"        ./backend/\n",[151,49659,49660,49662],{"class":469,"line":3140},[151,49661,29541],{"class":503},[151,49663,49664],{"class":481},"docker push $CI_REGISTRY_IMAGE/backend:$CI_COMMIT_SHORT_SHA\n",[11,49666,49667,49670,49671,49674,49675,49677],{},[30,49668,49669],{},"build-nginx"," is almost identical, but the ",[30,49672,49673],{},"docker build"," arguments are slightly different. There are three arguments for the ",[30,49676,49673],{}," command that I'm using here:",[700,49679,49680,49685,49691],{},[79,49681,49682,49684],{},[30,49683,40260],{},": the tag to tag the built image with",[79,49686,49687,49690],{},[30,49688,49689],{},"-f",": the Dockerfile to be used for building the image",[79,49692,49693,49694,49696],{},"the ",[30,49695,39524],{}," that is sent to the docker daemon when we build the image",[11,49698,49699,49701,49702,187,49705,18952,49708,49711,49712,38574,49715,49717,49718,49721,49722,49724,49725,49727,49728,5389],{},[30,49700,40260],{}," makes use of two predefined GitLab CI variables: ",[30,49703,49704],{},"CI_REGISTRY_IMAGE",[30,49706,49707],{},"CI_COMMIT_SHORT_SHA",[30,49709,49710],{},"$CI_REGISTRY_IMAGE"," is the URL for the private image registry on gitlab.com that we push our images to that is specific to our project: ",[30,49713,49714],{},"registry.gitlab.com/\u003Cgitlab_username>/\u003Cproject_name>",[30,49716,49707],{}," is an character alphanumeric value that contains the truncated name of the commit hash, this is known as the ",[30,49719,49720],{},"tag",", even though we pass in more than just this value. We combine these two values with ",[30,49723,19883],{}," and the name of the image we are building, such as ",[30,49726,26811],{},", so the full value being passed to ",[30,49729,40260],{},[459,49731,49734],{"className":49732,"code":49733,"language":997},[995],"registry.gitlab.com/gitlab-username/my-project/backend:abcd1234\n",[30,49735,49733],{"__ignoreMap":464},[11,49737,49738,49740,49741,49744,49745,49747],{},[30,49739,49689],{}," is the path to the ",[30,49742,49743],{},"Dockerfile"," we are using relative to the directory where we are running the ",[30,49746,49673],{}," command, which is the root directory of the project.",[11,49749,49750,49751,49753,49754,187,49757,49760,49761,187,49763,26792,49766,49768,49769,306,49771,49773,49774,306,49776,49778,49779,49781,49782,49784,49785,49787],{},"The final argument defines the context that we are using to build the image, and this is an important part for understanding how Docker works. This argument defines the directory that is zipped up and sent to the docker daemon via the docker API. When we build an image with ",[30,49752,49673],{},", we are essentially using the docker CLI to make a POST request to our docker daemon (server) where the POST data contains all of the files that we will have access to in the steps of the Dockerfile (such as ",[30,49755,49756],{},"ADD",[30,49758,49759],{},"COPY"," -- we will get to these soon). There's a key difference between the ",[30,49762,26811],{},[30,49764,49765],{},"nginx",[30,49767,49673],{}," commands: the context for ",[30,49770,26811],{},[30,49772,26811],{},", but the context for ",[30,49775,49765],{},[30,49777,643],{}," (the root of the project). This is because we may want access to another top level directory in our project that contains, for example, a Vue.js or React application, that we will build into our NGINX container. In order to be able to access both files in the ",[30,49780,49765],{}," and the folder containing our frontend app, we need to send a context that contains both of these directories. Sending too many files to to the docker daemon when you run docker build will usually cause the ",[30,49783,49673],{}," command to hang. The first line of output from a ",[30,49786,49673],{}," command should be something like this:",[459,49789,49792],{"className":49790,"code":49791,"language":997},[995],"Sending build context to Docker daemon  24.58kB\n",[30,49793,49791],{"__ignoreMap":464},[11,49795,49796,49797,49800,49801,49803],{},"If this number is too high, you should use a ",[30,49798,49799],{},".dockerignore"," file that ignores any files or directories you don't want to send to the docker daemon (similar to how ",[30,49802,12546],{}," works with git).",[11,49805,49806,49807,49810,49811,49814,49815,187,49818,49821,49822,49825,49826,49828,49829,49831,49832,187,49834,49836,49837,49839,49840,49842],{},"To be able to pull and push (read and write) images to our private project container image registry, we need login with our docker client using the ",[30,49808,49809],{},"docker login"," command in the ",[30,49812,49813],{},"before_script"," as well two other predefined GitLab CI variables: ",[30,49816,49817],{},"CI_JOB_TOKEN",[30,49819,49820],{},"CI_REGISTRY",". This all happens using a special service called ",[30,49823,49824],{},"docker-in-docker"," which I won't go into too much detail here, but it is a common practice when working with containers in a CI/CD environment that itself which is also based on containers, such as GitLab CI (each job runs in a container -- the key ",[30,49827,19666],{}," -- and can define additional containers -- the ",[30,49830,15643],{}," key -- to help with the CI job). Once the two images for ",[30,49833,26811],{},[30,49835,49765],{}," have been built and pushed, our GitLab CI pipeline moves on to the next stage: ",[30,49838,49550],{},". In the ",[30,49841,49550],{}," stage, we will start these and other containers on our DigitalOcean droplet, so we are getting close, but there is a lot more to explain. Before we deploy our containers, we need to do some one-time setup:",[700,49844,49845,49848],{},[79,49846,49847],{},"initialize a single-node docker swarm cluster on our Droplet and",[79,49849,49850],{},"create a docker network that our cluster's services (containers) will use",[56,49852,49854],{"id":49853},"setup-a-docker-swarm-cluster","Setup a docker swarm cluster",[11,49856,49857,49858,49861,49862,49864,49865,49868,49869,49872],{},"To setup a docker swarm cluster, SSH into the Droplet with the command we introduced above (",[30,49859,49860],{},"ssh -i ~/.ssh/a1_rsa root@123.45.578.91"," where ",[30,49863,49456],{}," is the name of the private key file -- you can ignore the ",[30,49866,49867],{},"-i ~/.ssh/a1_rsa"," part if you are using an SSH key called ",[30,49870,49871],{},"id_rsa","), and run the following command:",[459,49874,49877],{"className":49875,"code":49876,"language":997},[995],"docker swarm init --advertise-addr DROPLET_IP\n",[30,49878,49876],{"__ignoreMap":464},[11,49880,49881,49882,49884,49885,49888,49889,49893],{},"Replace ",[30,49883,49400],{}," with your Droplet's IP address (e.g. ",[30,49886,49887],{},"123.45.578.91","). Check out ",[20,49890,49337],{"href":49891,"rel":49892},"https://www.digitalocean.com/community/tutorials/how-to-create-a-cluster-of-docker-containers-with-docker-swarm-and-digitalocean-on-ubuntu-16-04",[24]," for some additional information about using docker swarm on GitLab. It is a little bit outdated, but the main ideas should still hold up. Docker swarm is designed to orchestrate containers running on a group (or swarm) of multiple machines. However, it is perfectly fine to run a single-node cluster as we are doing here.",[11,49895,49896,49897,49899,49900,49903,49904,49906,49907,49909,49910,49912],{},"Docker swarm uses docker-compose files, but using docker swarm is very different from running ",[30,49898,47326],{},", a command which you might see people running both locally and in production and which also uses docker-compose files. As a best practice, you should not be using ",[30,49901,49902],{},"docker-compose"," (the command) in production. Many people do this, and several official tutorials will often end with \"now just run ",[30,49905,47326],{}," and you are done\". The first time I ran containers in the cloud I pulled my git repo into a VM, installed docker and docker-compose and ran ",[30,49908,47326],{},". It is pretty easy and it works very similarly in both local and production environments, but this guide will be using ",[30,49911,49902],{}," in production. There is more I could say here, but the main point is that docker swarm is a simplified version of something like Kubernetes, but it comes built-in to docker and is very simple to use.",[56,49914,49916],{"id":49915},"defining-an-overlay-network","Defining an overlay network",[11,49918,49919],{},"SSH into your droplet and run the following command:",[459,49921,49924],{"className":49922,"code":49923,"language":997},[995],"docker network create --driver=overlay traefik-public\n",[30,49925,49923],{"__ignoreMap":464},[210,49927,49928],{},[11,49929,49930],{},"Usually we define networks in our docker-compose file, but this network needs to be defined first and then referenced in our docker-compose file. Here's a thread on SO that goes into a little bit more on why this is necessary, but I still don't have a very clear idea of why this is the case. With another configuration or perhaps docker-compose version, this may not be needed. I'll update this part of the article if I figure anything out.",[11,49932,49933,49934,49937,49938,49940,49941,49944,49945,49950],{},"Let's go over one more docker concept that will helpful in the next few steps. When you run ",[30,49935,49936],{},"docker ps"," on your local machine, the docker CLI first looks to see if the ",[30,49939,47600],{}," environment variable is set. If it is not, then docker defaults to ",[30,49942,49943],{},"unix:///var/run/docker.sock",", a UNIX socket. Check out ",[20,49946,49949],{"href":49947,"rel":49948},"https://stackoverflow.com/questions/35110146/can-anyone-explain-docker-sock",[24],"this SO post"," titled \"Can anyone explain docker.sock?\"",[11,49952,49953],{},"We change the docker host that our local docker CLI is talking to by setting this environment variable, and one nice way to set this environment variables uses an SSH connection:",[459,49955,49958],{"className":49956,"code":49957,"language":997},[995],"DOCKER_HOST=ssh://root@$DOCKER_IP\n",[30,49959,49957],{"__ignoreMap":464},[11,49961,49962,49963,643],{},"See this article for a more in-depth discussion: ",[20,49964,49965],{"href":49965,"rel":49966},"https://www.digitalocean.com/community/tutorials/how-to-use-a-remote-docker-server-to-speed-up-your-workflow",[24],[11,49968,49969,49970,49972,49973,49975],{},"You can try this out locally. Run a container locally, check it with ",[30,49971,49936],{},", then export the ",[30,49974,47600],{}," environment variable with the following command:",[459,49977,49980],{"className":49978,"code":49979,"language":997},[995],"export DOCKER_HOST=ssh://root@123.45.678.91\n",[30,49981,49979],{"__ignoreMap":464},[11,49983,49984,49985,49988,49989,49991],{},"Replacing ",[30,49986,49987],{},"123.45.678.91"," with your Droplet IP. Run ",[30,49990,49936],{}," again and you should see nothing (or any other containers that you started on your Droplet). Finally, run:",[459,49993,49996],{"className":49994,"code":49995,"language":997},[995],"unset DOCKER_HOST\n",[30,49997,49995],{"__ignoreMap":464},[11,49999,50000,50001,50003,50004,10552],{},"Running ",[30,50002,49936],{}," again you should see the containers on your machine. We will be using this idea in the next step when we look at the ",[30,50005,50006],{},"docker stack deploy",[56,50008,50010],{"id":50009},"docker-stack-deploy",[30,50011,50006],{},[11,50013,50014,50015,50017,50018,50020,50021,33226],{},"Now that we have done our one-time-setup steps, let's look at the ",[30,50016,49550],{}," stage of ",[30,50019,38128],{},", the GitLab CI job that will get our containers running on our Droplet. First, let's break down the ",[30,50022,50023],{},"deploy-digital-ocean",[459,50025,50027],{"className":21928,"code":50026,"language":21930,"meta":464,"style":464},"deploy-digital-ocean:\n  stage: deploy\n  image: docker:19.03.1\n  services:\n    - docker:19.03.5-dind\n  variables:\n    DOCKER_HOST: 'ssh://root@$DROPLET_IP'\n  before_script:\n    - apk update && apk add openssh-client bash\n    - mkdir -p ~/.ssh\n    - echo \"$SSH_PRIVATE_KEY\" > ~/.ssh/id_rsa\n    - chmod 600 ~/.ssh/id_rsa\n    - eval \"$(ssh-agent -s)\"\n    - ssh-add ~/.ssh/id_rsa\n    - ssh-keyscan -H $DROPLET_IP >> ~/.ssh/known_hosts\n    - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY\n  script:\n    - docker stack deploy -c stack.yml my-stack --with-registry-auth\n",[30,50028,50029,50035,50043,50051,50057,50063,50070,50080,50086,50093,50100,50107,50114,50121,50128,50135,50141,50147],{"__ignoreMap":464},[151,50030,50031,50033],{"class":469,"line":470},[151,50032,50023],{"class":14368},[151,50034,14372],{"class":503},[151,50036,50037,50039,50041],{"class":469,"line":488},[151,50038,38176],{"class":14368},[151,50040,6208],{"class":503},[151,50042,20676],{"class":481},[151,50044,50045,50047,50049],{"class":469,"line":500},[151,50046,22226],{"class":14368},[151,50048,6208],{"class":503},[151,50050,49598],{"class":481},[151,50052,50053,50055],{"class":469,"line":509},[151,50054,49603],{"class":14368},[151,50056,14372],{"class":503},[151,50058,50059,50061],{"class":469,"line":517},[151,50060,29541],{"class":503},[151,50062,49612],{"class":481},[151,50064,50065,50068],{"class":469,"line":534},[151,50066,50067],{"class":14368},"  variables",[151,50069,14372],{"class":503},[151,50071,50072,50075,50077],{"class":469,"line":1413},[151,50073,50074],{"class":14368},"    DOCKER_HOST",[151,50076,6208],{"class":503},[151,50078,50079],{"class":481},"'ssh://root@$DROPLET_IP'\n",[151,50081,50082,50084],{"class":469,"line":1418},[151,50083,38213],{"class":14368},[151,50085,14372],{"class":503},[151,50087,50088,50090],{"class":469,"line":2462},[151,50089,29541],{"class":503},[151,50091,50092],{"class":481},"apk update && apk add openssh-client bash\n",[151,50094,50095,50097],{"class":469,"line":2471},[151,50096,29541],{"class":503},[151,50098,50099],{"class":481},"mkdir -p ~/.ssh\n",[151,50101,50102,50104],{"class":469,"line":2480},[151,50103,29541],{"class":503},[151,50105,50106],{"class":481},"echo \"$SSH_PRIVATE_KEY\" > ~/.ssh/id_rsa\n",[151,50108,50109,50111],{"class":469,"line":2489},[151,50110,29541],{"class":503},[151,50112,50113],{"class":481},"chmod 600 ~/.ssh/id_rsa\n",[151,50115,50116,50118],{"class":469,"line":2497},[151,50117,29541],{"class":503},[151,50119,50120],{"class":481},"eval \"$(ssh-agent -s)\"\n",[151,50122,50123,50125],{"class":469,"line":3140},[151,50124,29541],{"class":503},[151,50126,50127],{"class":481},"ssh-add ~/.ssh/id_rsa\n",[151,50129,50130,50132],{"class":469,"line":3149},[151,50131,29541],{"class":503},[151,50133,50134],{"class":481},"ssh-keyscan -H $DROPLET_IP >> ~/.ssh/known_hosts\n",[151,50136,50137,50139],{"class":469,"line":3158},[151,50138,29541],{"class":503},[151,50140,49625],{"class":481},[151,50142,50143,50145],{"class":469,"line":3167},[151,50144,38240],{"class":14368},[151,50146,14372],{"class":503},[151,50148,50149,50151],{"class":469,"line":3175},[151,50150,29541],{"class":503},[151,50152,50153],{"class":481},"docker stack deploy -c stack.yml my-stack --with-registry-auth\n",[11,50155,50156,50157,187,50159,50161,50162,50164,50165,313,50167,50170,50171,50173,50174,50177,50178,30583,50181,50183,50184,208],{},"This job uses the same ",[30,50158,19666],{},[30,50160,15643],{}," that our ",[30,50163,49547],{}," stage jobs used. Notice that we set ",[30,50166,47600],{},[30,50168,50169],{},"\"ssh://root@$DROPLET_IP\"",", this means that any docker CLI commands in this job will be communicating with the docker daemon on our Droplet. The ",[30,50172,49813],{}," has a lot going on, but all we are doing is preparing to use the SSH private key that we previously added to our GitLab project's CI/CD environment variables. The base image for this job, ",[30,50175,50176],{},"docker:19.03.1"," is based on Alpine Linus. This version of Linux is super light weight and doesn't come with ",[30,50179,50180],{},"openssh-client",[30,50182,463],{},", so our first step is to install these with the Alpine package manager, ",[30,50185,50186],{},"apk",[459,50188,50191],{"className":50189,"code":50092,"language":50190,"meta":464,"style":464},"language-sh shiki shiki-themes github-light github-dark monokai","sh",[30,50192,50193],{"__ignoreMap":464},[151,50194,50195,50197,50200,50203,50205,50207,50210],{"class":469,"line":470},[151,50196,50186],{"class":473},[151,50198,50199],{"class":481}," update",[151,50201,50202],{"class":503}," && ",[151,50204,50186],{"class":473},[151,50206,2164],{"class":481},[151,50208,50209],{"class":481}," openssh-client",[151,50211,50212],{"class":481}," bash\n",[11,50214,50215,50216,50218,50219,50221,50222,50225,50226,187,50229,208],{},"Next, we add the ",[30,50217,33109],{}," environment variable into the body of the ",[30,50220,49871],{}," private key file, change the permission of this file and then add the key to our SSH agent. Here's an excerpt from ",[30,50223,50224],{},"man ssh-agent"," that provides a little bit more context into why we need to run ",[30,50227,50228],{},"eval \"$(ssh-agent -s)\"",[30,50230,50231],{},"ssh-add ~/.ssh/id_rsa",[459,50233,50236],{"className":50234,"code":50235,"language":997},[995],"DESCRIPTION\n     ssh-agent is a program to hold private keys used for public key authentication (RSA, DSA, ECDSA, Ed25519)ssh-agent is usually started in the beginning of an X-session or a login session, and all other windows or programs are started as clients to the ssh-agent program.  Through use of environment variables the agent can be located and automatically used for authentication when logging in to other machines using ssh(1).\n\n     The agent initially does not have any private keys.  Keys are added using ssh(1) (see AddKeysToAgent in ssh_config(5) for details) or ssh-add(1).  Multiple identities may be stored in ssh-agent concurrently and ssh(1) will automatically use them if present.  ssh-add(1) is also used to remove keys from ssh-agent and to query the keys that are held in one.\n",[30,50237,50235],{"__ignoreMap":464},[11,50239,50240,50241,50244,50245,50248],{},"Next, ",[30,50242,50243],{},"ssh-keyscan -H $DROPLET_IP >> ~/.ssh/known_hosts"," tells our SSH agent about our Droplet so that it doesn't prompt us with a ",[30,50246,50247],{},"Do you want to add this server to known hosts? (yes/no)",", or whatever the equivalent of that is for the docker CLI when it attempts to connect to a remote docker daemon over SSH.",[11,50250,50251],{},"Finally, we login to our our GitLab private registry using the same command from before when we built and pushed images to our private registry on gitlab.com:",[459,50253,50255],{"className":50254,"code":49625,"language":997},[995],[30,50256,49625],{"__ignoreMap":464},[11,50258,50259,50260,187,50262,50264],{},"This essentially gives our DigitalOcean Droplet access to the ",[30,50261,26811],{},[30,50263,49765],{}," images in our private GitLab CI image registry, even tho we are running this command in a contain, in a container that is probably running in Kubernetes on GCP. Next, we are actually going to use these images.",[11,50266,50267,50268,50270],{},"The last command in the ",[30,50269,50023],{}," job is:",[459,50272,50274],{"className":50273,"code":50153,"language":997},[995],[30,50275,50153],{"__ignoreMap":464},[11,50277,50278,50279,18952,50283,50286],{},"Check out this link from the docker docs on docker stacks ",[20,50280,50281],{"href":50281,"rel":50282},"https://docs.docker.com/engine/swarm/stack-deploy/",[24],[30,50284,50285],{},"--with-registry-auth"," is important, our command will complete if this is not included, but our application won't start.",[56,50288,50290],{"id":50289},"stackyml",[30,50291,50292],{},"stack.yml",[11,50294,50295,50296,50298,50299,50301,50302,50304,50305,50307],{},"Now we are ready to tackle the last big file in our repo: ",[30,50297,50292],{},". This is a the docker-compose file that we use to deploy our project. The only reason we needed to run ",[30,50300,49809],{}," above is because ",[30,50303,50292],{}," references the two images we built and pushed to our GitLab private repo. There's a lot going on in this file, let's start with the ",[30,50306,26811],{}," service:",[736,50309,26811],{"id":26811},[459,50311,50313],{"className":21928,"code":50312,"language":21930,"meta":464,"style":464},"version: '3.4'\nservices:\n  backend:\n  image: ${CI_REGISTRY_IMAGE}/backend:${CI_COMMIT_SHORT_SHA}\n  networks:\n    - main\n  environment:\n    - POSTGRES_PASSWORD\n    - SECRET_KEY\n    - DEBUG\n  volumes:\n    - backendassets:/code/assets\n  depends_on:\n    - postgres\n    - redis\n    - web\n",[30,50314,50315,50325,50331,50337,50346,50353,50360,50367,50374,50381,50388,50395,50402,50409,50416,50422],{"__ignoreMap":464},[151,50316,50317,50320,50322],{"class":469,"line":470},[151,50318,50319],{"class":14368},"version",[151,50321,6208],{"class":503},[151,50323,50324],{"class":481},"'3.4'\n",[151,50326,50327,50329],{"class":469,"line":488},[151,50328,15643],{"class":14368},[151,50330,14372],{"class":503},[151,50332,50333,50335],{"class":469,"line":500},[151,50334,27970],{"class":14368},[151,50336,14372],{"class":503},[151,50338,50339,50341,50343],{"class":469,"line":509},[151,50340,22226],{"class":14368},[151,50342,6208],{"class":503},[151,50344,50345],{"class":481},"${CI_REGISTRY_IMAGE}/backend:${CI_COMMIT_SHORT_SHA}\n",[151,50347,50348,50351],{"class":469,"line":517},[151,50349,50350],{"class":14368},"  networks",[151,50352,14372],{"class":503},[151,50354,50355,50357],{"class":469,"line":534},[151,50356,29541],{"class":503},[151,50358,50359],{"class":481},"main\n",[151,50361,50362,50365],{"class":469,"line":1413},[151,50363,50364],{"class":14368},"  environment",[151,50366,14372],{"class":503},[151,50368,50369,50371],{"class":469,"line":1418},[151,50370,29541],{"class":503},[151,50372,50373],{"class":481},"POSTGRES_PASSWORD\n",[151,50375,50376,50378],{"class":469,"line":2462},[151,50377,29541],{"class":503},[151,50379,50380],{"class":481},"SECRET_KEY\n",[151,50382,50383,50385],{"class":469,"line":2471},[151,50384,29541],{"class":503},[151,50386,50387],{"class":481},"DEBUG\n",[151,50389,50390,50393],{"class":469,"line":2480},[151,50391,50392],{"class":14368},"  volumes",[151,50394,14372],{"class":503},[151,50396,50397,50399],{"class":469,"line":2489},[151,50398,29541],{"class":503},[151,50400,50401],{"class":481},"backendassets:/code/assets\n",[151,50403,50404,50407],{"class":469,"line":2497},[151,50405,50406],{"class":14368},"  depends_on",[151,50408,14372],{"class":503},[151,50410,50411,50413],{"class":469,"line":3140},[151,50412,29541],{"class":503},[151,50414,50415],{"class":481},"postgres\n",[151,50417,50418,50420],{"class":469,"line":3149},[151,50419,29541],{"class":503},[151,50421,15686],{"class":481},[151,50423,50424,50426],{"class":469,"line":3158},[151,50425,29541],{"class":503},[151,50427,50428],{"class":481},"web\n",[11,50430,19225,50431,50433,50434,50436],{},[30,50432,19666],{}," is essentially what we defined in ",[30,50435,38128],{},", but the syntax is slightly different:",[459,50438,50440],{"className":50439,"code":50345,"language":997},[995],[30,50441,50345],{"__ignoreMap":464},[11,50443,50444,50445,50447],{},"We pass environment variables that we defined in GitLab CI via the ",[30,50446,28577],{}," key.",[11,50449,50450,50451,50454,50455,50458,50459,50462,50463,50466],{},"The volume ",[30,50452,50453],{},"backendassets"," is used for storing static assets (CSS, JS, etc.) as well as media assets (images, videos, any other file type). We mount this directory at ",[30,50456,50457],{},"/code/assets"," and then define our ",[30,50460,50461],{},"STATIC_ROOT"," in Django's ",[30,50464,50465],{},"settings.py"," to be:",[459,50468,50470],{"className":24401,"code":50469,"language":24403,"meta":464,"style":464},"os.path.join(BASE_DIR, \"assets\", \"static\")\n",[30,50471,50472],{"__ignoreMap":464},[151,50473,50474,50477,50480,50482,50485,50487,50490],{"class":469,"line":470},[151,50475,50476],{"class":503},"os.path.join(",[151,50478,50479],{"class":477},"BASE_DIR",[151,50481,106],{"class":503},[151,50483,50484],{"class":481},"\"assets\"",[151,50486,106],{"class":503},[151,50488,50489],{"class":481},"\"static\"",[151,50491,3640],{"class":503},[11,50493,50494,50495,50497],{},"Later, when we run ",[30,50496,27586],{},", files are copied to this location in our container, and since this is the location of the volume, the files are actually copied to the volume and will be persisted if we destroy the backend container and restart it. When the container restarts, the volume is mounted again and the static files will still be available to our application.",[11,50499,50500,187,50503,50505,50506,50508,50509,50512,50513,50515,50516,50519],{},[30,50501,50502],{},"network",[30,50504,30601],{}," related to to the other services that this application will communicate with. ",[30,50507,26180],{}," is a network defined in the ",[30,50510,50511],{},"networks"," part of ",[30,50514,50292],{},", notice that we reference the ",[30,50517,50518],{},"traefik-public"," network here that we created earlier, as well.",[210,50521,50522],{},[11,50523,50524,50525,50528],{},"Depends on helps with service startup order, but it is a better practice to use ",[30,50526,50527],{},"./wait-for-it.sh",". However, I have never had any issues related to startup order. I'll try adding this later to make things more robust.",[11,50530,50531,187,50533,50535,50536,50538,50539,187,50541,50543,50544,106,50546,187,50548,50550,50551,50553,50554,50557,50558,50561,50562,50564],{},[30,50532,47282],{},[30,50534,27500],{}," will start before ",[30,50537,26811],{},". Our Django application will communicate to these services by their hostnames: ",[30,50540,47282],{},[30,50542,27500],{},". The fact that ",[30,50545,26811],{},[30,50547,47282],{},[30,50549,27500],{}," are all on the same network (",[30,50552,26180],{},") means that they can resolve each other by these names. For example, the connection string to redis will look like: ",[30,50555,50556],{},"redis://redis:6379",". Let's look at the ",[30,50559,50560],{},"DATABASES"," configuration in ",[30,50563,50465],{}," to see how we connect to Postgres:",[459,50566,50568],{"className":24401,"code":50567,"language":24403,"meta":464,"style":464},"DATABASES = {\n    \"default\": {\n        \"ENGINE\": \"django.db.backends.postgresql_psycopg2\",\n        \"NAME\": os.environ.get(\"POSTGRES_NAME\", \"postgres\"),\n        \"USER\": os.environ.get(\"POSTGRES_USERNAME\", \"postgres\"),\n        \"PASSWORD\": os.environ.get(\"POSTGRES_PASSWORD\", \"postgres\"),\n        \"HOST\": os.environ.get(\"POSTGRES_SERVICE_HOST\", \"postgres\"),\n        \"PORT\": os.environ.get(\"POSTGRES_SERVICE_PORT\", 5432),\n    }\n}\n",[30,50569,50570,50578,50585,50597,50614,50630,50646,50662,50679,50683],{"__ignoreMap":464},[151,50571,50572,50574,50576],{"class":469,"line":470},[151,50573,50560],{"class":477},[151,50575,19865],{"class":1869},[151,50577,19833],{"class":503},[151,50579,50580,50583],{"class":469,"line":488},[151,50581,50582],{"class":481},"    \"default\"",[151,50584,21223],{"class":503},[151,50586,50587,50590,50592,50595],{"class":469,"line":500},[151,50588,50589],{"class":481},"        \"ENGINE\"",[151,50591,6208],{"class":503},[151,50593,50594],{"class":481},"\"django.db.backends.postgresql_psycopg2\"",[151,50596,9417],{"class":503},[151,50598,50599,50602,50604,50607,50609,50612],{"class":469,"line":509},[151,50600,50601],{"class":481},"        \"NAME\"",[151,50603,38033],{"class":503},[151,50605,50606],{"class":481},"\"POSTGRES_NAME\"",[151,50608,106],{"class":503},[151,50610,50611],{"class":481},"\"postgres\"",[151,50613,37985],{"class":503},[151,50615,50616,50619,50621,50624,50626,50628],{"class":469,"line":517},[151,50617,50618],{"class":481},"        \"USER\"",[151,50620,38033],{"class":503},[151,50622,50623],{"class":481},"\"POSTGRES_USERNAME\"",[151,50625,106],{"class":503},[151,50627,50611],{"class":481},[151,50629,37985],{"class":503},[151,50631,50632,50635,50637,50640,50642,50644],{"class":469,"line":534},[151,50633,50634],{"class":481},"        \"PASSWORD\"",[151,50636,38033],{"class":503},[151,50638,50639],{"class":481},"\"POSTGRES_PASSWORD\"",[151,50641,106],{"class":503},[151,50643,50611],{"class":481},[151,50645,37985],{"class":503},[151,50647,50648,50651,50653,50656,50658,50660],{"class":469,"line":1413},[151,50649,50650],{"class":481},"        \"HOST\"",[151,50652,38033],{"class":503},[151,50654,50655],{"class":481},"\"POSTGRES_SERVICE_HOST\"",[151,50657,106],{"class":503},[151,50659,50611],{"class":481},[151,50661,37985],{"class":503},[151,50663,50664,50667,50669,50672,50674,50677],{"class":469,"line":1418},[151,50665,50666],{"class":481},"        \"PORT\"",[151,50668,38033],{"class":503},[151,50670,50671],{"class":481},"\"POSTGRES_SERVICE_PORT\"",[151,50673,106],{"class":503},[151,50675,50676],{"class":477},"5432",[151,50678,37985],{"class":503},[151,50680,50681],{"class":469,"line":2462},[151,50682,9461],{"class":503},[151,50684,50685],{"class":469,"line":2471},[151,50686,6274],{"class":503},[11,50688,50689,50690,50693,50694,50697,50698,50701],{},"I have defined the ",[30,50691,50692],{},"HOST"," to be based on the environment variable ",[30,50695,50696],{},"POSTGRES_HOST",", but I have not defined this environment variable, so why didn't I just say ",[30,50699,50700],{},"\"HOST\": \"postgres\"","? I could have, but if I want to change the database in the future, the only change will be adding an environment variable; I won't have to worry about hardcoded values.",[11,50703,50704],{},"I'm choosing to run Postgres in a container and not use a managed database (which DigitalOcean does offer) in order to save on costs and also to get more practice managing my own database. I use RDS with AWS and it handles a lot of things that I don't have to worry about, such as backups, and it allows me to quickly restore from a snapshot. I'm interested in learning more about how I can do these tasks with a database that I run myself.",[736,50706,50707],{"id":49765},"NGINX",[11,50709,50710,50711,50713],{},"The next service we should go over is ",[30,50712,49197],{},", the service that runs the NGINX container that we pushed to our private GitLab image registry. This service has a couple of functions that I'll walk through:",[700,50715,50716,50719,50722],{},[79,50717,50718],{},"Reverse proxy",[79,50720,50721],{},"Serve static files for Django",[79,50723,50724],{},"Serve a frontend Javascript application",[11,50726,50727,50728,208],{},"Here's the definition of this service in ",[30,50729,50292],{},[459,50731,50733],{"className":21928,"code":50732,"language":21930,"meta":464,"style":464},"services:\n  web:\n    image: ${CI_REGISTRY_IMAGE}/nginx:${CI_COMMIT_SHORT_SHA}\n    networks:\n      - traefik-public\n      - main\n    volumes:\n      - backendassets:/usr/src/app/assets\n    deploy:\n      labels:\n        - 'traefik.enable=true'\n        - 'traefik.http.routers.nginx-web.rule=Host(`mysite.com`)'\n        - 'traefik.http.routers.nginx-web.entrypoints=websecure'\n        - 'traefik.http.routers.nginx-web.tls.certresolver=letsencryptresolver'\n        - 'traefik.http.services.nginx-web.loadbalancer.server.port=80'\n",[30,50734,50735,50741,50748,50757,50764,50771,50777,50783,50790,50797,50804,50811,50818,50825,50832],{"__ignoreMap":464},[151,50736,50737,50739],{"class":469,"line":470},[151,50738,15643],{"class":14368},[151,50740,14372],{"class":503},[151,50742,50743,50746],{"class":469,"line":488},[151,50744,50745],{"class":14368},"  web",[151,50747,14372],{"class":503},[151,50749,50750,50752,50754],{"class":469,"line":500},[151,50751,15657],{"class":14368},[151,50753,6208],{"class":503},[151,50755,50756],{"class":481},"${CI_REGISTRY_IMAGE}/nginx:${CI_COMMIT_SHORT_SHA}\n",[151,50758,50759,50762],{"class":469,"line":509},[151,50760,50761],{"class":14368},"    networks",[151,50763,14372],{"class":503},[151,50765,50766,50768],{"class":469,"line":517},[151,50767,14459],{"class":503},[151,50769,50770],{"class":481},"traefik-public\n",[151,50772,50773,50775],{"class":469,"line":534},[151,50774,14459],{"class":503},[151,50776,50359],{"class":481},[151,50778,50779,50781],{"class":469,"line":1413},[151,50780,15667],{"class":14368},[151,50782,14372],{"class":503},[151,50784,50785,50787],{"class":469,"line":1418},[151,50786,14459],{"class":503},[151,50788,50789],{"class":481},"backendassets:/usr/src/app/assets\n",[151,50791,50792,50795],{"class":469,"line":2462},[151,50793,50794],{"class":14368},"    deploy",[151,50796,14372],{"class":503},[151,50798,50799,50802],{"class":469,"line":2471},[151,50800,50801],{"class":14368},"      labels",[151,50803,14372],{"class":503},[151,50805,50806,50808],{"class":469,"line":2480},[151,50807,14430],{"class":503},[151,50809,50810],{"class":481},"'traefik.enable=true'\n",[151,50812,50813,50815],{"class":469,"line":2489},[151,50814,14430],{"class":503},[151,50816,50817],{"class":481},"'traefik.http.routers.nginx-web.rule=Host(`mysite.com`)'\n",[151,50819,50820,50822],{"class":469,"line":2497},[151,50821,14430],{"class":503},[151,50823,50824],{"class":481},"'traefik.http.routers.nginx-web.entrypoints=websecure'\n",[151,50826,50827,50829],{"class":469,"line":3140},[151,50828,14430],{"class":503},[151,50830,50831],{"class":481},"'traefik.http.routers.nginx-web.tls.certresolver=letsencryptresolver'\n",[151,50833,50834,50836],{"class":469,"line":3149},[151,50835,14430],{"class":503},[151,50837,50838],{"class":481},"'traefik.http.services.nginx-web.loadbalancer.server.port=80'\n",[11,50840,50841,50842,50844,50845,50848],{},"For now, ignore the contents under the ",[30,50843,49550],{}," key; we will cover this next when we go over the ",[30,50846,50847],{},"traefik"," service.",[11,50850,50851,50852,30583,50854,50856,50857,50859,50860,50863],{},"NGINX acts as a reverse proxy when it sends request starting with ",[30,50853,30582],{},[30,50855,30586],{}," to the ",[30,50858,26811],{}," container. Two blocks in ",[30,50861,50862],{},"prod.conf"," enable this behavior:",[459,50865,50868],{"className":50866,"code":50867,"language":997},[995],"  upstream backend {\n    server backend:8000;\n  }\n",[30,50869,50867],{"__ignoreMap":464},[11,50871,50872,50873,50875,50876,18952,50878,50880,50881,50883,50884,50886,50887,50889,50890,30583,50892,50894,50895,50897,50898,643],{},"This hostname ",[30,50874,26811],{}," is defined as ",[30,50877,46836],{},[30,50879,46836],{}," can be resolved by the ",[30,50882,49197],{}," service because it is on the same ",[30,50885,26180],{}," network that ",[30,50888,26811],{}," is on. If either of the ",[30,50891,49197],{},[30,50893,26811],{}," wasn't on the ",[30,50896,26180],{}," network, NGINX would not be able to make sense of ",[30,50899,46836],{},[459,50901,50904],{"className":50902,"code":50903,"language":997},[995],"    # backend urls\n    location ~ ^/(admin|api) {\n      proxy_redirect off;\n      proxy_pass http://backend;\n      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;\n      proxy_set_header Host $http_host;\n    }\n",[30,50905,50903],{"__ignoreMap":464},[11,50907,50908,50909,50911,50912,50915],{},"This block does that actual request forwarding. ",[30,50910,47369],{}," references the ",[30,50913,50914],{},"upstream backend {}"," block defined above.",[11,50917,50450,50918,50920,50921,50924,50925,50927],{},[30,50919,50453],{}," is referenced here and mounts to ",[30,50922,50923],{},"/usr/src/app/assets",". This path is then referenced in ",[30,50926,50862],{},", the NGINX configuration file that is used in our custom NGINX-based image:",[459,50929,50932],{"className":50930,"code":50931,"language":997},[995],"    # static files\n    location /static {\n      autoindex on;\n      alias /usr/src/app/assets/static;\n    }\n",[30,50933,50931],{"__ignoreMap":464},[11,50935,50936,50937,50939,50940,50943,50944,50946,50947,50950,50951,50953,50954,50957,50958,50960,50961,50963,50964,50966,50967,50969],{},"In this block of ",[30,50938,50862],{},", we tell NGINX to serve files from ",[30,50941,50942],{},"/usr/src/app/assets/static"," for requests that start with ",[30,50945,49089],{},". A request made to ",[30,50948,50949],{},"https://mysite.com/static/base.css"," would return a file located at ",[30,50952,50942],{}," if that file existed. Remember, when we run the ",[30,50955,50956],{},"collecstatic"," management command in our Django container, it will collect our static files to ",[30,50959,50453],{},". Since ",[30,50962,50453],{}," is mounted to the ",[30,50965,49197],{}," service at ",[30,50968,50923],{},", NGINX will have access to these files by way of the volume mount and they will persist across restarts of the web service and its NGINX container.",[11,50971,50972,50973,50975],{},"Finally, NGINX can serve a Javascript SPA or similar if we choose to use one in our project. To understand how this is done, we need to understand multistage Dockerfiles. Here's the Dockerfile used for the ",[30,50974,49765],{}," container:",[459,50977,50981],{"className":50978,"code":50979,"language":50980,"meta":464,"style":464},"language-dockerfile shiki shiki-themes github-light github-dark monokai","# # build stage\n# FROM node:10-alpine as build-stage\n# WORKDIR /app/\n# COPY frontend/package.json /app/\n# RUN npm cache verify\n# RUN npm install\n# COPY frontend /app/\n# RUN npm run build\n\n# production stage\n# FROM nginx:1.19.1-alpine as production-stage\nFROM nginx:1.19.1-alpine\nCOPY nginx/prod/prod.conf /etc/nginx/nginx.conf\nCOPY nginx/prod/index.html /dist/\n# COPY --from=build-stage /app/dist /dist/\nEXPOSE 80\nCMD [\"nginx\", \"-g\", \"daemon off;\"]\n","dockerfile",[30,50982,50983,50988,50993,50998,51003,51008,51013,51018,51023,51027,51032,51037,51045,51052,51059,51064,51072],{"__ignoreMap":464},[151,50984,50985],{"class":469,"line":470},[151,50986,50987],{"class":1527},"# # build stage\n",[151,50989,50990],{"class":469,"line":488},[151,50991,50992],{"class":1527},"# FROM node:10-alpine as build-stage\n",[151,50994,50995],{"class":469,"line":500},[151,50996,50997],{"class":1527},"# WORKDIR /app/\n",[151,50999,51000],{"class":469,"line":509},[151,51001,51002],{"class":1527},"# COPY frontend/package.json /app/\n",[151,51004,51005],{"class":469,"line":517},[151,51006,51007],{"class":1527},"# RUN npm cache verify\n",[151,51009,51010],{"class":469,"line":534},[151,51011,51012],{"class":1527},"# RUN npm install\n",[151,51014,51015],{"class":469,"line":1413},[151,51016,51017],{"class":1527},"# COPY frontend /app/\n",[151,51019,51020],{"class":469,"line":1418},[151,51021,51022],{"class":1527},"# RUN npm run build\n",[151,51024,51025],{"class":469,"line":2462},[151,51026,1090],{"emptyLinePlaceholder":609},[151,51028,51029],{"class":469,"line":2471},[151,51030,51031],{"class":1527},"# production stage\n",[151,51033,51034],{"class":469,"line":2480},[151,51035,51036],{"class":1527},"# FROM nginx:1.19.1-alpine as production-stage\n",[151,51038,51039,51042],{"class":469,"line":2489},[151,51040,51041],{"class":1869},"FROM",[151,51043,51044],{"class":503}," nginx:1.19.1-alpine\n",[151,51046,51047,51049],{"class":469,"line":2497},[151,51048,49759],{"class":1869},[151,51050,51051],{"class":503}," nginx/prod/prod.conf /etc/nginx/nginx.conf\n",[151,51053,51054,51056],{"class":469,"line":3140},[151,51055,49759],{"class":1869},[151,51057,51058],{"class":503}," nginx/prod/index.html /dist/\n",[151,51060,51061],{"class":469,"line":3149},[151,51062,51063],{"class":1527},"# COPY --from=build-stage /app/dist /dist/\n",[151,51065,51066,51069],{"class":469,"line":3158},[151,51067,51068],{"class":1869},"EXPOSE",[151,51070,51071],{"class":503}," 80\n",[151,51073,51074,51077,51079,51082,51084,51087,51089,51092],{"class":469,"line":3167},[151,51075,51076],{"class":1869},"CMD",[151,51078,6604],{"class":503},[151,51080,51081],{"class":481},"\"nginx\"",[151,51083,106],{"class":503},[151,51085,51086],{"class":481},"\"-g\"",[151,51088,106],{"class":503},[151,51090,51091],{"class":481},"\"daemon off;\"",[151,51093,3691],{"class":503},[11,51095,51096],{},"Currently I don't have a SPA setup, but this is how we could setup one using Vue.js. The important part is this line:",[459,51098,51101],{"className":51099,"code":51100,"language":49743,"meta":464,"style":464},"language-Dockerfile shiki shiki-themes github-light github-dark monokai","COPY --from=build-stage /app/dist /dist/\n",[30,51102,51103],{"__ignoreMap":464},[151,51104,51105],{"class":469,"line":470},[151,51106,51100],{},[11,51108,51109,51110,51113,51114,51116],{},"This would copy the build files for our Javascript application into the ",[30,51111,51112],{},"/dist/"," folder of our NGINX container. Another few declarations and blocks in ",[30,51115,50862],{}," allow all other requests to be served by the contents of this folder:",[459,51118,51121],{"className":51119,"code":51120,"language":997},[995],"    root /dist/;\n    index index.html;\n",[30,51122,51120],{"__ignoreMap":464},[11,51124,51125],{},"This sets the root and the index document for our NGINX webserver.",[459,51127,51130],{"className":51128,"code":51129,"language":997},[995],"    # frontend\n    location / {\n      try_files $uri $uri/ @rewrites;\n    }\n\n    location @rewrites {\n      rewrite ^(.+)$ /index.html last;\n    }\n",[30,51131,51129],{"__ignoreMap":464},[11,51133,51134,51135,51137,51138,51140,51141,106,51143,30583,51145,51147],{},"These two blocks route all other requests to the frontend Javascript app's ",[30,51136,46303],{}," file location in ",[30,51139,51112],{}," (any request that doesn't start with ",[30,51142,49089],{},[30,51144,30582],{},[30,51146,30586],{},"). We may wish to change this behavior if you want Django to serve most of your requests and possibly serve a single page application on another path.",[11,51149,51150,51151,51153,51154,51157,51158,51161,51162,50848],{},"Lastly, the ",[30,51152,49197],{}," service's ",[30,51155,51156],{},"deployment"," key has some ",[30,51159,51160],{},"labels"," defined for Traefik. Let's come back to these after having a look at the ",[30,51163,50847],{},[736,51165,51166],{"id":50847},"Traefik",[11,51168,51169],{},[2718,51170],{"alt":20386,"src":51171},"https://docs.traefik.io/assets/img/traefik-architecture.png",[210,51173,51174],{},[11,51175,51176,51177],{},"Traefik is an open-source Edge Router that makes publishing your services a fun and easy experience. It receives requests on behalf of your system and finds out which components are responsible for handling them. -- ",[20,51178,51179],{"href":51179,"rel":51180},"https://docs.traefik.io/",[24],[11,51182,51183],{},"Traefik has three main functions in my application:",[700,51185,51186,51189,51192],{},[79,51187,51188],{},"Request TLS certificates from Let's Encrypt that allow us to encrypt our web traffic with HTTPS",[79,51190,51191],{},"Do TLS termination",[79,51193,51194],{},"Route all requests to NGINX",[11,51196,51197],{},"The one thing that Traefik cannot do is serve static files; it is not a webserver, unlike NGINX which is a webserver. NGINX is also capable of requesting TLS certs from Let's Encrypt, so we don't technically need Traefik, but it is indeed \"fun and easy\", especially when it comes to requesting certificates.",[210,51199,51200],{},[11,51201,51202],{},"I have tried setting up Certbot with NGINX a long time ago but I never go it to work, and I didn't like the idea about how to run a chron job to refresh old certs.",[11,51204,51205],{},"There are two main ways to set up Traefik:",[700,51207,51208,51217],{},[79,51209,51210,51211,51214,51215,748],{},"write a ",[30,51212,51213],{},"traefik.toml"," file and build this into your own custom image (similar to what we do with NGINX and ",[30,51216,50862],{},[79,51218,51219],{},"use a base image and specify all configure options through command line arguments.",[11,51221,51222,51223,51225,51226,51228],{},"I started out with the first approach, and I did get it to work, but I have decided that the second way is better. It requires one less image to build in our deployment process and it is easy to parametrize the command line arguments in ",[30,51224,50292],{}," (for now all the values I'm using in the ",[30,51227,50847],{}," service are hard-coded, this is one more item for my ToDo list on this project).",[11,51230,51231,51232,51234,51235,51240,51241,12282],{},"I had a hard time finding good examples of how to use Traefik version 2 with Docker Swarm in the Traefik docs. Their official example for using docker uses ",[30,51233,47326],{},". There is a Swarm example, but it is for an older version of Traefik (1.7). This article titled ",[20,51236,51239],{"href":51237,"rel":51238},"https://blog.creekorful.com/2019/10/how-to-install-traefik-2-docker-swarm/",[24],"How to install Traefik 2.x on a Docker Swarm"," helped me a lot in figuring out how to get everything working. Thank you for the great article, ",[20,51242,51245],{"href":51243,"rel":51244},"https://github.com/creekorful",[24],"Aloïs",[11,51247,51248],{},"Here's the code that sets up the traefik service:",[459,51250,51252],{"className":21928,"code":51251,"language":21930,"meta":464,"style":464},"services:\n  traefik:\n    image: traefik:v2.0.2\n    ports:\n      - '80:80'\n      - '443:443'\n    command:\n      - '--providers.docker.endpoint=unix:///var/run/docker.sock'\n      - '--providers.docker.swarmMode=true'\n      - '--providers.docker.exposedbydefault=false'\n      - '--providers.docker.network=traefik-public'\n      - '--entrypoints.web.address=:80'\n      - '--entrypoints.websecure.address=:443'\n      - '--certificatesresolvers.letsencryptresolver.acme.httpchallenge=true'\n      - '--certificatesresolvers.letsencryptresolver.acme.httpchallenge.entrypoint=web'\n      - '--certificatesresolvers.letsencryptresolver.acme.email=brian@email.com'\n      - '--certificatesresolvers.letsencryptresolver.acme.storage=/letsencrypt/acme.json'\n    volumes:\n      - letsencrypt:/letsencrypt\n      - /var/run/docker.sock:/var/run/docker.sock\n    networks:\n      - traefik-public\n    deploy:\n      placement:\n        constraints:\n          - node.role == manager\n",[30,51253,51254,51260,51267,51276,51282,51289,51296,51303,51310,51317,51324,51331,51338,51345,51352,51359,51366,51373,51379,51386,51393,51399,51405,51411,51418,51425],{"__ignoreMap":464},[151,51255,51256,51258],{"class":469,"line":470},[151,51257,15643],{"class":14368},[151,51259,14372],{"class":503},[151,51261,51262,51265],{"class":469,"line":488},[151,51263,51264],{"class":14368},"  traefik",[151,51266,14372],{"class":503},[151,51268,51269,51271,51273],{"class":469,"line":500},[151,51270,15657],{"class":14368},[151,51272,6208],{"class":503},[151,51274,51275],{"class":481},"traefik:v2.0.2\n",[151,51277,51278,51280],{"class":469,"line":509},[151,51279,15691],{"class":14368},[151,51281,14372],{"class":503},[151,51283,51284,51286],{"class":469,"line":517},[151,51285,14459],{"class":503},[151,51287,51288],{"class":481},"'80:80'\n",[151,51290,51291,51293],{"class":469,"line":534},[151,51292,14459],{"class":503},[151,51294,51295],{"class":481},"'443:443'\n",[151,51297,51298,51301],{"class":469,"line":1413},[151,51299,51300],{"class":14368},"    command",[151,51302,14372],{"class":503},[151,51304,51305,51307],{"class":469,"line":1418},[151,51306,14459],{"class":503},[151,51308,51309],{"class":481},"'--providers.docker.endpoint=unix:///var/run/docker.sock'\n",[151,51311,51312,51314],{"class":469,"line":2462},[151,51313,14459],{"class":503},[151,51315,51316],{"class":481},"'--providers.docker.swarmMode=true'\n",[151,51318,51319,51321],{"class":469,"line":2471},[151,51320,14459],{"class":503},[151,51322,51323],{"class":481},"'--providers.docker.exposedbydefault=false'\n",[151,51325,51326,51328],{"class":469,"line":2480},[151,51327,14459],{"class":503},[151,51329,51330],{"class":481},"'--providers.docker.network=traefik-public'\n",[151,51332,51333,51335],{"class":469,"line":2489},[151,51334,14459],{"class":503},[151,51336,51337],{"class":481},"'--entrypoints.web.address=:80'\n",[151,51339,51340,51342],{"class":469,"line":2497},[151,51341,14459],{"class":503},[151,51343,51344],{"class":481},"'--entrypoints.websecure.address=:443'\n",[151,51346,51347,51349],{"class":469,"line":3140},[151,51348,14459],{"class":503},[151,51350,51351],{"class":481},"'--certificatesresolvers.letsencryptresolver.acme.httpchallenge=true'\n",[151,51353,51354,51356],{"class":469,"line":3149},[151,51355,14459],{"class":503},[151,51357,51358],{"class":481},"'--certificatesresolvers.letsencryptresolver.acme.httpchallenge.entrypoint=web'\n",[151,51360,51361,51363],{"class":469,"line":3158},[151,51362,14459],{"class":503},[151,51364,51365],{"class":481},"'--certificatesresolvers.letsencryptresolver.acme.email=brian@email.com'\n",[151,51367,51368,51370],{"class":469,"line":3167},[151,51369,14459],{"class":503},[151,51371,51372],{"class":481},"'--certificatesresolvers.letsencryptresolver.acme.storage=/letsencrypt/acme.json'\n",[151,51374,51375,51377],{"class":469,"line":3175},[151,51376,15667],{"class":14368},[151,51378,14372],{"class":503},[151,51380,51381,51383],{"class":469,"line":3184},[151,51382,14459],{"class":503},[151,51384,51385],{"class":481},"letsencrypt:/letsencrypt\n",[151,51387,51388,51390],{"class":469,"line":3193},[151,51389,14459],{"class":503},[151,51391,51392],{"class":481},"/var/run/docker.sock:/var/run/docker.sock\n",[151,51394,51395,51397],{"class":469,"line":3720},[151,51396,50761],{"class":14368},[151,51398,14372],{"class":503},[151,51400,51401,51403],{"class":469,"line":3729},[151,51402,14459],{"class":503},[151,51404,50770],{"class":481},[151,51406,51407,51409],{"class":469,"line":3735},[151,51408,50794],{"class":14368},[151,51410,14372],{"class":503},[151,51412,51413,51416],{"class":469,"line":3745},[151,51414,51415],{"class":14368},"      placement",[151,51417,14372],{"class":503},[151,51419,51420,51423],{"class":469,"line":3754},[151,51421,51422],{"class":14368},"        constraints",[151,51424,14372],{"class":503},[151,51426,51427,51430],{"class":469,"line":3760},[151,51428,51429],{"class":503},"          - ",[151,51431,51432],{"class":481},"node.role == manager\n",[210,51434,51435],{},[11,51436,51437],{},"The one thing I don't like about this setup is that it uses a 1GB volume to store one small JSON file. I think that 1GB is the smallest block storage volume I can request using REX-Ray. This only adds $0.10/month to our project costs which is not that bad.",[11,51439,51440,187,51442,51445,51446,51448],{},[30,51441,49197],{},[30,51443,51444],{},"websecure"," refer to values declared on the ",[30,51447,49197],{}," service. Let's take a look at those values:",[459,51450,51452],{"className":14359,"code":51451,"language":14361,"meta":464,"style":464},"deploy:\n  labels:\n    - 'traefik.enable=true'\n    - 'traefik.http.routers.nginx-web.rule=Host(`mysite.com`)'\n    - 'traefik.http.routers.nginx-web.entrypoints=websecure'\n    - 'traefik.http.routers.nginx-web.tls.certresolver=letsencryptresolver'\n    - 'traefik.http.services.nginx-web.loadbalancer.server.port=80'\n",[30,51453,51454,51460,51467,51473,51479,51485,51491],{"__ignoreMap":464},[151,51455,51456,51458],{"class":469,"line":470},[151,51457,49550],{"class":14368},[151,51459,14372],{"class":503},[151,51461,51462,51465],{"class":469,"line":488},[151,51463,51464],{"class":14368},"  labels",[151,51466,14372],{"class":503},[151,51468,51469,51471],{"class":469,"line":500},[151,51470,29541],{"class":503},[151,51472,50810],{"class":481},[151,51474,51475,51477],{"class":469,"line":509},[151,51476,29541],{"class":503},[151,51478,50817],{"class":481},[151,51480,51481,51483],{"class":469,"line":517},[151,51482,29541],{"class":503},[151,51484,50824],{"class":481},[151,51486,51487,51489],{"class":469,"line":534},[151,51488,29541],{"class":503},[151,51490,50831],{"class":481},[151,51492,51493,51495],{"class":469,"line":1413},[151,51494,29541],{"class":503},[151,51496,50838],{"class":481},[11,51498,51499,51500,51502],{},"I still need to setup HTTP -> HTTPS redirecting, so for now only ",[30,51501,51444],{}," is defined, but Aloïs explains this clearly in his article.",[210,51504,51505],{},[11,51506,51507],{},"For me this is the most complicated part of the setup. I'm still not familiar with exactly how Traefik and Let's Encrypt work. Hopefully I can run through this process a few more times with some variations to better understand the rough edges. Otherwise for this simple way to get TLS certificates. AWS makes this very easy with Amazon Certificate Manager (ACM) which makes the requesting of certificates very simple, especially within CDK.",[11,51509,51510,51511,51513],{},"That wraps up our overview of ",[30,51512,50292],{},", we left off with the following command:",[459,51515,51517],{"className":51516,"code":50153,"language":997},[995],[30,51518,50153],{"__ignoreMap":464},[11,51520,51521,51522,51524],{},"We can check out the status of our docker stack deployment by running a few different docker CLI commands. You can either SSH into your Droplet or configure the ",[30,51523,47600],{}," environment variable that I showed you earlier and run these commands from your local terminal:",[459,51526,51529],{"className":51527,"code":51528,"language":997},[995],"docker stack ps my-stack --no-trunc\n",[30,51530,51528],{"__ignoreMap":464},[11,51532,51533,51536,51537,643],{},[30,51534,51535],{},"--no-trunc"," is important because important error messages tend to be cut off. This option will show the full version of each column returned by ",[30,51538,51539],{},"docker stack ps",[459,51541,51544],{"className":51542,"code":51543,"language":997},[995],"docker service ls\n",[30,51545,51543],{"__ignoreMap":464},[11,51547,51548],{},"This command shows some the active services on our Droplet.",[459,51550,51553],{"className":51551,"code":51552,"language":997},[995],"docker ps\n",[30,51554,51552],{"__ignoreMap":464},[11,51556,51557],{},"This command is useful for shelling into a container to run commands and poke around for debugging.",[11,51559,51560],{},"You can get the Container ID of a running container and access it with the following command:",[459,51562,51565],{"className":51563,"code":51564,"language":997},[995],"docker exec -it 0da8370ab283 bash\n",[30,51566,51564],{"__ignoreMap":464},[11,51568,51569,51570,51572,51573,643],{},"This assumes that ",[30,51571,463],{}," is installed on the container with ID ",[30,51574,51575],{},"0da8370ab283",[56,51577,51579],{"id":51578},"management-commands","Management commands",[11,51581,51582],{},"Once the site is deployed we stil need to run a few commands to set up our Djnago application:",[700,51584,51585,51587,51589],{},[79,51586,27586],{},[79,51588,27589],{},[79,51590,51591],{},"createsuperuser",[210,51593,51594],{},[11,51595,51596],{},"One other ToDo is to figure out how to run these commands through manual GitLab CI jobs.",[11,51598,51599],{},"That's most of what I wanted to cover on a first pass. This should be a good starting point for working with a Django application in Docker Swarm on DigitalOcean.",[56,51601,47504],{"id":47138},[11,51603,51604],{},"Here are some ideas about the next steps I could take on this project.",[736,51606,51608],{"id":51607},"local-environment","Local environment",[11,51610,51611],{},"I'll probably need to create another docker-compose file to bring up everything locally. It might be a good way to experiment with different Traefik settings.",[736,51613,51615],{"id":51614},"infrastructure-as-code-setup","Infrastructure as Code setup",[11,51617,51618],{},"There might be a good opportunity to learn more about Terreform or Ansible here. There are a number of manual, one time setup steps. Some of these can't be automated, but some of them would probably fit very neatly into one of these tools. Pulumi would also be a good option to explore as it is more analagous to CDK.",[736,51620,51622],{"id":51621},"scaling-out-docker-swarm-services-across-multiple-machines","Scaling out docker swarm services across multiple machines",[11,51624,51625],{},"I have only scratched the surface of what docker swarm can do. There are lots of other settings that would be helpful to setup for learning purposes, especially around resource limits for services. For simplicity I haven't touch on any of these options yet. I'm curious to know how many containers I can fit onto one small Droplet, and if resource limits could help with compute and memory-intensive workloads.",[736,51627,51629],{"id":51628},"kubernetes-on-digitalocean","Kubernetes on DigitalOcean",[11,51631,51632],{},"DigitalOcean now offers simplified Kubernetes solutions. It would be interesting to try this out once I get better with docker swarm. I have used Kubernetes a with minikube and to a limited extent with GCP.",[736,51634,51636],{"id":51635},"deploying-locally-without-using-gitlab-ci","Deploying locally, without using GitLab CI",[11,51638,51639],{},"CDK makes deploying locally very easy, especially with the Lambda project I put together. This project as it stands might be a little bit more difficult to deploy locally. It assumes that the images we want to deploy are private. It might be possible, but for now I am fine with deploying through GitLab CI since the pipeline only takes a few minutes to complete for the build and deploy stages.",[589,51641,51642],{},"html pre.shiki code .s5clZ, html code.shiki .s5clZ{--shiki-default:#22863A;--shiki-dark:#85E89D;--shiki-sepia:#F92672}html pre.shiki code .sMOD_, html code.shiki .sMOD_{--shiki-default:#24292E;--shiki-dark:#E1E4E8;--shiki-sepia:#F8F8F2}html pre.shiki code .sstjo, html code.shiki .sstjo{--shiki-default:#032F62;--shiki-dark:#9ECBFF;--shiki-sepia:#E6DB74}html pre.shiki code .sC2Qs, html code.shiki .sC2Qs{--shiki-default:#D73A49;--shiki-dark:#F97583;--shiki-sepia:#F92672}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html .sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html.sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html pre.shiki code .srTi1, html code.shiki .srTi1{--shiki-default:#6F42C1;--shiki-dark:#B392F0;--shiki-sepia:#A6E22E}html pre.shiki code .s7F3e, html code.shiki .s7F3e{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#AE81FF}html pre.shiki code .s8-w5, html code.shiki .s8-w5{--shiki-default:#6A737D;--shiki-dark:#6A737D;--shiki-sepia:#88846F}",{"title":464,"searchDepth":488,"depth":488,"links":51644},[51645,51646,51647,51648,51649,51650,51651,51652,51653,51654,51655,51656,51661,51662],{"id":40601,"depth":488,"text":40602},{"id":49309,"depth":488,"text":49310},{"id":49341,"depth":488,"text":49342},{"id":49374,"depth":488,"text":49375},{"id":49424,"depth":488,"text":49425},{"id":49441,"depth":488,"text":49442},{"id":49460,"depth":488,"text":49461},{"id":49516,"depth":488,"text":38128},{"id":49853,"depth":488,"text":49854},{"id":49915,"depth":488,"text":49916},{"id":50009,"depth":488,"text":50006},{"id":50289,"depth":488,"text":50292,"children":51657},[51658,51659,51660],{"id":26811,"depth":500,"text":26811},{"id":49765,"depth":500,"text":50707},{"id":50847,"depth":500,"text":51166},{"id":51578,"depth":488,"text":51579},{"id":47138,"depth":488,"text":47504,"children":51663},[51664,51665,51666,51667,51668],{"id":51607,"depth":500,"text":51608},{"id":51614,"depth":500,"text":51615},{"id":51621,"depth":500,"text":51622},{"id":51628,"depth":500,"text":51629},{"id":51635,"depth":500,"text":51636},"2020-08-09","A guide to deploying Django applications with docker using popular open-source tools","/static/shark.jpg",{"layout":48045},"/2020/08/09/digital-ocean-docker-swarm-django-traefik-nginx",{"title":49201,"description":51670},"2020/08/09/digital-ocean-docker-swarm-django-traefik-nginx",[30122,30129,51677,12646,47672,51678,50847,49765,51679],"digital-ocean","rex-ray","swarm","UGRvIwMyOl0yC8tH05qTbiq6HZGShacystGuGK7L5Y8",{"id":51682,"title":51683,"body":51684,"comments":609,"date":54423,"description":464,"draft":602,"extension":605,"external":606,"image":51695,"meta":54424,"navigation":609,"path":54425,"seo":54426,"stem":54427,"tags":54428,"__hash__":54429},"blog/2020/08/01/django-and-lambda-with-cdk-and-api-gateway.md","Deploying serverless Django applications to AWS with CDK on a tiny budget using Lambda, API Gateway, awsgi and the Lambda proxy pattern",{"type":8,"value":51685,"toc":54408},[51686,51688,51691,51696,51699,51702,51705,51713,51716,51723,51732,51736,51758,51856,51865,51888,52043,52053,52055,52062,52114,52140,52143,52150,52153,52194,52197,52324,52327,52552,52596,52605,52613,52658,52674,52809,52812,52815,52818,53024,53031,53054,53059,53290,53293,53337,53341,53361,53364,53524,53527,53544,53547,53713,53720,53726,53746,53762,53776,53799,53803,53810,53835,53845,53851,53855,53861,53867,53879,53897,53914,53931,53935,53938,53946,53950,53957,53963,53969,53991,53998,54004,54010,54016,54019,54025,54031,54038,54044,54051,54056,54059,54069,54074,54078,54081,54194,54201,54332,54339,54341,54344,54403,54405],[56,51687,16116],{"id":16115},[11,51689,51690],{},"One way to deploy Django apps to AWS on a budget is to use the Lambda proxy pattern. The main idea is this: web requests are made to API Gateway that calls a Lambda outside of our VPC (proxy Lambda), which then invokes a second lambda (Django Lambda) inside of our VPC so that it can access VPC resources, namely RDS. The handler for the Django Lambda translates the API Gateway event into a WSGI-compatible request, processes the request and returns the response to the user back through the proxy Lambda. The big caveat is that you can't easily access the internet in Django's request/response cycle without network address translation (NAT) services which add additional costs. One other caveat is that there is high latency associated with the initial request. There are three cold starts to wait for: the cold start for the proxy Lambda, the cold start for the Django Lambda and the cold start for the Aurora Postgres Serverless database. Here's an architecture diagram:",[11,51692,51693],{},[2718,51694],{"alt":20386,"src":51695},"/static/djambda.png",[11,51697,51698],{},"Route53 and CloudFront (1 and 2), discuessed later on, are optional. Starting with 5, API Gateway requests are passed to a proxy lambda (6) which calls a Lambda in a VPC that contains our Django code and special Django handler (7). This function can access RDS (8) and S3 (4) via a VPC Gateway Endpoint (9), but it cannot access the internet.",[56,51700,51701],{"id":642},"Why?",[11,51703,51704],{},"There are a few different reasons for why I put this project together. Before I get into these reasons, here is a little bit of background on how I typically deploy web apps and the motivations for this project.",[11,51706,51707,51708,51712],{},"Django is my web framework of choice, and I'm most comfortable working with Django in docker containers. I previously wrote an article about ",[20,51709,51711],{"href":49209,"rel":51710},[24],"deploying Django applications with AWS Fargate",". This approach technically is \"serverless\", but it requires the use of \"always on\" services: an Application Load Balancer, NAT Gateways and long-running Fargate tasks to run gunicorn processes for our Django application. Even if you choose to run workloads in public subnets and don't use NAT Gateways or NAT instances, the costs are prohbitively high for many types of Django projects: personal projects, proof-of-concept projects, internal projects, toy projects, etc.",[11,51714,51715],{},"The other serverless compute platform on AWS is Lambda. There is a popular framework for deploying serverless Django applications on Lambda called Zappa. I have been meaning to try out Zappa for a while now, and I still haven't used it, but it provided a good reference for how to do \"serverless Django\". I'll come back to Zappa in more detail later, but the main reason I didn't want to use it for my serverless Django project is because I already have a great deployment and Infrastructure as Code tool: CDK.",[11,51717,51718,51719,643],{},"CDK, or Cloud Development Kit, is a tool that allows you define and deploy AWS infrastructure using any popular programming language, Python in my case. It uses CloudFormation in the background, and it has great support for lots of AWS services. If you are looking for an introuction to CDK, check out ",[20,51720,51721],{"href":51721,"rel":51722},"https://cdkworkshop.com/",[24],[11,51724,51725,51726,51731],{},"Zappa makes it really easy to deploy serveless Django applications with Django, but it requires that you ",[20,51727,51730],{"href":51728,"rel":51729},"https://github.com/Miserlou/Zappa#running-tasks-in-a-vpc",[24],"setup NAT and other VPC resources before using it with RDS",". Since CDK is really good at setting up these kinds of resources, as well as the resources that Zappa creates (including Lambda functions and API Gateway), I wanted to know how far I could get in setting up a servless Django application only using CDK. I also didn't want everything abstracted away by a high level framework; I'm interested in a lower level view to get more familiar with how serverless technologies work. It's important to note that the biggest motivation of all is to learn try something out of my comfort zone, make mistakes and get feedback.",[56,51733,51735],{"id":51734},"django-lambda-djambda","Django + Lambda = djambda",[11,51737,51738,51739,51744,51745,26802,51750,51753,51754,51757],{},"The name for this project came pretty easily. I created ",[20,51740,51743],{"href":51741,"rel":51742},"https://gitlab.com/briancaffey/djambda",[24],"a repo on GitLab called djambda"," and then discovered ",[20,51746,51749],{"href":51747,"rel":51748},"https://github.com/netsome/djambda",[24],"a similar project by the same name on GitHub: netsome/djambda",[30,51751,51752],{},"netsome/djambda"," project uses Terraform, and I encourage you to check it out and leave it a GitHub star. Before I came across this project, I hacked together a very basic djambda protoype using some code from Zappa's handler. A ",[30,51755,51756],{},"handler"," is the name for the function that is called in your Lambda function's code when the Lambda function is invoked. Here's some pseudo code that shows how Zappa's handler works:",[459,51759,51761],{"className":24401,"code":51760,"language":24403,"meta":464,"style":464},"from werkzeug.wrappers import Response\n\ndef handler(event, context):\n    with Response.from_app(wsgi_app, wsgi_request(event)) as response:\n        zappa_returndict = dict()\n        zappa_returndict['body'] = response.data\n        zappa_returndict['statusCode'] = response.status_code\n        return zappa_returndict\n",[30,51762,51763,51775,51779,51795,51808,51820,51835,51849],{"__ignoreMap":464},[151,51764,51765,51767,51770,51772],{"class":469,"line":470},[151,51766,16853],{"class":1869},[151,51768,51769],{"class":503}," werkzeug.wrappers ",[151,51771,16859],{"class":1869},[151,51773,51774],{"class":503}," Response\n",[151,51776,51777],{"class":469,"line":488},[151,51778,1090],{"emptyLinePlaceholder":609},[151,51780,51781,51783,51785,51787,51789,51791,51793],{"class":469,"line":500},[151,51782,16925],{"class":12347},[151,51784,39514],{"class":473},[151,51786,12386],{"class":503},[151,51788,39519],{"class":15232},[151,51790,106],{"class":503},[151,51792,39524],{"class":15232},[151,51794,15264],{"class":503},[151,51796,51797,51800,51803,51805],{"class":469,"line":509},[151,51798,51799],{"class":1869},"    with",[151,51801,51802],{"class":503}," Response.from_app(wsgi_app, wsgi_request(event)) ",[151,51804,16998],{"class":1869},[151,51806,51807],{"class":503}," response:\n",[151,51809,51810,51813,51815,51818],{"class":469,"line":517},[151,51811,51812],{"class":503},"        zappa_returndict ",[151,51814,1876],{"class":1869},[151,51816,51817],{"class":6205}," dict",[151,51819,12461],{"class":503},[151,51821,51822,51825,51828,51830,51832],{"class":469,"line":534},[151,51823,51824],{"class":503},"        zappa_returndict[",[151,51826,51827],{"class":481},"'body'",[151,51829,16654],{"class":503},[151,51831,1876],{"class":1869},[151,51833,51834],{"class":503}," response.data\n",[151,51836,51837,51839,51842,51844,51846],{"class":469,"line":1413},[151,51838,51824],{"class":503},[151,51840,51841],{"class":481},"'statusCode'",[151,51843,16654],{"class":503},[151,51845,1876],{"class":1869},[151,51847,51848],{"class":503}," response.status_code\n",[151,51850,51851,51853],{"class":469,"line":1418},[151,51852,16833],{"class":1869},[151,51854,51855],{"class":503}," zappa_returndict\n",[11,51857,51858,51859,51864],{},"For reference, ",[20,51860,51863],{"href":51861,"rel":51862},"https://github.com/Miserlou/Zappa/blob/master/zappa/handler.py#L540",[24],"here is the link to the line in Zappa's source code that starts processing API Gateway requests"," on which the above psuedo code is loosly based.",[11,51866,19225,51867,51869,51870,51874,51875,413,51878,51881,51882,51884,51885,51887],{},[30,51868,51752],{}," project makes use of a package called ",[20,51871,39581],{"href":51872,"rel":51873},"https://github.com/slank/awsgi",[24]," that has active contributions from people at AWS. Similar to djambda, it is a mashup of words (acronyms): (",[30,51876,51877],{},"AWS",[30,51879,51880],{},"wsgi"," = ",[30,51883,39581],{},"). It does most of the work that Zappa's handler does, so I replaced the adapted Zappa handler code with a very elagant handler (borrowed from ",[30,51886,51752],{},"):",[459,51889,51891],{"className":24401,"code":51890,"language":24403,"meta":464,"style":464},"# djambda/src/djambda/awsgi.py\nimport io\n\nimport awsgi\nfrom django.core import management\n\nfrom .wsgi import application\n\n\ndef lambda_handler(event, context):\n    if \"manage\" in event:\n        output = io.StringIO()\n        management.call_command(*event[\"manage\"].split(\" \"), stdout=output)\n        return {\"output\": output.getvalue()}\n    else:\n        return awsgi.response(application, event, context)\n",[30,51892,51893,51898,51905,51909,51915,51927,51931,51943,51947,51951,51968,51980,51990,52017,52030,52036],{"__ignoreMap":464},[151,51894,51895],{"class":469,"line":470},[151,51896,51897],{"class":1527},"# djambda/src/djambda/awsgi.py\n",[151,51899,51900,51902],{"class":469,"line":488},[151,51901,16859],{"class":1869},[151,51903,51904],{"class":503}," io\n",[151,51906,51907],{"class":469,"line":500},[151,51908,1090],{"emptyLinePlaceholder":609},[151,51910,51911,51913],{"class":469,"line":509},[151,51912,16859],{"class":1869},[151,51914,38651],{"class":503},[151,51916,51917,51919,51922,51924],{"class":469,"line":517},[151,51918,16853],{"class":1869},[151,51920,51921],{"class":503}," django.core ",[151,51923,16859],{"class":1869},[151,51925,51926],{"class":503}," management\n",[151,51928,51929],{"class":469,"line":534},[151,51930,1090],{"emptyLinePlaceholder":609},[151,51932,51933,51935,51938,51940],{"class":469,"line":1413},[151,51934,16853],{"class":1869},[151,51936,51937],{"class":503}," .wsgi ",[151,51939,16859],{"class":1869},[151,51941,51942],{"class":503}," application\n",[151,51944,51945],{"class":469,"line":1418},[151,51946,1090],{"emptyLinePlaceholder":609},[151,51948,51949],{"class":469,"line":2462},[151,51950,1090],{"emptyLinePlaceholder":609},[151,51952,51953,51955,51958,51960,51962,51964,51966],{"class":469,"line":2471},[151,51954,16925],{"class":12347},[151,51956,51957],{"class":473}," lambda_handler",[151,51959,12386],{"class":503},[151,51961,39519],{"class":15232},[151,51963,106],{"class":503},[151,51965,39524],{"class":15232},[151,51967,15264],{"class":503},[151,51969,51970,51972,51975,51977],{"class":469,"line":2480},[151,51971,23327],{"class":1869},[151,51973,51974],{"class":481}," \"manage\"",[151,51976,2820],{"class":1869},[151,51978,51979],{"class":503}," event:\n",[151,51981,51982,51985,51987],{"class":469,"line":2489},[151,51983,51984],{"class":503},"        output ",[151,51986,1876],{"class":1869},[151,51988,51989],{"class":503}," io.StringIO()\n",[151,51991,51992,51995,51997,52000,52003,52006,52008,52010,52012,52014],{"class":469,"line":2497},[151,51993,51994],{"class":503},"        management.call_command(",[151,51996,23268],{"class":1869},[151,51998,51999],{"class":503},"event[",[151,52001,52002],{"class":481},"\"manage\"",[151,52004,52005],{"class":503},"].split(",[151,52007,24311],{"class":481},[151,52009,24817],{"class":503},[151,52011,27640],{"class":15210},[151,52013,1876],{"class":1869},[151,52015,52016],{"class":503},"output)\n",[151,52018,52019,52021,52024,52027],{"class":469,"line":3140},[151,52020,16833],{"class":1869},[151,52022,52023],{"class":503}," {",[151,52025,52026],{"class":481},"\"output\"",[151,52028,52029],{"class":503},": output.getvalue()}\n",[151,52031,52032,52034],{"class":469,"line":3149},[151,52033,38878],{"class":1869},[151,52035,14372],{"class":503},[151,52037,52038,52040],{"class":469,"line":3158},[151,52039,16833],{"class":1869},[151,52041,52042],{"class":503}," awsgi.response(application, event, context)\n",[11,52044,52045,52046,106,52048,106,52050,52052],{},"This handler can takes care of translating API Gateway requests to WSGI requests as well as management commands such as ",[30,52047,27589],{},[30,52049,51591],{},[30,52051,27586],{}," and other custom management commands. Let's talk about the management commands first as they relate to the two main AWS services that our Django application uses: RDS and S3. Then we will talk about API Gateway, the proxy Lambda and how the Lambda proxy pattern works.",[56,52054,26856],{"id":26801},[11,52056,52057,52058,52061],{},"To run the management commands for our application, we can use the AWS CLI to invoke the Lambda function with a payload that will trigger the ",[30,52059,52060],{},"if \"manage\" in event:"," code block from the above handler:",[459,52063,52065],{"className":461,"code":52064,"language":463,"meta":464,"style":464},"aws lambda invoke \\\n    --function-name my-djambda-lambda \\\n    --invocation-type RequestResponse \\\n    --payload '{\"manage\": \"migrate --no-input\"}' \\\n    resp.json\n",[30,52066,52067,52079,52089,52099,52109],{"__ignoreMap":464},[151,52068,52069,52071,52074,52077],{"class":469,"line":470},[151,52070,27264],{"class":473},[151,52072,52073],{"class":481}," lambda",[151,52075,52076],{"class":481}," invoke",[151,52078,485],{"class":477},[151,52080,52081,52084,52087],{"class":469,"line":488},[151,52082,52083],{"class":477},"    --function-name",[151,52085,52086],{"class":481}," my-djambda-lambda",[151,52088,485],{"class":477},[151,52090,52091,52094,52097],{"class":469,"line":500},[151,52092,52093],{"class":477},"    --invocation-type",[151,52095,52096],{"class":481}," RequestResponse",[151,52098,485],{"class":477},[151,52100,52101,52104,52107],{"class":469,"line":509},[151,52102,52103],{"class":477},"    --payload",[151,52105,52106],{"class":481}," '{\"manage\": \"migrate --no-input\"}'",[151,52108,485],{"class":477},[151,52110,52111],{"class":469,"line":517},[151,52112,52113],{"class":481},"    resp.json\n",[11,52115,52116,52117,52119,52120,21420,52123,52126,52127,52130,52131,187,52133,52136,52137,52139],{},"This will run the ",[30,52118,27589],{}," command by connecting to RDS from our Django application with a special version of ",[30,52121,52122],{},"psycopg2",[30,52124,52125],{},"aws-psycopg2"," (check ",[30,52128,52129],{},"django/requirements.txt"," for this dependency). ",[30,52132,52122],{},[30,52134,52135],{},"psycopg2-binary"," both gave error messages when trying to access the database, but the ",[30,52138,52125],{}," package had no issues.",[11,52141,52142],{},"RDS can sometimes be the most expensive part of a project on AWS. To reduce costs, we are using Aurora Postgres Serverless. This AWS service is ideal for small, infrequently-used projects that are in development. The database \"goes to sleep\" after a period of inactivity (5 minutes, I think). When new requests to the database are made, it can take up to 30 seconds for the database to wake up. For this reason, we need the timeout of the Django Lambda to be able to clear the wakeup period of the Aurora Postgres Serverless database.",[11,52144,52145,52146,52149],{},"Since our Lambda function is in public subnet of our VPC and our RDS Aurora database cluster is in an isolated VPC, we need to make sure that the Lambda function can talk to the RDS cluster. CDK makes this really easy. I'll highlight a few important permission related parts of ",[30,52147,52148],{},"awscdk/vpc.py",", the code that defines the nested CloudFormation stack where we define our VPC and resources related to our application (including API Gateway, which is technically not located in our VPC).",[11,52151,52152],{},"First, we define a new security group for our Django Lambda:",[459,52154,52156],{"className":24401,"code":52155,"language":24403,"meta":464,"style":464},"        self.lambda_security_group = ec2.SecurityGroup(\n            self, \"LambdaSecurityGroup\", vpc=self.vpc\n        )\n",[30,52157,52158,52170,52190],{"__ignoreMap":464},[151,52159,52160,52162,52165,52167],{"class":469,"line":470},[151,52161,37901],{"class":15289},[151,52163,52164],{"class":503},".lambda_security_group ",[151,52166,1876],{"class":1869},[151,52168,52169],{"class":503}," ec2.SecurityGroup(\n",[151,52171,52172,52174,52176,52179,52181,52183,52185,52187],{"class":469,"line":488},[151,52173,15290],{"class":15289},[151,52175,106],{"class":503},[151,52177,52178],{"class":481},"\"LambdaSecurityGroup\"",[151,52180,106],{"class":503},[151,52182,26897],{"class":15210},[151,52184,1876],{"class":1869},[151,52186,15277],{"class":15289},[151,52188,52189],{"class":503},".vpc\n",[151,52191,52192],{"class":469,"line":500},[151,52193,16824],{"class":503},[11,52195,52196],{},"Then, we reference this security group in the list of security group ingresses for our database cluster's security group:",[459,52198,52200],{"className":24401,"code":52199,"language":24403,"meta":464,"style":464},"        self.db_security_group = ec2.CfnSecurityGroup(\n            self,\n            \"DBSecurityGroup\",\n            vpc_id=self.vpc.vpc_id,\n            group_description=\"DBSecurityGroup\",\n            security_group_ingress=[\n                ec2.CfnSecurityGroup.IngressProperty(\n                    ip_protocol=\"tcp\",\n                    to_port=5432,\n                    from_port=5432,\n                    source_security_group_id=self.lambda_security_group.security_group_id,\n                )\n            ],\n        )\n",[30,52201,52202,52214,52220,52227,52239,52251,52260,52265,52277,52288,52299,52311,52315,52320],{"__ignoreMap":464},[151,52203,52204,52206,52209,52211],{"class":469,"line":470},[151,52205,37901],{"class":15289},[151,52207,52208],{"class":503},".db_security_group ",[151,52210,1876],{"class":1869},[151,52212,52213],{"class":503}," ec2.CfnSecurityGroup(\n",[151,52215,52216,52218],{"class":469,"line":488},[151,52217,15290],{"class":15289},[151,52219,9417],{"class":503},[151,52221,52222,52225],{"class":469,"line":500},[151,52223,52224],{"class":481},"            \"DBSecurityGroup\"",[151,52226,9417],{"class":503},[151,52228,52229,52232,52234,52236],{"class":469,"line":509},[151,52230,52231],{"class":15210},"            vpc_id",[151,52233,1876],{"class":1869},[151,52235,15277],{"class":15289},[151,52237,52238],{"class":503},".vpc.vpc_id,\n",[151,52240,52241,52244,52246,52249],{"class":469,"line":517},[151,52242,52243],{"class":15210},"            group_description",[151,52245,1876],{"class":1869},[151,52247,52248],{"class":481},"\"DBSecurityGroup\"",[151,52250,9417],{"class":503},[151,52252,52253,52256,52258],{"class":469,"line":534},[151,52254,52255],{"class":15210},"            security_group_ingress",[151,52257,1876],{"class":1869},[151,52259,37620],{"class":503},[151,52261,52262],{"class":469,"line":1413},[151,52263,52264],{"class":503},"                ec2.CfnSecurityGroup.IngressProperty(\n",[151,52266,52267,52270,52272,52275],{"class":469,"line":1418},[151,52268,52269],{"class":15210},"                    ip_protocol",[151,52271,1876],{"class":1869},[151,52273,52274],{"class":481},"\"tcp\"",[151,52276,9417],{"class":503},[151,52278,52279,52282,52284,52286],{"class":469,"line":2462},[151,52280,52281],{"class":15210},"                    to_port",[151,52283,1876],{"class":1869},[151,52285,50676],{"class":477},[151,52287,9417],{"class":503},[151,52289,52290,52293,52295,52297],{"class":469,"line":2471},[151,52291,52292],{"class":15210},"                    from_port",[151,52294,1876],{"class":1869},[151,52296,50676],{"class":477},[151,52298,9417],{"class":503},[151,52300,52301,52304,52306,52308],{"class":469,"line":2480},[151,52302,52303],{"class":15210},"                    source_security_group_id",[151,52305,1876],{"class":1869},[151,52307,15277],{"class":15289},[151,52309,52310],{"class":503},".lambda_security_group.security_group_id,\n",[151,52312,52313],{"class":469,"line":2489},[151,52314,16814],{"class":503},[151,52316,52317],{"class":469,"line":2497},[151,52318,52319],{"class":503},"            ],\n",[151,52321,52322],{"class":469,"line":3140},[151,52323,16824],{"class":503},[11,52325,52326],{},"Now let's take a look at the code for the Django lambda itself:",[459,52328,52330],{"className":24401,"code":52329,"language":24403,"meta":464,"style":464},"        self.djambda_lambda = _lambda.Function(\n            self,\n            \"DjambdaLambda\",\n            runtime=_lambda.Runtime.PYTHON_3_8,\n            code=_lambda.AssetCode('./django'),\n            function_name=f\"{scope.full_app_name}-djambda-lambda\",\n            handler=\"djambda.awsgi.lambda_handler\",\n            layers=[self.djambda_layer],\n            timeout=core.Duration.seconds(60),\n            vpc=self.vpc,\n            vpc_subnets=ec2.SubnetSelection(subnets=self.vpc.isolated_subnets),\n            environment={**self.env_vars},\n            security_groups=[self.lambda_security_group],\n        )\n\n        # Use raw override because Lambda's can't be placed in\n        # public subnets using CDK: https://github.com/aws/aws-cdk/issues/8935\n        self.djambda_lambda.node.default_child.add_override(\n            \"Properties.VpcConfig.SubnetIds\",\n            [subnet.subnet_id for subnet in self.vpc.public_subnets],\n        )\n",[30,52331,52332,52343,52349,52356,52368,52381,52403,52414,52427,52439,52451,52471,52485,52499,52503,52507,52512,52517,52524,52531,52548],{"__ignoreMap":464},[151,52333,52334,52336,52339,52341],{"class":469,"line":470},[151,52335,37901],{"class":15289},[151,52337,52338],{"class":503},".djambda_lambda ",[151,52340,1876],{"class":1869},[151,52342,39734],{"class":503},[151,52344,52345,52347],{"class":469,"line":488},[151,52346,15290],{"class":15289},[151,52348,9417],{"class":503},[151,52350,52351,52354],{"class":469,"line":500},[151,52352,52353],{"class":481},"            \"DjambdaLambda\"",[151,52355,9417],{"class":503},[151,52357,52358,52360,52362,52364,52366],{"class":469,"line":509},[151,52359,39752],{"class":15210},[151,52361,1876],{"class":1869},[151,52363,39757],{"class":503},[151,52365,39711],{"class":477},[151,52367,9417],{"class":503},[151,52369,52370,52372,52374,52376,52379],{"class":469,"line":517},[151,52371,39688],{"class":15210},[151,52373,1876],{"class":1869},[151,52375,39693],{"class":503},[151,52377,52378],{"class":481},"'./django'",[151,52380,37985],{"class":503},[151,52382,52383,52385,52387,52389,52391,52393,52396,52398,52401],{"class":469,"line":534},[151,52384,39779],{"class":15210},[151,52386,1876],{"class":1869},[151,52388,13214],{"class":12347},[151,52390,8592],{"class":481},[151,52392,5729],{"class":477},[151,52394,52395],{"class":503},"scope.full_app_name",[151,52397,2001],{"class":477},[151,52399,52400],{"class":481},"-djambda-lambda\"",[151,52402,9417],{"class":503},[151,52404,52405,52407,52409,52412],{"class":469,"line":1413},[151,52406,39791],{"class":15210},[151,52408,1876],{"class":1869},[151,52410,52411],{"class":481},"\"djambda.awsgi.lambda_handler\"",[151,52413,9417],{"class":503},[151,52415,52416,52418,52420,52422,52424],{"class":469,"line":1418},[151,52417,39803],{"class":15210},[151,52419,1876],{"class":1869},[151,52421,6698],{"class":503},[151,52423,15277],{"class":15289},[151,52425,52426],{"class":503},".djambda_layer],\n",[151,52428,52429,52431,52433,52435,52437],{"class":469,"line":2462},[151,52430,39817],{"class":15210},[151,52432,1876],{"class":1869},[151,52434,39822],{"class":503},[151,52436,39825],{"class":477},[151,52438,37985],{"class":503},[151,52440,52441,52444,52446,52448],{"class":469,"line":2471},[151,52442,52443],{"class":15210},"            vpc",[151,52445,1876],{"class":1869},[151,52447,15277],{"class":15289},[151,52449,52450],{"class":503},".vpc,\n",[151,52452,52453,52456,52458,52461,52464,52466,52468],{"class":469,"line":2480},[151,52454,52455],{"class":15210},"            vpc_subnets",[151,52457,1876],{"class":1869},[151,52459,52460],{"class":503},"ec2.SubnetSelection(",[151,52462,52463],{"class":15210},"subnets",[151,52465,1876],{"class":1869},[151,52467,15277],{"class":15289},[151,52469,52470],{"class":503},".vpc.isolated_subnets),\n",[151,52472,52473,52475,52477,52479,52481,52483],{"class":469,"line":2489},[151,52474,38021],{"class":15210},[151,52476,1876],{"class":1869},[151,52478,5729],{"class":503},[151,52480,24677],{"class":1869},[151,52482,15277],{"class":15289},[151,52484,39842],{"class":503},[151,52486,52487,52490,52492,52494,52496],{"class":469,"line":2497},[151,52488,52489],{"class":15210},"            security_groups",[151,52491,1876],{"class":1869},[151,52493,6698],{"class":503},[151,52495,15277],{"class":15289},[151,52497,52498],{"class":503},".lambda_security_group],\n",[151,52500,52501],{"class":469,"line":3140},[151,52502,16824],{"class":503},[151,52504,52505],{"class":469,"line":3149},[151,52506,1090],{"emptyLinePlaceholder":609},[151,52508,52509],{"class":469,"line":3158},[151,52510,52511],{"class":1527},"        # Use raw override because Lambda's can't be placed in\n",[151,52513,52514],{"class":469,"line":3167},[151,52515,52516],{"class":1527},"        # public subnets using CDK: https://github.com/aws/aws-cdk/issues/8935\n",[151,52518,52519,52521],{"class":469,"line":3175},[151,52520,37901],{"class":15289},[151,52522,52523],{"class":503},".djambda_lambda.node.default_child.add_override(\n",[151,52525,52526,52529],{"class":469,"line":3184},[151,52527,52528],{"class":481},"            \"Properties.VpcConfig.SubnetIds\"",[151,52530,9417],{"class":503},[151,52532,52533,52536,52538,52541,52543,52545],{"class":469,"line":3193},[151,52534,52535],{"class":503},"            [subnet.subnet_id ",[151,52537,16732],{"class":1869},[151,52539,52540],{"class":503}," subnet ",[151,52542,16417],{"class":1869},[151,52544,15451],{"class":15289},[151,52546,52547],{"class":503},".vpc.public_subnets],\n",[151,52549,52550],{"class":469,"line":3720},[151,52551,16824],{"class":503},[11,52553,52554,52556,52557,25983,52559,52562,52563,52566,52567,52570,52571,52573,52574,52577,52578,52581,52582,19977,52585,10480,52588,52591,52592,52595],{},[30,52555,43773],{}," is a reserved word in Python, so we import ",[30,52558,43773],{},[30,52560,52561],{},"_lambda"," from ",[30,52564,52565],{},"aws_cdk",". Note that we have a reference to the ",[30,52568,52569],{},"awsgi.py"," handler function in the ",[30,52572,51756],{}," parameter of ",[30,52575,52576],{},"self.djambda_lambda",". I initially put the lambda in ",[30,52579,52580],{},"isolated_subnets"," because CDK won't let you define a ",[30,52583,52584],{},"_lambda.Function",[30,52586,52587],{},"public_subnets",[30,52589,52590],{},"vpc_subnets",". We can override this using ",[30,52593,52594],{},"add_override"," below the Lambda definition to place it in a public subnet instead.",[210,52597,52598],{},[11,52599,52600,52601,52604],{},"I'm not sure if this is necessary, or if this is recommended. Things get a little bit confusing for me here so I would love some insight if anyone knows what would be best to do here. The VPC construct for CDK doesn't allow you to have private subnets when ",[30,52602,52603],{},"nat_gateways=0",". Would I be better off placing the lambda in the isolated subnet with the RDS cluster? Would I still be able to access S3 via the VPC Gateway Endpoint from an isolated subnet?",[11,52606,52607,52608,52610,52611,10552],{},"Putting aside my uncertainties about the correct way to configure our Lambdas functions in a VPC, let's continue! Here's the Lambda invocation for ",[30,52609,51591],{}," that we can run once we have successfully invoked the ",[30,52612,27589],{},[459,52614,52616],{"className":461,"code":52615,"language":463,"meta":464,"style":464},"aws lambda invoke \\\n    --function-name dev-mysite-com-djambda-lambda \\\n    --invocation-type RequestResponse \\\n    --payload '{\"manage\": \"createsuperuser --no-input --username admin --email brian@email.com\"}' \\\n    resp.json\n",[30,52617,52618,52628,52637,52645,52654],{"__ignoreMap":464},[151,52619,52620,52622,52624,52626],{"class":469,"line":470},[151,52621,27264],{"class":473},[151,52623,52073],{"class":481},[151,52625,52076],{"class":481},[151,52627,485],{"class":477},[151,52629,52630,52632,52635],{"class":469,"line":488},[151,52631,52083],{"class":477},[151,52633,52634],{"class":481}," dev-mysite-com-djambda-lambda",[151,52636,485],{"class":477},[151,52638,52639,52641,52643],{"class":469,"line":500},[151,52640,52093],{"class":477},[151,52642,52096],{"class":481},[151,52644,485],{"class":477},[151,52646,52647,52649,52652],{"class":469,"line":509},[151,52648,52103],{"class":477},[151,52650,52651],{"class":481}," '{\"manage\": \"createsuperuser --no-input --username admin --email brian@email.com\"}'",[151,52653,485],{"class":477},[151,52655,52656],{"class":469,"line":517},[151,52657,52113],{"class":481},[11,52659,52660,52661,52663,52664,10480,52667,52670,52671,208],{},"This assumes that there is an environment variable defined in our Lambda's ",[30,52662,28577],{}," variables that we passed in ",[30,52665,52666],{},"{**self.env_vars}",[30,52668,52669],{},"DJANGO_SUPERUSER_USERNAME",". Here's what we have in ",[30,52672,52673],{},"env_vars",[459,52675,52677],{"className":24401,"code":52676,"language":24403,"meta":464,"style":464},"        self.env_vars = {\n            \"POSTGRES_SERVICE_HOST\": self.rds_cluster.get_att(\n                \"Endpoint.Address\"\n            ).to_string(),\n            \"POSTGRES_PASSWORD\": os.environ.get(\"DB_PASSWORD\", \"db-password\"),\n            \"AWS_STORAGE_BUCKET_NAME\": f\"{scope.full_app_name}-assets\",\n            \"DEBUG\": \"\",\n            \"DJANGO_SUPERUSER_PASSWORD\": os.environ.get(\n                \"DJANGO_SUPERUSER_PASSWORD\", \"Mypassword1!\"\n            ),\n            \"DJANGO_SUPERUSER_USERNAME\": os.environ.get(\n                \"DJANGO_SUPERUSER_USERNAME\", \"admin\"\n            ),\n        }\n",[30,52678,52679,52689,52701,52706,52711,52728,52750,52761,52769,52779,52784,52791,52801,52805],{"__ignoreMap":464},[151,52680,52681,52683,52685,52687],{"class":469,"line":470},[151,52682,37901],{"class":15289},[151,52684,39610],{"class":503},[151,52686,1876],{"class":1869},[151,52688,19833],{"class":503},[151,52690,52691,52694,52696,52698],{"class":469,"line":488},[151,52692,52693],{"class":481},"            \"POSTGRES_SERVICE_HOST\"",[151,52695,6208],{"class":503},[151,52697,15277],{"class":15289},[151,52699,52700],{"class":503},".rds_cluster.get_att(\n",[151,52702,52703],{"class":469,"line":500},[151,52704,52705],{"class":481},"                \"Endpoint.Address\"\n",[151,52707,52708],{"class":469,"line":509},[151,52709,52710],{"class":503},"            ).to_string(),\n",[151,52712,52713,52716,52718,52721,52723,52726],{"class":469,"line":517},[151,52714,52715],{"class":481},"            \"POSTGRES_PASSWORD\"",[151,52717,38033],{"class":503},[151,52719,52720],{"class":481},"\"DB_PASSWORD\"",[151,52722,106],{"class":503},[151,52724,52725],{"class":481},"\"db-password\"",[151,52727,37985],{"class":503},[151,52729,52730,52733,52735,52737,52739,52741,52743,52745,52748],{"class":469,"line":534},[151,52731,52732],{"class":481},"            \"AWS_STORAGE_BUCKET_NAME\"",[151,52734,6208],{"class":503},[151,52736,13214],{"class":12347},[151,52738,8592],{"class":481},[151,52740,5729],{"class":477},[151,52742,52395],{"class":503},[151,52744,2001],{"class":477},[151,52746,52747],{"class":481},"-assets\"",[151,52749,9417],{"class":503},[151,52751,52752,52755,52757,52759],{"class":469,"line":1413},[151,52753,52754],{"class":481},"            \"DEBUG\"",[151,52756,6208],{"class":503},[151,52758,38471],{"class":481},[151,52760,9417],{"class":503},[151,52762,52763,52766],{"class":469,"line":1418},[151,52764,52765],{"class":481},"            \"DJANGO_SUPERUSER_PASSWORD\"",[151,52767,52768],{"class":503},": os.environ.get(\n",[151,52770,52771,52774,52776],{"class":469,"line":2462},[151,52772,52773],{"class":481},"                \"DJANGO_SUPERUSER_PASSWORD\"",[151,52775,106],{"class":503},[151,52777,52778],{"class":481},"\"Mypassword1!\"\n",[151,52780,52781],{"class":469,"line":2471},[151,52782,52783],{"class":503},"            ),\n",[151,52785,52786,52789],{"class":469,"line":2480},[151,52787,52788],{"class":481},"            \"DJANGO_SUPERUSER_USERNAME\"",[151,52790,52768],{"class":503},[151,52792,52793,52796,52798],{"class":469,"line":2489},[151,52794,52795],{"class":481},"                \"DJANGO_SUPERUSER_USERNAME\"",[151,52797,106],{"class":503},[151,52799,52800],{"class":481},"\"admin\"\n",[151,52802,52803],{"class":469,"line":2497},[151,52804,52783],{"class":503},[151,52806,52807],{"class":469,"line":3140},[151,52808,23390],{"class":503},[11,52810,52811],{},"We can use environment variables to set an initial password for our superuser. We also define additionl environment variables that we will use for our datbase connection, and the S3 bucket that we will use for Django's static and media assets.",[56,52813,26847],{"id":52814},"s3",[11,52816,52817],{},"To access S3 from our Lambda function in a VPC, we need to use a VPC Gateway Endpoint for S3. This is a free service that enables us to access S3 directly from our VPC without going through the internet, and instead using a private connection . This again has to do with the fact that our Django Lambda can't access the internet because the network interfaces created by Lambda only have private IP addresses and would require NAT in order to connect to resources on the internet. Here's how we define the VPC as well as the VPC Gateway Endpoint using CDK:",[459,52819,52821],{"className":24401,"code":52820,"language":24403,"meta":464,"style":464},"        self.vpc = ec2.Vpc(\n            self,\n            \"Vpc\",\n            max_azs=2,\n            cidr=\"10.0.0.0/16\",\n            nat_gateways=0,\n            subnet_configuration=[\n                ec2.SubnetConfiguration(\n                    subnet_type=ec2.SubnetType.PUBLIC,\n                    name=\"Public\",\n                    cidr_mask=24,\n                ),\n                ec2.SubnetConfiguration(\n                    subnet_type=ec2.SubnetType.ISOLATED,\n                    name=\"Isolated\",\n                    cidr_mask=24,\n                ),\n            ],\n        )\n\n        self.vpc.add_gateway_endpoint(\n            \"S3Gateway\", service=ec2.GatewayVpcEndpointAwsService('s3')\n        )\n",[30,52822,52823,52835,52841,52848,52859,52871,52882,52891,52896,52911,52923,52934,52939,52943,52956,52967,52977,52981,52985,52989,52993,53000,53020],{"__ignoreMap":464},[151,52824,52825,52827,52830,52832],{"class":469,"line":470},[151,52826,37901],{"class":15289},[151,52828,52829],{"class":503},".vpc ",[151,52831,1876],{"class":1869},[151,52833,52834],{"class":503}," ec2.Vpc(\n",[151,52836,52837,52839],{"class":469,"line":488},[151,52838,15290],{"class":15289},[151,52840,9417],{"class":503},[151,52842,52843,52846],{"class":469,"line":500},[151,52844,52845],{"class":481},"            \"Vpc\"",[151,52847,9417],{"class":503},[151,52849,52850,52853,52855,52857],{"class":469,"line":509},[151,52851,52852],{"class":15210},"            max_azs",[151,52854,1876],{"class":1869},[151,52856,6619],{"class":477},[151,52858,9417],{"class":503},[151,52860,52861,52864,52866,52869],{"class":469,"line":517},[151,52862,52863],{"class":15210},"            cidr",[151,52865,1876],{"class":1869},[151,52867,52868],{"class":481},"\"10.0.0.0/16\"",[151,52870,9417],{"class":503},[151,52872,52873,52876,52878,52880],{"class":469,"line":534},[151,52874,52875],{"class":15210},"            nat_gateways",[151,52877,1876],{"class":1869},[151,52879,9181],{"class":477},[151,52881,9417],{"class":503},[151,52883,52884,52887,52889],{"class":469,"line":1413},[151,52885,52886],{"class":15210},"            subnet_configuration",[151,52888,1876],{"class":1869},[151,52890,37620],{"class":503},[151,52892,52893],{"class":469,"line":1418},[151,52894,52895],{"class":503},"                ec2.SubnetConfiguration(\n",[151,52897,52898,52901,52903,52906,52909],{"class":469,"line":2462},[151,52899,52900],{"class":15210},"                    subnet_type",[151,52902,1876],{"class":1869},[151,52904,52905],{"class":503},"ec2.SubnetType.",[151,52907,52908],{"class":477},"PUBLIC",[151,52910,9417],{"class":503},[151,52912,52913,52916,52918,52921],{"class":469,"line":2471},[151,52914,52915],{"class":15210},"                    name",[151,52917,1876],{"class":1869},[151,52919,52920],{"class":481},"\"Public\"",[151,52922,9417],{"class":503},[151,52924,52925,52928,52930,52932],{"class":469,"line":2480},[151,52926,52927],{"class":15210},"                    cidr_mask",[151,52929,1876],{"class":1869},[151,52931,7728],{"class":477},[151,52933,9417],{"class":503},[151,52935,52936],{"class":469,"line":2489},[151,52937,52938],{"class":503},"                ),\n",[151,52940,52941],{"class":469,"line":2497},[151,52942,52895],{"class":503},[151,52944,52945,52947,52949,52951,52954],{"class":469,"line":3140},[151,52946,52900],{"class":15210},[151,52948,1876],{"class":1869},[151,52950,52905],{"class":503},[151,52952,52953],{"class":477},"ISOLATED",[151,52955,9417],{"class":503},[151,52957,52958,52960,52962,52965],{"class":469,"line":3149},[151,52959,52915],{"class":15210},[151,52961,1876],{"class":1869},[151,52963,52964],{"class":481},"\"Isolated\"",[151,52966,9417],{"class":503},[151,52968,52969,52971,52973,52975],{"class":469,"line":3158},[151,52970,52927],{"class":15210},[151,52972,1876],{"class":1869},[151,52974,7728],{"class":477},[151,52976,9417],{"class":503},[151,52978,52979],{"class":469,"line":3167},[151,52980,52938],{"class":503},[151,52982,52983],{"class":469,"line":3175},[151,52984,52319],{"class":503},[151,52986,52987],{"class":469,"line":3184},[151,52988,16824],{"class":503},[151,52990,52991],{"class":469,"line":3193},[151,52992,1090],{"emptyLinePlaceholder":609},[151,52994,52995,52997],{"class":469,"line":3720},[151,52996,37901],{"class":15289},[151,52998,52999],{"class":503},".vpc.add_gateway_endpoint(\n",[151,53001,53002,53005,53007,53010,53012,53015,53018],{"class":469,"line":3729},[151,53003,53004],{"class":481},"            \"S3Gateway\"",[151,53006,106],{"class":503},[151,53008,53009],{"class":15210},"service",[151,53011,1876],{"class":1869},[151,53013,53014],{"class":503},"ec2.GatewayVpcEndpointAwsService(",[151,53016,53017],{"class":481},"'s3'",[151,53019,3640],{"class":503},[151,53021,53022],{"class":469,"line":3735},[151,53023,16824],{"class":503},[11,53025,53026,53027,53030],{},"With this VPC endpoint in place, we are almost ready to run our collectstatic command, but it won't work yet. This is because we haven't given our Lambda function access to write files to the S3 assets bucket for our Django application's static and media files. We can grant this permission with the following snippet from ",[30,53028,53029],{},"awscdk/app_stack.py",", the file that defines the root CloudFormation stack for our application:",[459,53032,53034],{"className":24401,"code":53033,"language":24403,"meta":464,"style":464},"        self.backend_assets_bucket.grant_read_write(\n            self.vpc_stack.djambda_lambda\n        )\n",[30,53035,53036,53043,53050],{"__ignoreMap":464},[151,53037,53038,53040],{"class":469,"line":470},[151,53039,37901],{"class":15289},[151,53041,53042],{"class":503},".backend_assets_bucket.grant_read_write(\n",[151,53044,53045,53047],{"class":469,"line":488},[151,53046,15290],{"class":15289},[151,53048,53049],{"class":503},".vpc_stack.djambda_lambda\n",[151,53051,53052],{"class":469,"line":500},[151,53053,16824],{"class":503},[11,53055,53056,53057,208],{},"Finally, we need to add the following settings to ",[30,53058,50465],{},[459,53060,53062],{"className":24401,"code":53061,"language":24403,"meta":464,"style":464},"STATIC_URL = '/static/'\nSTATIC_ROOT = os.path.join(BASE_DIR, 'static')\nSTATICFILES_STORAGE = \"djambda.storage_backends.StaticStorage\"\n\nAWS_DEFAULT_ACL = None\nAWS_STORAGE_BUCKET_NAME = os.environ.get(\n    \"AWS_STORAGE_BUCKET_NAME\", \"bucketname\"\n)\nAWS_S3_OBJECT_PARAMETERS = {\n    \"CacheControl\": \"max-age=86400\",\n}\nAWS_PRIVATE_MEDIA_LOCATION = \"media/private\"\nAWS_STATIC_LOCATION = \"static\"\nPRIVATE_FILE_STORAGE = \"backend.storage_backends.PrivateMediaStorage\"\nAWS_S3_CUSTOM_DOMAIN = f\"{AWS_STORAGE_BUCKET_NAME}.s3.amazonaws.com\"\n\nif not DEBUG:\n    MEDIA_ROOT = \"media\"\n    MEDIA_URL = f\"https://{AWS_S3_CUSTOM_DOMAIN}/{MEDIA_ROOT}/\"\n    STATIC_URL = f\"https://{AWS_S3_CUSTOM_DOMAIN}/{AWS_STATIC_LOCATION}/\"\n    FORCE_SCRIPT_NAME = \"/prod\"\n",[30,53063,53064,53074,53092,53102,53106,53116,53126,53136,53140,53149,53161,53165,53175,53185,53195,53212,53216,53227,53237,53260,53280],{"__ignoreMap":464},[151,53065,53066,53069,53071],{"class":469,"line":470},[151,53067,53068],{"class":477},"STATIC_URL",[151,53070,19865],{"class":1869},[151,53072,53073],{"class":481}," '/static/'\n",[151,53075,53076,53078,53080,53083,53085,53087,53090],{"class":469,"line":488},[151,53077,50461],{"class":477},[151,53079,19865],{"class":1869},[151,53081,53082],{"class":503}," os.path.join(",[151,53084,50479],{"class":477},[151,53086,106],{"class":503},[151,53088,53089],{"class":481},"'static'",[151,53091,3640],{"class":503},[151,53093,53094,53097,53099],{"class":469,"line":500},[151,53095,53096],{"class":477},"STATICFILES_STORAGE",[151,53098,19865],{"class":1869},[151,53100,53101],{"class":481}," \"djambda.storage_backends.StaticStorage\"\n",[151,53103,53104],{"class":469,"line":509},[151,53105,1090],{"emptyLinePlaceholder":609},[151,53107,53108,53111,53113],{"class":469,"line":517},[151,53109,53110],{"class":477},"AWS_DEFAULT_ACL",[151,53112,19865],{"class":1869},[151,53114,53115],{"class":477}," None\n",[151,53117,53118,53121,53123],{"class":469,"line":534},[151,53119,53120],{"class":477},"AWS_STORAGE_BUCKET_NAME",[151,53122,19865],{"class":1869},[151,53124,53125],{"class":503}," os.environ.get(\n",[151,53127,53128,53131,53133],{"class":469,"line":1413},[151,53129,53130],{"class":481},"    \"AWS_STORAGE_BUCKET_NAME\"",[151,53132,106],{"class":503},[151,53134,53135],{"class":481},"\"bucketname\"\n",[151,53137,53138],{"class":469,"line":1418},[151,53139,3640],{"class":503},[151,53141,53142,53145,53147],{"class":469,"line":2462},[151,53143,53144],{"class":477},"AWS_S3_OBJECT_PARAMETERS",[151,53146,19865],{"class":1869},[151,53148,19833],{"class":503},[151,53150,53151,53154,53156,53159],{"class":469,"line":2471},[151,53152,53153],{"class":481},"    \"CacheControl\"",[151,53155,6208],{"class":503},[151,53157,53158],{"class":481},"\"max-age=86400\"",[151,53160,9417],{"class":503},[151,53162,53163],{"class":469,"line":2480},[151,53164,6274],{"class":503},[151,53166,53167,53170,53172],{"class":469,"line":2489},[151,53168,53169],{"class":477},"AWS_PRIVATE_MEDIA_LOCATION",[151,53171,19865],{"class":1869},[151,53173,53174],{"class":481}," \"media/private\"\n",[151,53176,53177,53180,53182],{"class":469,"line":2497},[151,53178,53179],{"class":477},"AWS_STATIC_LOCATION",[151,53181,19865],{"class":1869},[151,53183,53184],{"class":481}," \"static\"\n",[151,53186,53187,53190,53192],{"class":469,"line":3140},[151,53188,53189],{"class":477},"PRIVATE_FILE_STORAGE",[151,53191,19865],{"class":1869},[151,53193,53194],{"class":481}," \"backend.storage_backends.PrivateMediaStorage\"\n",[151,53196,53197,53200,53202,53204,53206,53209],{"class":469,"line":3149},[151,53198,53199],{"class":477},"AWS_S3_CUSTOM_DOMAIN",[151,53201,19865],{"class":1869},[151,53203,36853],{"class":12347},[151,53205,8592],{"class":481},[151,53207,53208],{"class":477},"{AWS_STORAGE_BUCKET_NAME}",[151,53210,53211],{"class":481},".s3.amazonaws.com\"\n",[151,53213,53214],{"class":469,"line":3158},[151,53215,1090],{"emptyLinePlaceholder":609},[151,53217,53218,53220,53222,53225],{"class":469,"line":3167},[151,53219,17218],{"class":1869},[151,53221,4191],{"class":1869},[151,53223,53224],{"class":477}," DEBUG",[151,53226,14372],{"class":503},[151,53228,53229,53232,53234],{"class":469,"line":3175},[151,53230,53231],{"class":477},"    MEDIA_ROOT",[151,53233,19865],{"class":1869},[151,53235,53236],{"class":481}," \"media\"\n",[151,53238,53239,53242,53244,53246,53249,53252,53254,53257],{"class":469,"line":3184},[151,53240,53241],{"class":477},"    MEDIA_URL",[151,53243,19865],{"class":1869},[151,53245,36853],{"class":12347},[151,53247,53248],{"class":481},"\"https://",[151,53250,53251],{"class":477},"{AWS_S3_CUSTOM_DOMAIN}",[151,53253,19883],{"class":481},[151,53255,53256],{"class":477},"{MEDIA_ROOT}",[151,53258,53259],{"class":481},"/\"\n",[151,53261,53262,53265,53267,53269,53271,53273,53275,53278],{"class":469,"line":3193},[151,53263,53264],{"class":477},"    STATIC_URL",[151,53266,19865],{"class":1869},[151,53268,36853],{"class":12347},[151,53270,53248],{"class":481},[151,53272,53251],{"class":477},[151,53274,19883],{"class":481},[151,53276,53277],{"class":477},"{AWS_STATIC_LOCATION}",[151,53279,53259],{"class":481},[151,53281,53282,53285,53287],{"class":469,"line":3720},[151,53283,53284],{"class":477},"    FORCE_SCRIPT_NAME",[151,53286,19865],{"class":1869},[151,53288,53289],{"class":481}," \"/prod\"\n",[11,53291,53292],{},"We can run the collectstatic command with the following Lambda invocation:",[459,53294,53296],{"className":461,"code":53295,"language":463,"meta":464,"style":464},"aws lambda invoke \\\n    --function-name dev-mysite-com-djambda-lambda \\\n    --invocation-type RequestResponse \\\n    --payload '{\"manage\": \"collectstatic --no-input\"}' \\\n    resp.json\n",[30,53297,53298,53308,53316,53324,53333],{"__ignoreMap":464},[151,53299,53300,53302,53304,53306],{"class":469,"line":470},[151,53301,27264],{"class":473},[151,53303,52073],{"class":481},[151,53305,52076],{"class":481},[151,53307,485],{"class":477},[151,53309,53310,53312,53314],{"class":469,"line":488},[151,53311,52083],{"class":477},[151,53313,52634],{"class":481},[151,53315,485],{"class":477},[151,53317,53318,53320,53322],{"class":469,"line":500},[151,53319,52093],{"class":477},[151,53321,52096],{"class":481},[151,53323,485],{"class":477},[151,53325,53326,53328,53331],{"class":469,"line":509},[151,53327,52103],{"class":477},[151,53329,53330],{"class":481}," '{\"manage\": \"collectstatic --no-input\"}'",[151,53332,485],{"class":477},[151,53334,53335],{"class":469,"line":517},[151,53336,52113],{"class":481},[56,53338,53340],{"id":53339},"api-gateway-and-the-lambda-proxy-pattern","API Gateway and the Lambda proxy pattern",[11,53342,53343,53344,53349,53350,53354,53355,53360],{},"I first read about the Lambda proxy pattern in an article titled ",[20,53345,53348],{"href":53346,"rel":53347},"https://serverlessfirst.com/lambda-vpc-internet-access-no-nat-gateway/",[24],"How to access VPC and internet resources from Lambda without paying for a NAT Gateway"," on Paul Swail's website ",[20,53351,53352],{"href":53352,"rel":53353},"https://serverlessfirst.com/",[24],". This site has tons of great serverless content, check out the ",[20,53356,53359],{"href":53357,"rel":53358},"https://serverlessfirst.com/articles/",[24],"list of articles",". Thank you Paul for putting out great serverless resources!",[11,53362,53363],{},"Let's take a look at the Proxy Lambda function next. The function itself is pretty straightforward:",[459,53365,53367],{"className":24401,"code":53366,"language":24403,"meta":464,"style":464},"import json\nimport os\nimport boto3\n\nlambda_client = boto3.client('lambda', region_name='us-east-1')\n\n\ndef handler(event, context):\n    invoke_response = lambda_client.invoke(\n        FunctionName=os.environ.get(\"FUNCTION_NAME\", None),\n        InvocationType='RequestResponse',\n        Payload=json.dumps(event),\n    )\n\n    data = invoke_response['Payload'].read()\n\n    return data\n",[30,53368,53369,53375,53381,53387,53391,53415,53419,53423,53439,53449,53467,53479,53489,53493,53497,53513,53517],{"__ignoreMap":464},[151,53370,53371,53373],{"class":469,"line":470},[151,53372,16859],{"class":1869},[151,53374,24063],{"class":503},[151,53376,53377,53379],{"class":469,"line":488},[151,53378,16859],{"class":1869},[151,53380,24070],{"class":503},[151,53382,53383,53385],{"class":469,"line":500},[151,53384,16859],{"class":1869},[151,53386,38658],{"class":503},[151,53388,53389],{"class":469,"line":509},[151,53390,1090],{"emptyLinePlaceholder":609},[151,53392,53393,53396,53398,53400,53403,53405,53408,53410,53413],{"class":469,"line":517},[151,53394,53395],{"class":503},"lambda_client ",[151,53397,1876],{"class":1869},[151,53399,38718],{"class":503},[151,53401,53402],{"class":481},"'lambda'",[151,53404,106],{"class":503},[151,53406,53407],{"class":15210},"region_name",[151,53409,1876],{"class":1869},[151,53411,53412],{"class":481},"'us-east-1'",[151,53414,3640],{"class":503},[151,53416,53417],{"class":469,"line":534},[151,53418,1090],{"emptyLinePlaceholder":609},[151,53420,53421],{"class":469,"line":1413},[151,53422,1090],{"emptyLinePlaceholder":609},[151,53424,53425,53427,53429,53431,53433,53435,53437],{"class":469,"line":1418},[151,53426,16925],{"class":12347},[151,53428,39514],{"class":473},[151,53430,12386],{"class":503},[151,53432,39519],{"class":15232},[151,53434,106],{"class":503},[151,53436,39524],{"class":15232},[151,53438,15264],{"class":503},[151,53440,53441,53444,53446],{"class":469,"line":2462},[151,53442,53443],{"class":503},"    invoke_response ",[151,53445,1876],{"class":1869},[151,53447,53448],{"class":503}," lambda_client.invoke(\n",[151,53450,53451,53454,53456,53458,53461,53463,53465],{"class":469,"line":2471},[151,53452,53453],{"class":15210},"        FunctionName",[151,53455,1876],{"class":1869},[151,53457,39030],{"class":503},[151,53459,53460],{"class":481},"\"FUNCTION_NAME\"",[151,53462,106],{"class":503},[151,53464,15437],{"class":477},[151,53466,37985],{"class":503},[151,53468,53469,53472,53474,53477],{"class":469,"line":2480},[151,53470,53471],{"class":15210},"        InvocationType",[151,53473,1876],{"class":1869},[151,53475,53476],{"class":481},"'RequestResponse'",[151,53478,9417],{"class":503},[151,53480,53481,53484,53486],{"class":469,"line":2489},[151,53482,53483],{"class":15210},"        Payload",[151,53485,1876],{"class":1869},[151,53487,53488],{"class":503},"json.dumps(event),\n",[151,53490,53491],{"class":469,"line":2497},[151,53492,39567],{"class":503},[151,53494,53495],{"class":469,"line":3140},[151,53496,1090],{"emptyLinePlaceholder":609},[151,53498,53499,53502,53504,53507,53510],{"class":469,"line":3149},[151,53500,53501],{"class":503},"    data ",[151,53503,1876],{"class":1869},[151,53505,53506],{"class":503}," invoke_response[",[151,53508,53509],{"class":481},"'Payload'",[151,53511,53512],{"class":503},"].read()\n",[151,53514,53515],{"class":469,"line":3158},[151,53516,1090],{"emptyLinePlaceholder":609},[151,53518,53519,53521],{"class":469,"line":3167},[151,53520,17496],{"class":1869},[151,53522,53523],{"class":503}," data\n",[11,53525,53526],{},"There are just a few things to setup in our CDK code to make the proxy pattern work:",[700,53528,53529,53532,53535],{},[79,53530,53531],{},"Define the Lambda function",[79,53533,53534],{},"Give the proxy Lambda permission to invoke the Django Lambda",[79,53536,53537,53538,53541,53542],{},"Define a ",[30,53539,53540],{},"LambdaRestApi"," construct with the Proxy Lambda as the ",[30,53543,51756],{},[11,53545,53546],{},"Here's what that code looks like:",[459,53548,53550],{"className":24401,"code":53549,"language":24403,"meta":464,"style":464},"        self.proxy_lambda = _lambda.Function(\n            self,\n            \"ProxyLambda\",\n            code=_lambda.AssetCode(\"./awslambda\"),\n            runtime=_lambda.Runtime.PYTHON_3_8,\n            layers=[self.djambda_layer],\n            handler=\"proxy_lambda.handler\",\n            timeout=core.Duration.seconds(60),\n            environment={\"FUNCTION_NAME\": self.djambda_lambda.function_name},\n        )\n\n        self.djambda_lambda.grant_invoke(self.proxy_lambda)\n\n        self.apigw = apigw.LambdaRestApi(\n            self, 'DjambdaEndpoint', handler=self.proxy_lambda,\n        )\n",[30,53551,53552,53563,53569,53576,53589,53601,53613,53624,53636,53653,53657,53661,53673,53677,53689,53709],{"__ignoreMap":464},[151,53553,53554,53556,53559,53561],{"class":469,"line":470},[151,53555,37901],{"class":15289},[151,53557,53558],{"class":503},".proxy_lambda ",[151,53560,1876],{"class":1869},[151,53562,39734],{"class":503},[151,53564,53565,53567],{"class":469,"line":488},[151,53566,15290],{"class":15289},[151,53568,9417],{"class":503},[151,53570,53571,53574],{"class":469,"line":500},[151,53572,53573],{"class":481},"            \"ProxyLambda\"",[151,53575,9417],{"class":503},[151,53577,53578,53580,53582,53584,53587],{"class":469,"line":509},[151,53579,39688],{"class":15210},[151,53581,1876],{"class":1869},[151,53583,39693],{"class":503},[151,53585,53586],{"class":481},"\"./awslambda\"",[151,53588,37985],{"class":503},[151,53590,53591,53593,53595,53597,53599],{"class":469,"line":517},[151,53592,39752],{"class":15210},[151,53594,1876],{"class":1869},[151,53596,39757],{"class":503},[151,53598,39711],{"class":477},[151,53600,9417],{"class":503},[151,53602,53603,53605,53607,53609,53611],{"class":469,"line":534},[151,53604,39803],{"class":15210},[151,53606,1876],{"class":1869},[151,53608,6698],{"class":503},[151,53610,15277],{"class":15289},[151,53612,52426],{"class":503},[151,53614,53615,53617,53619,53622],{"class":469,"line":1413},[151,53616,39791],{"class":15210},[151,53618,1876],{"class":1869},[151,53620,53621],{"class":481},"\"proxy_lambda.handler\"",[151,53623,9417],{"class":503},[151,53625,53626,53628,53630,53632,53634],{"class":469,"line":1418},[151,53627,39817],{"class":15210},[151,53629,1876],{"class":1869},[151,53631,39822],{"class":503},[151,53633,39825],{"class":477},[151,53635,37985],{"class":503},[151,53637,53638,53640,53642,53644,53646,53648,53650],{"class":469,"line":2462},[151,53639,38021],{"class":15210},[151,53641,1876],{"class":1869},[151,53643,5729],{"class":503},[151,53645,53460],{"class":481},[151,53647,6208],{"class":503},[151,53649,15277],{"class":15289},[151,53651,53652],{"class":503},".djambda_lambda.function_name},\n",[151,53654,53655],{"class":469,"line":2471},[151,53656,16824],{"class":503},[151,53658,53659],{"class":469,"line":2480},[151,53660,1090],{"emptyLinePlaceholder":609},[151,53662,53663,53665,53668,53670],{"class":469,"line":2489},[151,53664,37901],{"class":15289},[151,53666,53667],{"class":503},".djambda_lambda.grant_invoke(",[151,53669,15277],{"class":15289},[151,53671,53672],{"class":503},".proxy_lambda)\n",[151,53674,53675],{"class":469,"line":2497},[151,53676,1090],{"emptyLinePlaceholder":609},[151,53678,53679,53681,53684,53686],{"class":469,"line":3140},[151,53680,37901],{"class":15289},[151,53682,53683],{"class":503},".apigw ",[151,53685,1876],{"class":1869},[151,53687,53688],{"class":503}," apigw.LambdaRestApi(\n",[151,53690,53691,53693,53695,53698,53700,53702,53704,53706],{"class":469,"line":3149},[151,53692,15290],{"class":15289},[151,53694,106],{"class":503},[151,53696,53697],{"class":481},"'DjambdaEndpoint'",[151,53699,106],{"class":503},[151,53701,51756],{"class":15210},[151,53703,1876],{"class":1869},[151,53705,15277],{"class":15289},[151,53707,53708],{"class":503},".proxy_lambda,\n",[151,53710,53711],{"class":469,"line":3158},[151,53712,16824],{"class":503},[11,53714,53715,53716,53719],{},"That's it! We can now access our Django project at the API Gateway ",[30,53717,53718],{},"execute-api"," endpoint. This is a special URL that has the format:",[459,53721,53724],{"className":53722,"code":53723,"language":997},[995],"https://abc123.execute-api.us-east-1.amazonaws.com/prod\n",[30,53725,53723],{"__ignoreMap":464},[76,53727,53728,53734,53740],{},[79,53729,53730,53733],{},[30,53731,53732],{},"abc123"," is the id of the API Gateway that we created (you can find the value of this id in the API Gateway section of the AWS management console)]",[79,53735,53736,53737,53739],{},"The region ",[30,53738,33086],{}," is the region where you deployed the API Gateway",[79,53741,53742,53745],{},[30,53743,53744],{},"/prod"," is the stage name",[11,53747,53748,53749,53751,53752,53754,53755,53757,53758,53761],{},"The stage name part is a little confusing for me. It is meant to be a URL suffix that indicates production, staging, etc., but I typically separate environments at the level of the CloudFormation stack, so all of my environments use the ",[30,53750,53744],{}," suffix. In order for Django to work properly, we need to tell it that our site will be served at this ",[30,53753,53744],{}," path (for example ",[30,53756,30586],{}," will now be served at ",[30,53759,53760],{},"/prod/admin",") by setting a special value in our settings module:",[459,53763,53765],{"className":24401,"code":53764,"language":24403,"meta":464,"style":464},"FORCE_SCRIPT_NAME = \"/prod\"\n",[30,53766,53767],{"__ignoreMap":464},[151,53768,53769,53772,53774],{"class":469,"line":470},[151,53770,53771],{"class":477},"FORCE_SCRIPT_NAME",[151,53773,19865],{"class":1869},[151,53775,53289],{"class":481},[11,53777,53778,53779,53782,53783,53785,53786,53788,53789,53791,53792,53795,53796,53798],{},"This will make dealing with URLs easier, mostly because we don't actually have to change anything in our ",[30,53780,53781],{},"urls.py",". I tried adding ",[30,53784,53744],{}," to my URL paths in ",[30,53787,53781],{},", but the Django admin doesn't work properly because going to ",[30,53790,53760],{}," will redirect you to ",[30,53793,53794],{},"/admin/login"," which will thrown an error with API Gateway since our application must be on the ",[30,53797,53744],{}," subpath.",[56,53800,53802],{"id":53801},"setting-a-custom-url-for-api-gateway","Setting a custom URL for API Gateway",[11,53804,53805,53806,53809],{},"If you don't mind having a URL in the form of ",[30,53807,53808],{},"https://abc123.execute-api.us-east-1.amazonaws.com/prod",", then everything should be good to go. If you do want a custom URL for API Gateway, there are only two things you need to do:",[700,53811,53812,53815,53823,53826],{},[79,53813,53814],{},"Get a domain and hosted zone set up in Route53 (this typically costs about $12/year)",[79,53816,33670,53817,29198,53819,53822],{},[30,53818,33097],{},[30,53820,53821],{},"HOSTED_ZONE_ID"," to environment variables (see below for how to do this)",[79,53824,53825],{},"Define an ACM certificate in our CDK code and",[79,53827,33348,53828,53830,53831,53834],{},[30,53829,53540],{},"'s ",[30,53832,53833],{},"add_domain_name"," method to add the domain name and certificate to the API Gateway endpoint.",[11,53836,53837,53838,33,53843,643],{},"Here's a ",[20,53839,53842],{"href":53840,"rel":53841},"https://docs.aws.amazon.com/cdk/api/latest/python/aws_cdk.aws_apigateway/LambdaRestApi.html#aws_cdk.aws_apigateway.LambdaRestApi.add_domain_name",[24],"link to the CDK Python documentation",[30,53844,53833],{},[11,53846,53847,53848,53850],{},"When you configure a custom domain name, you don't need to set the ",[30,53849,53771],{}," setting in Django settings.",[56,53852,53854],{"id":53853},"project-directory-structure","Project directory structure",[11,53856,53857,53858,208],{},"Let's take a quick look at the structure of the project using ",[30,53859,53860],{},"tree",[459,53862,53865],{"className":53863,"code":53864,"language":997},[995],"tree -L 3 .\n.\n├── awscdk\n│   ├── app.py\n│   ├── awscdk\n│   │   ├── app_stack.py  \u003C----------------- CDK application overview\n│   │   ├── backend_assets.py\n│   │   ├── cert.py\n│   │   ├── cloudfront.py\n│   │   ├── hosted_zone.py\n│   │   ├── __init__.py\n│   │   ├── static_site_bucket.py\n│   │   └── vpc.py \u003C------------------------ VPC, Lambdas, API Gateway and more\n│   ├── cdk.json                             defined here\n│   ├── README.md\n│   ├── requirements.txt\n│   ├── setup.py\n│   └── source.bat\n├── awslambda\n│   └── proxy_lambda.py \u003C------------------- Proxy Lambda\n├── django\n│   ├── core \u003C------------------------------ A sample Django app\n│   │   ├── admin.py\n│   │   ├── apps.py\n│   │   ├── __init__.py\n│   │   ├── migrations\n│   │   ├── models.py\n│   │   ├── tests.py\n│   │   ├── urls.py\n│   │   └── views.py\n│   ├── djambda\n│   │   ├── asgi.py\n│   │   ├── awsgi.py \u003C---------------------- Django Lambda handler\n│   │   ├── __init__.py\n│   │   ├── settings.py \u003C------------------- Django settings (RDS connections, S3 static/media)\n│   │   ├── storage_backends.py\n│   │   ├── urls.py\n│   │   └── wsgi.py\n│   ├── manage.py\n│   └── requirements.txt\n├── layers \u003C-------------------------------- Lambda Layers (python dependencies)\n│   └── django                               Note: This folder is not committed to git\n│       └── python\n├── ARTICLE.md \u003C---------------------------- This article\n├── gitlab-ci.yml\n├── .variables \u003C---------------------------- Environment variables needed for deployment\n└── README.md\n",[30,53866,53864],{"__ignoreMap":464},[11,53868,53869,53870,106,53873,187,53876,53878],{},"The three top level folders are ",[30,53871,53872],{},"awscdk",[30,53874,53875],{},"awslambda",[30,53877,30122],{},". To deploy our CDK code, we run the following command from the root of the project:",[459,53880,53882],{"className":461,"code":53881,"language":463,"meta":464,"style":464},"cdk deploy --app awscdk/app.py\n",[30,53883,53884],{"__ignoreMap":464},[151,53885,53886,53888,53891,53894],{"class":469,"line":470},[151,53887,30123],{"class":473},[151,53889,53890],{"class":481}," deploy",[151,53892,53893],{"class":477}," --app",[151,53895,53896],{"class":481}," awscdk/app.py\n",[11,53898,53899,53900,53903,53904,53907,53908,187,53911,53913],{},"We can also run ",[30,53901,53902],{},"cdk synth --app awscdk/app.py"," to preview changes to our CDK code by reviewing the CloudFormation templates generated by ",[30,53905,53906],{},"cdk synth",". Running the ",[30,53909,53910],{},"cdk deploy",[30,53912,53906],{}," commands at root of the project means that we need to specificy lambda code assets from the root of the project as well. This is why the Django application's Lambda function references the code with:",[459,53915,53917],{"className":24401,"code":53916,"language":24403,"meta":464,"style":464},"code=_lambda.AssetCode('./django'),\n",[30,53918,53919],{"__ignoreMap":464},[151,53920,53921,53923,53925,53927,53929],{"class":469,"line":470},[151,53922,30],{"class":503},[151,53924,1876],{"class":1869},[151,53926,39693],{"class":503},[151,53928,52378],{"class":481},[151,53930,37985],{"class":503},[56,53932,53934],{"id":53933},"deploying","Deploying",[11,53936,53937],{},"CDK is easy to use both locally and in a CI/CD tool such as GitLab CI, my CI/CD tool of choice. There are a few things to coordinate when deploying in both of these environments:",[700,53939,53940,53943],{},[79,53941,53942],{},"Environment variables",[79,53944,53945],{},"Dependencies",[736,53947,53949],{"id":53948},"deploying-from-a-terminal","Deploying from a terminal",[11,53951,53952,53953,53956],{},"If you are new to CDK, you will need to make sure that you have ran the ",[30,53954,53955],{},"cdk bootstrap"," command at least once on your account. This command sets up a CloudFormation stack that CDK uses internally to manage deployments.",[11,53958,53959,53960,208],{},"The environment variables needed for deployment are defined in ",[30,53961,53962],{},".variables.template",[459,53964,53967],{"className":53965,"code":53966,"language":997},[995],"export APP_NAME=\nexport AWS_ACCESS_KEY_ID=\nexport AWS_ACCOUNT_ID=\nexport AWS_DEFAULT_REGION=\nexport AWS_SECRET_ACCESS_KEY=\nexport DOMAIN_NAME=\nexport HOSTED_ZONE_ID=\n",[30,53968,53966],{"__ignoreMap":464},[76,53970,53971,53984],{},[79,53972,53973,53976,53977,53980,53981,53983],{},[30,53974,53975],{},"APP_NAME"," should be a URL compatible name, such as ",[30,53978,53979],{},"my-app"," (don't put ",[30,53982,643],{}," in this variable)",[79,53985,53986,187,53988,53990],{},[30,53987,33097],{},[30,53989,53821],{}," are needed for adding a custom domain name to API Gateway",[11,53992,53993,53994,53997],{},"Copy this template into a file in the root of the project called ",[30,53995,53996],{},".variables"," and set these environment variables with the following command:",[459,53999,54002],{"className":54000,"code":54001,"language":997},[995],". .variables\n",[30,54003,54001],{"__ignoreMap":464},[11,54005,54006,54007,54009],{},"Next, we need to install our Python dependencies locally with the ",[30,54008,40260],{}," target flag so that the Lambda layers we define in CDK can find the files needed to be packaged into our Lambda layer and made available to our Lambda function at runtime. Install dependencies to the target directory by running the following command from the root of the project:",[459,54011,54014],{"className":54012,"code":54013,"language":997},[995],"pip install -r django/requirements.txt -t layers/django/python\n",[30,54015,54013],{"__ignoreMap":464},[11,54017,54018],{},"Now we can deploy our serverless application to AWS using CDK. First, activate the CDK Python virtual environment with:",[459,54020,54023],{"className":54021,"code":54022,"language":997},[995],"source awscdk/.env/bin/activate\n",[30,54024,54022],{"__ignoreMap":464},[11,54026,54027,54028,643],{},"Also make sure that you are using a version of Node.js greater or equal to 10. I do this with ",[30,54029,54030],{},"nvm use 13",[11,54032,54033,54034,54037],{},"Make sure that all of the dependencies defined in ",[30,54035,54036],{},"awscdk/setup.py"," are up-to-date. Make sure that there are no issues witht the CDK code by running:",[459,54039,54042],{"className":54040,"code":54041,"language":997},[995],"cdk synth --app awscdk/app.py\n",[30,54043,54041],{"__ignoreMap":464},[11,54045,54046,54047,54050],{},"Inspect the contents of ",[30,54048,54049],{},"cdk.out","; these are the CloudFormation templates generated by CDK that will be used to deploy all of our resources. If everything looks good to go, deploy with the following command:",[459,54052,54054],{"className":54053,"code":54041,"language":997},[995],[30,54055,54041],{"__ignoreMap":464},[11,54057,54058],{},"Follow the output of this command and ensure that it completes successfully. You can also follow along in the CloudFormation section of the AWS management console to make sure that things are deploying properly.",[11,54060,54061,54062,106,54064,187,54066,54068],{},"Once everything has finished deploying, you will need to run ",[30,54063,27589],{},[30,54065,51591],{},[30,54067,50956],{}," after the first deployment.",[210,54070,54071],{},[11,54072,54073],{},"TODO: consolidate these steps with a Makefile or bash script",[736,54075,54077],{"id":54076},"deploying-with-gitlab-ci","Deploying with GitLab CI",[11,54079,54080],{},"Deploying locally is fine, but it is better to run CDK commands from a CI/CD process so we can keep track of the commits, pipeline results, commit messages, etc. The process is very similar to everything we do locally, but you will need to add environment variables to the GitLab project's Settings > CI/CD > Variables. I have broken out dependency installation into a separate stage. This is not entirely necessary, but it helps keep things easy to follow:",[459,54082,54084],{"className":14359,"code":54083,"language":14361,"meta":464,"style":464},"pip_install:\n  stage: build\n  artifacts:\n    paths:\n      - layers/django/python\n  script:\n    - pip install -r django/requirements.txt -t layers/django/python\n\ncdk_deploy:\n  stage: deploy\n  before_script:\n    - apt-get -qq update && apt-get -y install nodejs npm\n    - npm i -g aws-cdk\n    - pip3 install -e awscdk\n  script:\n    - cdk bootstrap --app awscdk/app.py aws://$AWS_ACCOUNT_ID/$AWS_DEFAULT_REGION\n    - cdk deploy --app awscdk/app.py --require-approval never\n",[30,54085,54086,54092,54100,54106,54112,54119,54125,54131,54135,54141,54149,54155,54161,54167,54174,54180,54187],{"__ignoreMap":464},[151,54087,54088,54090],{"class":469,"line":470},[151,54089,40186],{"class":14368},[151,54091,14372],{"class":503},[151,54093,54094,54096,54098],{"class":469,"line":488},[151,54095,38176],{"class":14368},[151,54097,6208],{"class":503},[151,54099,40159],{"class":481},[151,54101,54102,54104],{"class":469,"line":500},[151,54103,40225],{"class":14368},[151,54105,14372],{"class":503},[151,54107,54108,54110],{"class":469,"line":509},[151,54109,40232],{"class":14368},[151,54111,14372],{"class":503},[151,54113,54114,54116],{"class":469,"line":517},[151,54115,14459],{"class":503},[151,54117,54118],{"class":481},"layers/django/python\n",[151,54120,54121,54123],{"class":469,"line":534},[151,54122,38240],{"class":14368},[151,54124,14372],{"class":503},[151,54126,54127,54129],{"class":469,"line":1413},[151,54128,29541],{"class":503},[151,54130,54013],{"class":481},[151,54132,54133],{"class":469,"line":1418},[151,54134,1090],{"emptyLinePlaceholder":609},[151,54136,54137,54139],{"class":469,"line":2462},[151,54138,38169],{"class":14368},[151,54140,14372],{"class":503},[151,54142,54143,54145,54147],{"class":469,"line":2471},[151,54144,38176],{"class":14368},[151,54146,6208],{"class":503},[151,54148,20676],{"class":481},[151,54150,54151,54153],{"class":469,"line":2480},[151,54152,38213],{"class":14368},[151,54154,14372],{"class":503},[151,54156,54157,54159],{"class":469,"line":2489},[151,54158,29541],{"class":503},[151,54160,38222],{"class":481},[151,54162,54163,54165],{"class":469,"line":2497},[151,54164,29541],{"class":503},[151,54166,37580],{"class":481},[151,54168,54169,54171],{"class":469,"line":3140},[151,54170,29541],{"class":503},[151,54172,54173],{"class":481},"pip3 install -e awscdk\n",[151,54175,54176,54178],{"class":469,"line":3149},[151,54177,38240],{"class":14368},[151,54179,14372],{"class":503},[151,54181,54182,54184],{"class":469,"line":3158},[151,54183,29541],{"class":503},[151,54185,54186],{"class":481},"cdk bootstrap --app awscdk/app.py aws://$AWS_ACCOUNT_ID/$AWS_DEFAULT_REGION\n",[151,54188,54189,54191],{"class":469,"line":3167},[151,54190,29541],{"class":503},[151,54192,54193],{"class":481},"cdk deploy --app awscdk/app.py --require-approval never\n",[11,54195,54196,54197,54200],{},"For management commands, we use a ",[30,54198,54199],{},".base_task"," template and reuse this for each command:",[459,54202,54204],{"className":14359,"code":54203,"language":14361,"meta":464,"style":464},".base_task: &task\n  image: python:3.8\n  stage: deploy\n  rules:\n    - when: manual\n  before_script:\n    - pip install awscli\n  after_script:\n    - cat invoke_response.json\n\nmigrate:\n  \u003C\u003C: *task\n  script:\n    - |\n      aws lambda invoke \\\n        --function-name ${ENVIRONMENT}-${APP_NAME}-djambda-lambda \\\n        --payload '{\"manage\": \"migrate --no-input\"}' \\\n        invoke_response.json\n",[30,54205,54206,54218,54226,54234,54240,54252,54258,54265,54272,54279,54283,54289,54300,54306,54312,54317,54322,54327],{"__ignoreMap":464},[151,54207,54208,54210,54212,54215],{"class":469,"line":470},[151,54209,54199],{"class":14368},[151,54211,6208],{"class":503},[151,54213,54214],{"class":1869},"&",[151,54216,54217],{"class":15254},"task\n",[151,54219,54220,54222,54224],{"class":469,"line":488},[151,54221,22226],{"class":14368},[151,54223,6208],{"class":503},[151,54225,38160],{"class":481},[151,54227,54228,54230,54232],{"class":469,"line":500},[151,54229,38176],{"class":14368},[151,54231,6208],{"class":503},[151,54233,20676],{"class":481},[151,54235,54236,54238],{"class":469,"line":509},[151,54237,38185],{"class":14368},[151,54239,14372],{"class":503},[151,54241,54242,54244,54247,54249],{"class":469,"line":517},[151,54243,29541],{"class":503},[151,54245,54246],{"class":14368},"when",[151,54248,6208],{"class":503},[151,54250,54251],{"class":481},"manual\n",[151,54253,54254,54256],{"class":469,"line":534},[151,54255,38213],{"class":14368},[151,54257,14372],{"class":503},[151,54259,54260,54262],{"class":469,"line":1413},[151,54261,29541],{"class":503},[151,54263,54264],{"class":481},"pip install awscli\n",[151,54266,54267,54270],{"class":469,"line":1418},[151,54268,54269],{"class":14368},"  after_script",[151,54271,14372],{"class":503},[151,54273,54274,54276],{"class":469,"line":2462},[151,54275,29541],{"class":503},[151,54277,54278],{"class":481},"cat invoke_response.json\n",[151,54280,54281],{"class":469,"line":2471},[151,54282,1090],{"emptyLinePlaceholder":609},[151,54284,54285,54287],{"class":469,"line":2480},[151,54286,27589],{"class":14368},[151,54288,14372],{"class":503},[151,54290,54291,54294,54296,54298],{"class":469,"line":2489},[151,54292,54293],{"class":477},"  \u003C\u003C",[151,54295,6208],{"class":503},[151,54297,23268],{"class":1869},[151,54299,54217],{"class":503},[151,54301,54302,54304],{"class":469,"line":2497},[151,54303,38240],{"class":14368},[151,54305,14372],{"class":503},[151,54307,54308,54310],{"class":469,"line":3140},[151,54309,29541],{"class":503},[151,54311,20607],{"class":1869},[151,54313,54314],{"class":469,"line":3149},[151,54315,54316],{"class":481},"      aws lambda invoke \\\n",[151,54318,54319],{"class":469,"line":3158},[151,54320,54321],{"class":481},"        --function-name ${ENVIRONMENT}-${APP_NAME}-djambda-lambda \\\n",[151,54323,54324],{"class":469,"line":3167},[151,54325,54326],{"class":481},"        --payload '{\"manage\": \"migrate --no-input\"}' \\\n",[151,54328,54329],{"class":469,"line":3175},[151,54330,54331],{"class":481},"        invoke_response.json\n",[11,54333,54334,54335,54338],{},"These are ",[30,54336,54337],{},"manual"," commands, and can be started through the GitLab CI interface by pressing the \"Play\" button on the pipeline.",[56,54340,47139],{"id":47138},[11,54342,54343],{},"I still have lots of ideas and things to try out with this Django/Lambda architecture. Here are a few things that would be good to try:",[76,54345,54346,54356,54363,54370,54382,54396],{},[79,54347,54348,54349,30583,54352,54355],{},"Using another Lambda function that proxies web requests to a Lambda outside of the VPC for basic network requests using either ",[30,54350,54351],{},"urllib.request",[30,54353,54354],{},"requests",". This would be the easiest way to add simple internet access without needing a NAT Gateway",[79,54357,54358,54359,54362],{},"Use a NAT provider to add a cheap NAT instance using a ",[30,54360,54361],{},"t3a.nano"," instance ( about $5.00/month) to allow for internet access in the Django request/response cycle",[79,54364,54365,54366,54369],{},"Figure out a good solution for async processing. Zappa has a ",[30,54367,54368],{},"@task"," wrapper that allows you to run tasks asynchronously in separate Lambda functions. It would be interesting to experiment with direct invocation of async tasks as well as task queueing with SQS. Using SQS would involve a VPC Endpoint which does cost extra, or we could setup a dedicated SQS proxy function to do this in a way similar to how we would handle requests.",[79,54371,54372,54373,54375,54376,54381],{},"Lambda tuning. I'm pretty new to using Lambda and I would like to better understand how to fine-tune Lambda settings. There are lots of options in the ",[30,54374,52584],{}," construct, which would be a good place to start. This would be especially important for async tasks that require high memeory. From the ",[20,54377,54380],{"href":54378,"rel":54379},"https://docs.aws.amazon.com/lambda/latest/dg/lambda-dg.pdf",[24],"Lambda Developer Guide"," it looks like memory can be configured from 128 MB to 3,008 MB in 64 MB increments.",[79,54383,54384,54385,54387,54388,54391,54392,54395],{},"I typically setup Vue.js frontends with S3/CloudFront and use Django only for the admin and API with Django REST Framework. From what I understand, API Gateway uses CloudFront in the backround to do custom domains, but you don't have access to this CloudFront distribution. You can add the ",[30,54386,53718],{}," URL as a Custom Origin for a single CloudFront distribution, or keep a CloudFront distribution separate from the API Gateway custom domain and serve these on different subdomains, such as ",[30,54389,54390],{},"api.mysite.com"," for the API Gateway domain name and ",[30,54393,54394],{},"mysite.com"," for the CloudFront distribution serving Vue.js files.",[79,54397,54398,54399],{},"Pricing. I'm still unsure about exactly how much this setup costs. The only major costs should be Aurora Postgres Serverless if it is used heavily, but I'm still not sure about how the pricing for this service works. Here's an in-depth article from Jeremy Daly that has more information on Aurora Serverless: ",[20,54400,54401],{"href":54401,"rel":54402},"https://www.jeremydaly.com/aurora-serverless-the-good-the-bad-and-the-scalable/",[24],[11,54404,40731],{},[589,54406,54407],{},"html pre.shiki code .sC2Qs, html code.shiki .sC2Qs{--shiki-default:#D73A49;--shiki-dark:#F97583;--shiki-sepia:#F92672}html pre.shiki code .sMOD_, html code.shiki .sMOD_{--shiki-default:#24292E;--shiki-dark:#E1E4E8;--shiki-sepia:#F8F8F2}html pre.shiki code .sq6CD, html code.shiki .sq6CD{--shiki-default:#D73A49;--shiki-default-font-style:inherit;--shiki-dark:#F97583;--shiki-dark-font-style:inherit;--shiki-sepia:#66D9EF;--shiki-sepia-font-style:italic}html pre.shiki code .srTi1, html code.shiki .srTi1{--shiki-default:#6F42C1;--shiki-dark:#B392F0;--shiki-sepia:#A6E22E}html pre.shiki code .so59x, html code.shiki .so59x{--shiki-default:#24292E;--shiki-default-font-style:inherit;--shiki-dark:#E1E4E8;--shiki-dark-font-style:inherit;--shiki-sepia:#FD971F;--shiki-sepia-font-style:italic}html pre.shiki code .s-m8C, html code.shiki .s-m8C{--shiki-default:#005CC5;--shiki-default-font-style:inherit;--shiki-dark:#79B8FF;--shiki-dark-font-style:inherit;--shiki-sepia:#66D9EF;--shiki-sepia-font-style:italic}html pre.shiki code .sstjo, html code.shiki .sstjo{--shiki-default:#032F62;--shiki-dark:#9ECBFF;--shiki-sepia:#E6DB74}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html .sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html.sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html pre.shiki code .s8-w5, html code.shiki .s8-w5{--shiki-default:#6A737D;--shiki-dark:#6A737D;--shiki-sepia:#88846F}html pre.shiki code .sTHNf, html code.shiki .sTHNf{--shiki-default:#E36209;--shiki-default-font-style:inherit;--shiki-dark:#FFAB70;--shiki-dark-font-style:inherit;--shiki-sepia:#FD971F;--shiki-sepia-font-style:italic}html pre.shiki code .s7F3e, html code.shiki .s7F3e{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#AE81FF}html pre.shiki code .sP7S_, html code.shiki .sP7S_{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#FD971F}html pre.shiki code .s5clZ, html code.shiki .s5clZ{--shiki-default:#22863A;--shiki-dark:#85E89D;--shiki-sepia:#F92672}html pre.shiki code .sz2Vg, html code.shiki .sz2Vg{--shiki-default:#6F42C1;--shiki-default-text-decoration:inherit;--shiki-dark:#B392F0;--shiki-dark-text-decoration:inherit;--shiki-sepia:#A6E22E;--shiki-sepia-text-decoration:underline}",{"title":464,"searchDepth":488,"depth":488,"links":54409},[54410,54411,54412,54413,54414,54415,54416,54417,54418,54422],{"id":16115,"depth":488,"text":16116},{"id":642,"depth":488,"text":51701},{"id":51734,"depth":488,"text":51735},{"id":26801,"depth":488,"text":26856},{"id":52814,"depth":488,"text":26847},{"id":53339,"depth":488,"text":53340},{"id":53801,"depth":488,"text":53802},{"id":53853,"depth":488,"text":53854},{"id":53933,"depth":488,"text":53934,"children":54419},[54420,54421],{"id":53948,"depth":500,"text":53949},{"id":54076,"depth":500,"text":54077},{"id":47138,"depth":488,"text":47139},"2020-08-01",{"layout":48045},"/2020/08/01/django-and-lambda-with-cdk-and-api-gateway",{"title":51683,"description":464},"2020/08/01/django-and-lambda-with-cdk-and-api-gateway",[40767,30122,27264,43773,30123],"MTGjWF2aLz8_VOD6VyyVBqTh2fdx0fbGnKM5nlQOV6Q",{"id":54431,"title":54432,"body":54433,"comments":609,"date":57505,"description":464,"draft":602,"extension":605,"external":606,"image":54542,"meta":57506,"navigation":609,"path":57507,"seo":57508,"stem":57509,"tags":57510,"__hash__":57511},"blog/2020/06/02/django-postgres-vue-gitlab-ecs.md","Using AWS CDK, GitLab, Fargate and CloudFront for Django + Vue.js applications",{"type":8,"value":54434,"toc":57466},[54435,54449,54453,54460,54483,54486,54490,54493,54534,54538,54543,54545,54554,54557,54560,54563,54566,54569,54572,54575,54578,54581,54584,54587,54590,54593,54596,54599,54602,54605,54608,54611,54614,54617,54620,54623,54626,54629,54632,54635,54638,54641,54644,54647,54649,54671,54675,54699,54701,54715,54720,54759,54763,54806,54810,54832,54836,54844,54847,54851,54891,54895,54928,54934,54938,54941,54945,54956,54960,54963,54977,54981,54993,54996,55023,55027,55035,55052,55070,55074,55077,55085,55088,55092,55101,55111,55182,55187,55584,55590,55624,55631,55669,55675,55687,55693,55709,55720,55729,55857,55870,55919,55925,55931,55936,55940,55945,56023,56041,56066,56076,56080,56090,56138,56155,56165,56171,56177,56182,56190,56195,56201,56206,56223,56228,56243,56248,56251,56256,56259,56264,56271,56278,56297,56307,56313,56319,56323,56326,56330,56348,56353,56356,56367,56373,56379,56384,56387,56392,56399,56434,56439,56446,56451,56454,56470,56475,56478,56483,56492,56503,56506,56510,56517,56537,56540,56544,56547,56551,56554,56564,56567,56575,56579,56582,56586,56589,56603,56617,56620,56626,56633,56636,56659,56662,56665,56671,56674,56684,56687,56729,56732,56918,56932,56935,57149,57152,57421,57425,57444,57446,57463],[210,54436,54437],{},[11,54438,54439,54440,54444,54445],{},"This article was originally posted on ",[20,54441,54442],{"href":54442,"rel":54443},"https://dev.to/briancaffey/building-a-django-vue-js-application-with-aws-cdk-and-gitlab-ci-also-how-to-scale-celery-workers-to-zero-1o6i",[24],". It is also available on the documentation site for the project: ",[20,54446,54447],{"href":54447,"rel":54448},"https://verbose-equals-true.gitlab.io/django-postgres-vue-gitlab-ecs/start/overview/",[24],[14063,54450,54452],{"id":54451},"project-overview","Project Overview",[11,54454,54455,54456,54459],{},"This is an overview of a Proof-of-Concept web application I'm working on called ",[30,54457,54458],{},"django-postgres-vue-gitlab-ecs",". This project aims to demonstrate the development and deployment of a web application using some of my favorite tools, languages and frameworks including:",[76,54461,54462,54464,54467,54470,54473,54476,54478],{},[79,54463,11356],{},[79,54465,54466],{},"Django",[79,54468,54469],{},"JavaScript",[79,54471,54472],{},"Vue.js/Quasar Framework",[79,54474,54475],{},"GitLab",[79,54477,51877],{},[79,54479,54480,54481],{},"and last but not least, ",[15,54482,49223],{},[11,54484,54485],{},"This README will start by describing some features of the application I'm building and how the different technologies are used together. I also share my experience in adopting CDK for managing cloud infrastructure on AWS. Finally, I discuss my solution to a specific question I have been trying to answer: what's the best way to scale Celery workers to zero to reduce Total Cost of Ownership?",[56,54487,54489],{"id":54488},"development-philosophies-and-best-practices","Development Philosophies and Best Practices",[11,54491,54492],{},"Here are some of the best practices that this project aims to use:",[76,54494,54495,54498,54500,54502,54505,54508,54511,54518,54521,54524,54527],{},[79,54496,54497],{},"Open-source, MIT-Licensed",[79,54499,25956],{},[79,54501,9861],{},[79,54503,54504],{},"Containerization with Docker",[79,54506,54507],{},"Testing",[79,54509,54510],{},"GitOps",[79,54512,54513,54514],{},"Serverless* ",[15,54515,54516],{},[151,54517,6760],{},[79,54519,54520],{},"Project documentation",[79,54522,54523],{},"Cost containment and tracking",[79,54525,54526],{},"KISS & DRY",[79,54528,54529,54530,54533],{},"Initial AWS console interaction is strictly limited to what can ",[51,54531,54532],{},"only"," be done through the AWS console, otherwise AWS CDK and AWS CLI (preferably in CI/CD pipelines) are the primary means of interacting with AWS resources and the AWS Console is treated as a \"read-only\" convenience.",[56,54535,54537],{"id":54536},"topics","Topics",[11,54539,54540],{},[2718,54541],{"alt":20386,"src":54542},"/static/architecture_verbose_equals_true.png",[736,54544,46184],{"id":46183},[210,54546,54547],{},[11,54548,54549,54550],{},"This diagram was created with draw.io. Here's the link to the a read-only version of the diagram on draw.io: ",[20,54551,54552],{"href":54552,"rel":54553},"https://drive.google.com/file/d/1gU61zjoW80fCusUcswU1zhEE5VFB1Z5U/view?usp=sharing",[24],[11,54555,54556],{},"1 - GitLab is used to host the source code, test the source code and deploy the application to AWS.",[11,54558,54559],{},"2 - Unit testing (see .gitlab-ci.yml)",[11,54561,54562],{},"2a - Pytest",[11,54564,54565],{},"2b - Jest",[11,54567,54568],{},"2c - Cypress",[11,54570,54571],{},"3 - Deployment phase (see /gitlab-ci/aws/cdk.yml)",[11,54573,54574],{},"3a - Quasar PWA assets are built if there are changes in the quasar directory",[11,54576,54577],{},"3b - AWS Cloud Development Kit (CDK) defines all infrastructure in AWS (4a - 12)",[11,54579,54580],{},"3c - AWS CLI is used to run Fargate tasks through manual GitLab CI jobs",[11,54582,54583],{},"4 - CDK Assets (ECR and S3 buckets that CDK uses internally to manage build assets and artifacts)",[11,54585,54586],{},"4a - Elastic Container Repository is used to manage the Django docker image used in various parts of the application",[11,54588,54589],{},"4b - S3 bucket used to store files associated with CDK and CloudFormation",[11,54591,54592],{},"5 - Route53 is used to route traffic to the CloudFront distribution",[11,54594,54595],{},"6 - CloudFront distribution that serves as the \"front desk\" of the application. It routes requests to to the correct CloudFront Origin",[11,54597,54598],{},"7 - CloudFront Origin Configurations",[11,54600,54601],{},"7a - S3 bucket for Quasar PWA assets",[11,54603,54604],{},"7b - Application Load Balancer for Django application (/api/, /admin/, /flower/, /ws/, /graphql/)",[11,54606,54607],{},"7c - S3 bucket for Django assets (static files, public media and private media)",[11,54609,54610],{},"8 - Web server and websocket servers",[11,54612,54613],{},"8a - Fargate service running uvicorn process (REST, GraphQL, Django Channels)",[11,54615,54616],{},"8b - Autoscaling Group for Fargate Service that serves Django API",[11,54618,54619],{},"9 - Celery and celery worker autoscaling",[11,54621,54622],{},"9a - Fargate service that is autoscaled between 0 and N Fargate tasks for a given celery queue",[11,54624,54625],{},"9b - Scheduled Event that triggers a Lambda to make a request to Django backend which collects celery queue metrics and published\nmetrics to CloudWatch using boto3",[11,54627,54628],{},"9c - Lambda event the makes a request to /api/celery-metrics/",[11,54630,54631],{},"9d - CloudWatch alarm that is used to scale the Fargate service for a celery queue",[11,54633,54634],{},"9e - Autoscaling group for celery Fargate service",[11,54636,54637],{},"10 - Fargate tasks that run Django management commands such as migrate and collectstatic. These are triggered from manual GitLab CI\njobs using the AWS CLI (3c)",[11,54639,54640],{},"11 - ElastiCache for Redis, used for Caching, Celery Broker, Channels Layer, etc.",[11,54642,54643],{},"12 - Aurora Postgres Serverless",[11,54645,54646],{},"Here's a list of some of the major topics covered:",[736,54648,29584],{"id":29583},[76,54650,54651,54656,54659,54662,54665,54668],{},[79,54652,54653,54654],{},"Setting up a local development environment using docker with one easy command: ",[30,54655,47326],{},[79,54657,54658],{},"Hot-reloading for all parts of the application in local development",[79,54660,54661],{},"Structuring a Django application (apps, settings modules, environment variables)",[79,54663,54664],{},"Automatic code formatting with black and prettier",[79,54666,54667],{},"Monitoring services (flower, pgadmin4, redis-commander)",[79,54669,54670],{},"VSCode settings and extensions",[736,54672,54674],{"id":54673},"gitlab-ci","GitLab CI",[76,54676,54677,54680,54683,54686,54689,54696],{},[79,54678,54679],{},"GitLab CI for running unit and integration tests",[79,54681,54682],{},"GitLab CI configuration for deployment to multiple environments (dev, prod) using AWS CDK",[79,54684,54685],{},"GitLab CI scheduled jobs for updating all project dependencies through automated merge requests with Renovate",[79,54687,54688],{},"GitLab CI manual jobs for running admin and management commands (database migrations, collectstatic, etc.)",[79,54690,54691,54692,54695],{},"How to deploy to multiple environments (master -> staging, tags -> prod pattern as well as optional ephemeral environments per commit) using the new GitLab CI ",[30,54693,54694],{},"rules"," syntax",[79,54697,54698],{},"Recording Cypress test videos with GitLab CI artifacts for manual review",[736,54700,51877],{"id":27264},[76,54702,54703,54706,54709,54712],{},[79,54704,54705],{},"Initial account setup",[79,54707,54708],{},"Generating keys used in GitLab CI for deployment",[79,54710,54711],{},"Opting in to new ARN format for ECS services and tasks (used for resource tagging) and container insights",[79,54713,54714],{},"AWS Cloud Development Kit (CDK) for Infrastructure as Code to manage all AWS resources with CloudFormation Stacks - Nested stacks for grouping related resources under parent tasks (one parent task represent on environment for the application, such as production)",[54716,54717,54719],"h4",{"id":54718},"aws-resources","AWS Resources",[76,54721,54722,54725,54732,54735,54738,54741,54744,54747,54750,54753,54756],{},[79,54723,54724],{},"Automated provisioning of Amazon Certificate Manager certificates for TLS (HTTPS)",[79,54726,54727,54728,748],{},"Multi-AZ VPC with security groups and NACLs (no NAT Gateways/NAT instances used* ",[15,54729,54730],{},[151,54731,6619],{},[79,54733,54734],{},"S3 buckets for static site hosting and Django static and media assets",[79,54736,54737],{},"CloudFront for serving Vue.js SPA (PWA) static assets, Django static/media files and proxy to Application Load Balancer which forwards to ECS",[79,54739,54740],{},"Fargate for multiple Django processes: gunicorn for backend API, daphne for Django Channels, celery for asynchronous tasks and celery beat for scheduled tasks",[79,54742,54743],{},"Autoscaling (between 0 and N Fargate tasks) for infrequent, compute/memory intensive and long-running celery tasks with custom CloudWatch metrics",[79,54745,54746],{},"Hosted Redis with ElastiCache for celery broker, application caching, django-constance, etc.",[79,54748,54749],{},"Aurora Postgres and Aurora Postgres Serverless for cost savings in staging environments",[79,54751,54752],{},"CDK Assets for automating S3 asset deployment and automated container building during deployment with AWS CDK",[79,54754,54755],{},"Thorough resource tagging by both application and environment (dev, prod, etc.) for comprehensive cost tracking",[79,54757,54758],{},"Optional bastion host for SSH access to bring up a Django shell in a container running on a container instance (this requires an EC2 instance)",[736,54760,54762],{"id":54761},"django-application","Django Application",[76,54764,54765,54768,54771,54774,54781,54784,54787,54794,54797,54800,54803],{},[79,54766,54767],{},"Django REST Framework",[79,54769,54770],{},"JWT-based authentication with django-rest-framework-simplejwt",[79,54772,54773],{},"Custom user model with email as username",[79,54775,54776,54777,54780],{},"Social authentication with ",[30,54778,54779],{},"python-social-auth"," (Google, Facebook, GitHub)",[79,54782,54783],{},"Pytest for unit testing",[79,54785,54786],{},"Settings broken into separate modules (development, ci, production)",[79,54788,54789,54790,54793],{},"Django apps organized in ",[30,54791,54792],{},"apps"," directory",[79,54795,54796],{},"S3 for static assets, public media and private media files in production, local file system for assets in local development",[79,54798,54799],{},"Multiple Celery queues for asynchronous tasks",[79,54801,54802],{},"Django Channels for websocket support with daphne",[79,54804,54805],{},"Graphene for GraphQL",[736,54807,54809],{"id":54808},"vuejs-application","Vue.js application",[76,54811,54812,54815,54818,54821,54824,54827,54829],{},[79,54813,54814],{},"Quasar Framework for creating a Vue.js app with multiple build targets (SPA, PWA, Electron, SSR and Cordova)",[79,54816,54817],{},"Internationalization",[79,54819,54820],{},"Dark/Light mode",[79,54822,54823],{},"Vuex for state management",[79,54825,54826],{},"vue-router",[79,54828,47421],{},[79,54830,54831],{},"Vue Apollo",[736,54833,54835],{"id":54834},"aws-django-vuejs","AWS ∩ Django ∩ Vue.js",[11,54837,54838,54839,54843],{},"CloudFront is used as a CDN and proxy in order to serve the frontend Vue.js PWA* ",[15,54840,54841],{},[151,54842,6557],{},", static and media assets, Django API and other monitoring services all on the same URL with environments namespaced by subdomain (or a combination of domain and subdomains by production and non-production environments, respectively). The PWA uses Workbox to route certain URL paths to network only (not served by the local cache).",[11,54845,54846],{},"Here are some examples:",[54716,54848,54850],{"id":54849},"staging-environment-urls","Staging environment URLs",[76,54852,54853,54859,54865,54871,54877,54883],{},[79,54854,54855,54858],{},[30,54856,54857],{},"https://dev.mysite.com/api/*"," - Django REST Framework",[79,54860,54861,54864],{},[30,54862,54863],{},"https://dev.mysite.com/admin/*"," - Django Admin",[79,54866,54867,54870],{},[30,54868,54869],{},"https://dev.mysite.com/graphql/"," - Graphene/GraphQL",[79,54872,54873,54876],{},[30,54874,54875],{},"https://dev.mysite.com/flower/*"," - Flower (Celery monitoring utility)",[79,54878,54879,54882],{},[30,54880,54881],{},"https://dev.mysite.com/media/private/private-file.csv"," - private media files",[79,54884,54885,54888,54889,748],{},[30,54886,54887],{},"https://dev.mysite.com/*"," - Vue.js PWA (all other routes load ",[30,54890,46303],{},[54716,54892,54894],{"id":54893},"production-environment-urls","Production environment URLs",[76,54896,54897,54902,54907,54912,54917,54922],{},[79,54898,54899,54858],{},[30,54900,54901],{},"https://mysite.com/api/*",[79,54903,54904,54864],{},[30,54905,54906],{},"https://mysite.com/admin/*",[79,54908,54909,54870],{},[30,54910,54911],{},"https://mysite.com/graphql/",[79,54913,54914,54876],{},[30,54915,54916],{},"https://mysite.com/flower/*",[79,54918,54919,54882],{},[30,54920,54921],{},"https://mysite.com/media/private/private-file.csv",[79,54923,54924,54888,54926,748],{},[30,54925,47731],{},[30,54927,46303],{},[11,54929,54930,54931,643],{},"Alternatively, the production domain name could be configured to use a dedicated subdomain such as ",[30,54932,54933],{},"https://app.mysite.com",[736,54935,54937],{"id":54936},"sample-application-logic","Sample application logic",[11,54939,54940],{},"While this is mostly a Proof-of-Concept project that focuses on architecture, I have included some light-weight examples of what you can do with this application. These examples are all WIP.",[54716,54942,54944],{"id":54943},"social-authentication","Social Authentication",[76,54946,54947,54953],{},[79,54948,54949,54950,54952],{},"Uses ",[30,54951,54779],{}," to allow users to Sign Up/Login with Google, Facebook and GitHub accounts",[79,54954,54955],{},"Makes use of the custom user model",[54716,54957,54959],{"id":54958},"credit-card-statement-app","Credit Card Statement App",[11,54961,54962],{},"This is a simple application for users to upload credit card statements in CSV format.",[76,54964,54965,54968,54971,54974],{},[79,54966,54967],{},"Credit card transactions in CSV files are created using bulk inserts via the Django ORM in a celery task",[79,54969,54970],{},"CSV files are saved in private S3 storage",[79,54972,54973],{},"Basic visualization of spend over time",[79,54975,54976],{},"Download consolidated CSV file",[54716,54978,54980],{"id":54979},"hacker-news-clone","Hacker News Clone",[76,54982,54983,54990],{},[79,54984,54985,54986],{},"An implementation of the application from this tutorial: ",[20,54987,54988],{"href":54988,"rel":54989},"https://www.howtographql.com/graphql-python/0-introduction/",[24],[79,54991,54992],{},"Uses Vue Apollo GraphQL client",[736,54994,54507],{"id":54995},"testing",[76,54997,54998,55001,55004,55007,55010,55016],{},[79,54999,55000],{},"Unit testing with pytest and Jest",[79,55002,55003],{},"Test coverage reports",[79,55005,55006],{},"Integration testing with Cypress",[79,55008,55009],{},"Capture integration test run recordings as GitLab CI job artifacts",[79,55011,55012,55013],{},"Testing GitLab CI jobs locally with ",[30,55014,55015],{},"gitlab-runner",[79,55017,55018,55019],{},"Tests for CDK?* ",[15,55020,55021],{},[151,55022,9187],{},[736,55024,55026],{"id":55025},"caveats-questions-confusion-and-footnotes","Caveats, Questions, Confusion and Footnotes",[700,55028,55029,55032],{},[79,55030,55031],{},"In the context of this project, it is serverless in the sense that AWS Fargate is serverless: there are no EC2 instances to manage. It is not serverless in the way that Zappa can deploy a Django application as an AWS Lambda function with one invocation per request. There are \"always on\" processes that listen for incoming requests. I'm interested in trying Zappa, but I'm confused about how CDK, zappa-cli, SAM and Serverless Framework would all play together. Aurora Postgres Serverless is another nice \"serverless\" aspect of this project and contributes significantly to cost savings.",[79,55033,55034],{},"This project currently uses no NAT Gateways. The Fargate services and tasks running Django processes (gunicorn, celery, daphne as well as migration, collectstatic and other management commands) are launched in public subnets and the databases (RDS and ElastiCache) are placed in private subnets. This is primarily done to avoid the cost of running a NAT Gateway.",[76,55036,55037,55044],{},[79,55038,55039,55040,643],{},"I think it would fairly easy to switch to using a NAT Gatway to add an additional layer of security that is recommended in this article: ",[20,55041,55042],{"href":55042,"rel":55043},"https://aws.amazon.com/blogs/compute/task-networking-in-aws-fargate/",[24],[79,55045,55046,55047,55051],{},"I asked a question about this on the Stack Exchange Information Security forum: ",[20,55048,55049],{"href":55049,"rel":55050},"https://security.stackexchange.com/questions/232055/security-implications-of-using-public-subnets-in-aws-vpc-for-hosting-web-and-job",[24],"\nI'm curious to know if this is a reasonable tradeoff to make, as well as how secure the proposed solution is (using security groups and network ACLs).",[700,55053,55054,55060],{"start":500},[79,55055,55056,55057,55059],{},"I'm not focusing on SEO in this project. I'm using ",[30,55058,46303],{}," as the error document for the S3 website that CloudFront uses as the default behavior. This means that nested routes for the PWA have 404 response codes. I have heard about using lambda@edge to rewrite the the response code, I'm also curious about using this project in SSR mode (and also \"serverless-side rendering\"), but for now that is outside of the scope of what I want to do with this project.",[79,55061,55062,55065,55066,643],{},[15,55063,55064],{},"CDK is an awesome tool!"," Here is a great introduction by Nathan Peck, Senior Developer Advocate at Amazon Web Services: ",[20,55067,55068],{"href":55068,"rel":55069},"https://www.youtube.com/watch?v=184S7ki6fJA",[24],[56,55071,55073],{"id":55072},"my-experience-with-aws-cdk","My experience with AWS CDK",[11,55075,55076],{},"Having worked with CloudFormation for almost a year, I was not looking forward to learning a another Infrastructure as Code (IaC) tool. I struggled with CloudFormation and the whole concept of using an extended version of YAML to define infrastructure for multiple environments. I have limited experience with Terraform, but learning another DSL seemed like a lateral move that didn't seem worth it. I'm glad I did spend time learning and using CloudFormation, because CDK is an abstraction layer over CloudFormation. Here are some of my thoughts on adopting CDK.",[736,55078,55080,55081],{"id":55079},"start-here-httpscdkworkshopcom","Start here: ",[20,55082,55083],{"href":55083,"rel":55084},"https://cdkworkshop.com",[24],[11,55086,55087],{},"This is a great resource that was my first exposure to CDK. It focuses on using Lambda, DynamoDB and API Gateway, three technologies that I haven't used in production environments. In my mind this stack is the antithesis of the stack paradigm I am most experienced with: EC2, RDS and ELB. Regardless, I went ahead with it and was pretty much instantly hooked on CDK.",[736,55089,55091],{"id":55090},"the-stack-is-defined-by-a-small-number-of-inputs-used-for-namespacing","The stack is defined by a small number of inputs used for namespacing",[11,55093,55094,55097,55098,55100],{},[30,55095,55096],{},"awscdk/app.py"," is the entrypoint for ",[30,55099,30123],{}," commands. Here's what it contains:",[11,55102,55103,55104,55107,55108,55110],{},"Here are the main parameters for ",[30,55105,55106],{},"ApplicationStack",", which represents all of the resources for a specific environment environment. Each value is composed of environment variables set in GitLab CI when ",[30,55109,53910],{}," is called:",[76,55112,55113,55130,55136,55148,55163,55177],{},[79,55114,55115,55118,55119,106,55121,106,55124,106,55127,55129],{},[30,55116,55117],{},"environment_name"," - Possible examples could be ",[30,55120,10715],{},[30,55122,55123],{},"qa",[30,55125,55126],{},"some-feature-branch",[30,55128,26469],{},", etc.",[79,55131,55132,55135],{},[30,55133,55134],{},"base_domain_name"," - The domain name of the Hosted Zone that needs to be setup manually in Route53",[79,55137,55138,55141,55142,55145,55146],{},[30,55139,55140],{},"full_domain_name"," - ",[30,55143,55144],{},"f\"{environment_name}.{base_domain_name}\""," or optionally just ",[30,55147,55134],{},[79,55149,55150,55153,55154,55156,55157,55159,55160,55162],{},[30,55151,55152],{},"base_app_name"," - Based on the ",[30,55155,55134],{},", but ",[30,55158,643],{}," is replaced with ",[30,55161,12445],{}," in order to be used for naming certain AWS resources, used for resources tagging so we can easily look at the costs of all environments for our application with one tag.",[79,55164,55165,55168,55169,106,55171,55173,55174,55176],{},[30,55166,55167],{},"full_app_name"," - Base on ",[30,55170,55140],{},[30,55172,643],{}," replaced with ",[30,55175,12445],{}," as well.",[79,55178,55179],{},[30,55180,55181],{},"aws_region",[11,55183,55184,55185,208],{},"Expand to the following to view ",[30,55186,55096],{},[55188,55189,55190],"details",{},[459,55191,55193],{"className":13136,"code":55192,"language":12886,"meta":464,"style":464},"#!/usr/bin/env python3\nimport os\n\nfrom aws_cdk import core\n\nfrom awscdk.cdk_app_root import ApplicationStack\n\n# naming conventions, also used for ACM certs, DNS Records, resource naming\n# Dynamically generated resource names created in CDK are used in GitLab CI\n# such as cluster name, task definitions, etc.\nenvironment_name = f\"{os.environ.get('ENVIRONMENT', 'dev')}\"\nbase_domain_name = os.environ.get(\"DOMAIN_NAME\", \"mysite.com\")\n# if the the production environent subdomain should nott be included in the URL\n# redefine `full_domain_name` to `base_domain_name` for that environment\nfull_domain_name = f\"{environment_name}.{base_domain_name}\"  # dev.mysite.com\n# if environment_name == \"prod\":\n#     full_domain_name = base_domain_name\nbase_app_name = os.environ.get(\"APP_NAME\", \"mysite-com\")\nfull_app_name = f\"{environment_name}-{base_app_name}\"  # dev-mysite-com\naws_region = os.environ.get(\"AWS_DEFAULT_REGION\", \"us-east-1\")\n\n\napp = core.App()\nstack = ApplicationStack(\n    app,\n    f\"{full_app_name}-stack\",\n    environment_name=environment_name,\n    base_domain_name=base_domain_name,\n    full_domain_name=full_domain_name,\n    base_app_name=base_app_name,\n    full_app_name=full_app_name,\n    env={\"region\": aws_region},\n)\n\n# in order to be able to tag ECS resources, you need to go to\n# the ECS Console > Account Settings > Amazon ECS ARN and resource ID settings\n# and enable at least Service and Task. Optionally enable\n# CloudWatch Container Insights\nstack.node.apply_aspect(core.Tag(\"StackName\", full_app_name))\nstack.node.apply_aspect(core.Tag(\"StackName\", base_app_name))\n\napp.synth()\n",[30,55194,55195,55199,55205,55209,55219,55223,55235,55239,55244,55249,55254,55283,55302,55307,55312,55342,55347,55352,55371,55401,55417,55421,55425,55434,55444,55448,55465,55475,55485,55495,55505,55515,55528,55532,55536,55541,55546,55551,55556,55567,55576,55580],{"__ignoreMap":464},[151,55196,55197],{"class":469,"line":470},[151,55198,38352],{"class":1527},[151,55200,55201,55203],{"class":469,"line":488},[151,55202,16859],{"class":1869},[151,55204,24070],{"class":503},[151,55206,55207],{"class":469,"line":500},[151,55208,1090],{"emptyLinePlaceholder":609},[151,55210,55211,55213,55215,55217],{"class":469,"line":509},[151,55212,16853],{"class":1869},[151,55214,37698],{"class":503},[151,55216,16859],{"class":1869},[151,55218,38415],{"class":503},[151,55220,55221],{"class":469,"line":517},[151,55222,1090],{"emptyLinePlaceholder":609},[151,55224,55225,55227,55230,55232],{"class":469,"line":534},[151,55226,16853],{"class":1869},[151,55228,55229],{"class":503}," awscdk.cdk_app_root ",[151,55231,16859],{"class":1869},[151,55233,55234],{"class":503}," ApplicationStack\n",[151,55236,55237],{"class":469,"line":1413},[151,55238,1090],{"emptyLinePlaceholder":609},[151,55240,55241],{"class":469,"line":1418},[151,55242,55243],{"class":1527},"# naming conventions, also used for ACM certs, DNS Records, resource naming\n",[151,55245,55246],{"class":469,"line":2462},[151,55247,55248],{"class":1527},"# Dynamically generated resource names created in CDK are used in GitLab CI\n",[151,55250,55251],{"class":469,"line":2471},[151,55252,55253],{"class":1527},"# such as cluster name, task definitions, etc.\n",[151,55255,55256,55259,55261,55263,55265,55267,55269,55272,55274,55277,55279,55281],{"class":469,"line":2480},[151,55257,55258],{"class":503},"environment_name ",[151,55260,1876],{"class":1869},[151,55262,36853],{"class":12347},[151,55264,8592],{"class":481},[151,55266,5729],{"class":477},[151,55268,39030],{"class":503},[151,55270,55271],{"class":481},"'ENVIRONMENT'",[151,55273,106],{"class":503},[151,55275,55276],{"class":481},"'dev'",[151,55278,748],{"class":503},[151,55280,2001],{"class":477},[151,55282,16406],{"class":481},[151,55284,55285,55288,55290,55292,55295,55297,55300],{"class":469,"line":2489},[151,55286,55287],{"class":503},"base_domain_name ",[151,55289,1876],{"class":1869},[151,55291,36806],{"class":503},[151,55293,55294],{"class":481},"\"DOMAIN_NAME\"",[151,55296,106],{"class":503},[151,55298,55299],{"class":481},"\"mysite.com\"",[151,55301,3640],{"class":503},[151,55303,55304],{"class":469,"line":2497},[151,55305,55306],{"class":1527},"# if the the production environent subdomain should nott be included in the URL\n",[151,55308,55309],{"class":469,"line":3140},[151,55310,55311],{"class":1527},"# redefine `full_domain_name` to `base_domain_name` for that environment\n",[151,55313,55314,55317,55319,55321,55323,55325,55327,55329,55331,55333,55335,55337,55339],{"class":469,"line":3149},[151,55315,55316],{"class":503},"full_domain_name ",[151,55318,1876],{"class":1869},[151,55320,36853],{"class":12347},[151,55322,8592],{"class":481},[151,55324,5729],{"class":477},[151,55326,55117],{"class":503},[151,55328,2001],{"class":477},[151,55330,643],{"class":481},[151,55332,5729],{"class":477},[151,55334,55134],{"class":503},[151,55336,2001],{"class":477},[151,55338,8592],{"class":481},[151,55340,55341],{"class":1527},"  # dev.mysite.com\n",[151,55343,55344],{"class":469,"line":3158},[151,55345,55346],{"class":1527},"# if environment_name == \"prod\":\n",[151,55348,55349],{"class":469,"line":3167},[151,55350,55351],{"class":1527},"#     full_domain_name = base_domain_name\n",[151,55353,55354,55357,55359,55361,55364,55366,55369],{"class":469,"line":3175},[151,55355,55356],{"class":503},"base_app_name ",[151,55358,1876],{"class":1869},[151,55360,36806],{"class":503},[151,55362,55363],{"class":481},"\"APP_NAME\"",[151,55365,106],{"class":503},[151,55367,55368],{"class":481},"\"mysite-com\"",[151,55370,3640],{"class":503},[151,55372,55373,55376,55378,55380,55382,55384,55386,55388,55390,55392,55394,55396,55398],{"class":469,"line":3184},[151,55374,55375],{"class":503},"full_app_name ",[151,55377,1876],{"class":1869},[151,55379,36853],{"class":12347},[151,55381,8592],{"class":481},[151,55383,5729],{"class":477},[151,55385,55117],{"class":503},[151,55387,2001],{"class":477},[151,55389,12445],{"class":481},[151,55391,5729],{"class":477},[151,55393,55152],{"class":503},[151,55395,2001],{"class":477},[151,55397,8592],{"class":481},[151,55399,55400],{"class":1527},"  # dev-mysite-com\n",[151,55402,55403,55405,55407,55409,55411,55413,55415],{"class":469,"line":3193},[151,55404,38440],{"class":503},[151,55406,1876],{"class":1869},[151,55408,36806],{"class":503},[151,55410,38447],{"class":481},[151,55412,106],{"class":503},[151,55414,38452],{"class":481},[151,55416,3640],{"class":503},[151,55418,55419],{"class":469,"line":3720},[151,55420,1090],{"emptyLinePlaceholder":609},[151,55422,55423],{"class":469,"line":3729},[151,55424,1090],{"emptyLinePlaceholder":609},[151,55426,55427,55429,55431],{"class":469,"line":3735},[151,55428,38486],{"class":503},[151,55430,1876],{"class":1869},[151,55432,55433],{"class":503}," core.App()\n",[151,55435,55436,55439,55441],{"class":469,"line":3745},[151,55437,55438],{"class":503},"stack ",[151,55440,1876],{"class":1869},[151,55442,55443],{"class":503}," ApplicationStack(\n",[151,55445,55446],{"class":469,"line":3754},[151,55447,38501],{"class":503},[151,55449,55450,55452,55454,55456,55458,55460,55463],{"class":469,"line":3760},[151,55451,16382],{"class":12347},[151,55453,8592],{"class":481},[151,55455,5729],{"class":477},[151,55457,55167],{"class":503},[151,55459,2001],{"class":477},[151,55461,55462],{"class":481},"-stack\"",[151,55464,9417],{"class":503},[151,55466,55467,55470,55472],{"class":469,"line":3773},[151,55468,55469],{"class":15210},"    environment_name",[151,55471,1876],{"class":1869},[151,55473,55474],{"class":503},"environment_name,\n",[151,55476,55477,55480,55482],{"class":469,"line":3782},[151,55478,55479],{"class":15210},"    base_domain_name",[151,55481,1876],{"class":1869},[151,55483,55484],{"class":503},"base_domain_name,\n",[151,55486,55487,55490,55492],{"class":469,"line":3791},[151,55488,55489],{"class":15210},"    full_domain_name",[151,55491,1876],{"class":1869},[151,55493,55494],{"class":503},"full_domain_name,\n",[151,55496,55497,55500,55502],{"class":469,"line":3803},[151,55498,55499],{"class":15210},"    base_app_name",[151,55501,1876],{"class":1869},[151,55503,55504],{"class":503},"base_app_name,\n",[151,55506,55507,55510,55512],{"class":469,"line":3811},[151,55508,55509],{"class":15210},"    full_app_name",[151,55511,1876],{"class":1869},[151,55513,55514],{"class":503},"full_app_name,\n",[151,55516,55517,55519,55521,55523,55525],{"class":469,"line":3820},[151,55518,38513],{"class":15210},[151,55520,1876],{"class":1869},[151,55522,5729],{"class":503},[151,55524,38520],{"class":481},[151,55526,55527],{"class":503},": aws_region},\n",[151,55529,55530],{"class":469,"line":7084},[151,55531,3640],{"class":503},[151,55533,55534],{"class":469,"line":7148},[151,55535,1090],{"emptyLinePlaceholder":609},[151,55537,55538],{"class":469,"line":7211},[151,55539,55540],{"class":1527},"# in order to be able to tag ECS resources, you need to go to\n",[151,55542,55543],{"class":469,"line":7273},[151,55544,55545],{"class":1527},"# the ECS Console > Account Settings > Amazon ECS ARN and resource ID settings\n",[151,55547,55548],{"class":469,"line":7335},[151,55549,55550],{"class":1527},"# and enable at least Service and Task. Optionally enable\n",[151,55552,55553],{"class":469,"line":7398},[151,55554,55555],{"class":1527},"# CloudWatch Container Insights\n",[151,55557,55558,55561,55564],{"class":469,"line":7462},[151,55559,55560],{"class":503},"stack.node.apply_aspect(core.Tag(",[151,55562,55563],{"class":481},"\"StackName\"",[151,55565,55566],{"class":503},", full_app_name))\n",[151,55568,55569,55571,55573],{"class":469,"line":7467},[151,55570,55560],{"class":503},[151,55572,55563],{"class":481},[151,55574,55575],{"class":503},", base_app_name))\n",[151,55577,55578],{"class":469,"line":7532},[151,55579,1090],{"emptyLinePlaceholder":609},[151,55581,55582],{"class":469,"line":7537},[151,55583,38542],{"class":503},[736,55585,32889,55587,55589],{"id":55586},"using-cdk-synth-in-development",[30,55588,53906],{}," in development",[11,55591,55592,55593,55596,55597,55600,55601,55600,55604,55600,55607,55610,55611,55614,55615,55618,55619,55621,55622,10552],{},"When I first started using CDK, I defined a root stack containing multiple CDK constructs that each included groups of related CDK resources. For example, I had an ",[30,55594,55595],{},"RDSConstruct"," that contained a ",[30,55598,55599],{},"Secret",", a ",[30,55602,55603],{},"StringParameter",[30,55605,55606],{},"CfnSecurityGroup",[30,55608,55609],{},"CfnDBSubnetGroup"," and a ",[30,55612,55613],{},"CfnDBCluster",". Whenever I added a new section of CDK code, I would run ",[30,55616,55617],{},"cdk synth > stack.yml"," to generate a \"snapshot\" of my infrastructure in one file. ",[30,55620,50292],{}," was committed in my repo, and I could easily see how changes in CDK code changed the resulting CloudFormation YAML by repeating this ",[30,55623,53906],{},[736,55625,55627,55630],{"id":55626},"nestedstacks",[30,55628,55629],{},"NestedStack","s",[11,55632,55633,55635,55636,55638,55639,19977,55642,55645,55646,50911,55648,55651,55652,55655,55656,55658,55659,55661,55662,55665,55666,55668],{},[30,55634,50292],{}," grew larger and larger as I added more resources in my main \"master stack\" or \"skeleton stack\", and I realized that I was going to reach the hard limit of 200 resources per CloudFormation stack. Refactoring to use ",[30,55637,55629],{},"s was pretty straightforward, all I had to do was replace ",[30,55640,55641],{},"core.Construct",[30,55643,55644],{},"cloudformation.NestedStack",". With nested stacks, running ",[30,55647,55617],{},[30,55649,55650],{},"AWS::CloudFormation::Stack","s that are created, with ",[30,55653,55654],{},"TemplateURL"," and parameters from other stacks. It is more useful to look into contents of ",[30,55657,54049],{}," when working with ",[30,55660,55629],{},"s. The following command (executed from the root of the project) update the contents of the ",[30,55663,55664],{},"awscdk/cdk.out"," directory with templates for each of the ",[30,55667,55629],{},"s:",[459,55670,55673],{"className":55671,"code":55672,"language":997},[995],"cdk synth --app awscdk/app.py --output awscdk/cdk.out\n",[30,55674,55672],{"__ignoreMap":464},[11,55676,55677,55678,306,55680,55682,55683,55686],{},"The result is JSON, not YAML, and ",[30,55679,54049],{},[30,55681,12546],{},"d, but it can still be helpful in verifying that your CDK code is generating the correct CloudFormation templates. (",[30,55684,55685],{},"cdk diff"," may be the best option for seeing how CDK code changes will change your infrastructure).",[736,55688,55690,55692],{"id":55689},"cdk-deploy-and-cdk-in-gitlab-ci-pipelines",[30,55691,53910],{}," and CDK in GitLab CI pipelines",[11,55694,55695,55696,26792,55698,55701,55702,55705,55706,10552],{},"I like how ",[30,55697,53910],{},[30,55699,55700],{},"tail","s the output of CloudFormation events until the stack update finishes or fails. I previously had been using the AWS CLI to call ",[30,55703,55704],{},"aws cloudformation update-stack",". This command kicks off a stack update and the GitLab pipeline will succeed regardless of the success of failure of the ",[30,55707,55708],{},"update-stack",[210,55710,55711],{},[11,55712,55713,55714,55716,55717,55719],{},"With CDK, the CI pipeline that calls ",[30,55715,53910],{}," fails if ",[30,55718,53910],{}," results in a stack rollback.",[11,55721,55722,55723,55725,55726,55728],{},"I couldn't find any examples of how to run ",[30,55724,53910],{}," in a GitLab CI pipeline (using the Python version of CDK, or any version of CDK). It would be nice if there was an official image maintained for the different languages that are ready to run CDK deploy. Here's how I got ",[30,55727,53910],{}," working correctly in my CI/CD pipline:",[459,55730,55732],{"className":14359,"code":55731,"language":14361,"meta":464,"style":464},"cdk deploy:\n  image: docker:19.03.1\n  services:\n    - docker:19.03.5-dind\n  stage: deploy\n  only:\n    - master\n  variables:\n    ENVIRONMENT: dev\n    DOCKER_TLS_CERTDIR: ''\n  before_script:\n    - apk add nodejs-current npm\n    - npm i -g aws-cdk\n    - apk add --no-cache python3\n    - pip3 install -e awscdk\n  script:\n    - cdk bootstrap --app awscdk/app.py aws://$AWS_ACCOUNT_ID/$AWS_DEFAULT_REGION\n    - cdk deploy --app awscdk/app.py --require-approval never\n",[30,55733,55734,55740,55748,55754,55760,55768,55775,55781,55787,55797,55807,55813,55820,55826,55833,55839,55845,55851],{"__ignoreMap":464},[151,55735,55736,55738],{"class":469,"line":470},[151,55737,53910],{"class":14368},[151,55739,14372],{"class":503},[151,55741,55742,55744,55746],{"class":469,"line":488},[151,55743,22226],{"class":14368},[151,55745,6208],{"class":503},[151,55747,49598],{"class":481},[151,55749,55750,55752],{"class":469,"line":500},[151,55751,49603],{"class":14368},[151,55753,14372],{"class":503},[151,55755,55756,55758],{"class":469,"line":509},[151,55757,29541],{"class":503},[151,55759,49612],{"class":481},[151,55761,55762,55764,55766],{"class":469,"line":517},[151,55763,38176],{"class":14368},[151,55765,6208],{"class":503},[151,55767,20676],{"class":481},[151,55769,55770,55773],{"class":469,"line":534},[151,55771,55772],{"class":14368},"  only",[151,55774,14372],{"class":503},[151,55776,55777,55779],{"class":469,"line":1413},[151,55778,29541],{"class":503},[151,55780,20452],{"class":481},[151,55782,55783,55785],{"class":469,"line":1418},[151,55784,50067],{"class":14368},[151,55786,14372],{"class":503},[151,55788,55789,55792,55794],{"class":469,"line":2462},[151,55790,55791],{"class":14368},"    ENVIRONMENT",[151,55793,6208],{"class":503},[151,55795,55796],{"class":481},"dev\n",[151,55798,55799,55802,55804],{"class":469,"line":2471},[151,55800,55801],{"class":14368},"    DOCKER_TLS_CERTDIR",[151,55803,6208],{"class":503},[151,55805,55806],{"class":481},"''\n",[151,55808,55809,55811],{"class":469,"line":2480},[151,55810,38213],{"class":14368},[151,55812,14372],{"class":503},[151,55814,55815,55817],{"class":469,"line":2489},[151,55816,29541],{"class":503},[151,55818,55819],{"class":481},"apk add nodejs-current npm\n",[151,55821,55822,55824],{"class":469,"line":2497},[151,55823,29541],{"class":503},[151,55825,37580],{"class":481},[151,55827,55828,55830],{"class":469,"line":3140},[151,55829,29541],{"class":503},[151,55831,55832],{"class":481},"apk add --no-cache python3\n",[151,55834,55835,55837],{"class":469,"line":3149},[151,55836,29541],{"class":503},[151,55838,54173],{"class":481},[151,55840,55841,55843],{"class":469,"line":3158},[151,55842,38240],{"class":14368},[151,55844,14372],{"class":503},[151,55846,55847,55849],{"class":469,"line":3167},[151,55848,29541],{"class":503},[151,55850,54186],{"class":481},[151,55852,55853,55855],{"class":469,"line":3175},[151,55854,29541],{"class":503},[151,55856,54193],{"class":481},[11,55858,55859,55860,55862,55863,55865,55866,55869],{},"I have all of my CDK code in a top level directory called ",[30,55861,53872],{},", my Django code is in the top level ",[30,55864,26811],{}," directory and my frontend code is in the top level ",[30,55867,55868],{},"quasar"," directory. This directory structure is important, I'll come back to it shortly. Here are some things to note about this GitLab CI job definition:",[76,55871,55872,55895,55907],{},[79,55873,55874,55875,55877,55878,55881,55882,55884,55885,55888,55889,22326,55892,55894],{},"It uses the base ",[30,55876,30129],{}," image with ",[30,55879,55880],{},"docker:dind"," as a dependent service. This is necessary only if you hare using CDK constructs that build docker images and make use of ",[30,55883,53955],{},", such as the ",[30,55886,55887],{},"aws_ecs.AssetImage"," that I use to define ",[30,55890,55891],{},"self.image",[30,55893,55106],{}," class that defines that main stack of my application.",[79,55896,55897,55898,55900,55901,55903,55904,55906],{},"It requires node, npm python and pip, so these all need to be installed via ",[30,55899,50186],{},", the package manager for Alpine Linux. These are done in the ",[30,55902,49813],{},", the setup for the main \"script\" where ",[30,55905,53910],{}," is called.",[79,55908,55909,55910,55912,55913,55915,55916,55918],{},"The main ",[30,55911,19822],{}," section calls ",[30,55914,53955],{},". You only need to call ",[30,55917,53955],{}," once to initialize resources that CDK will use, so placing it here in my CI script is more of a reminder that we are using CDK assets (S3 buckets and ECR images); calling it again once it has been called initially on your AWS account does nothing, and you will see a message that communicates this:",[459,55920,55923],{"className":55921,"code":55922,"language":997},[995]," $ cdk bootstrap --app awscdk/app.py aws://$AWS_ACCOUNT_ID/$AWS_DEFAULT_REGION\n  ⏳  Bootstrapping environment aws://XXXXXXXXXXXX/us-east-1...\n  ✅  Environment aws://XXXXXXXXXXXX/us-east-1 bootstrapped (no changes).\n",[30,55924,55922],{"__ignoreMap":464},[736,55926,55928],{"id":55927},"cdk-bootstap",[30,55929,55930],{},"cdk bootstap",[11,55932,55933,55935],{},[30,55934,53955],{}," is one of my favorite features of CDK. As I mentioned earlier, it is used with S3 and ECR.",[54716,55937,55939],{"id":55938},"s3-assets","S3 Assets",[11,55941,55942,55943,208],{},"With S3, it allows you to populate the contents of an S3 bucket. Here's an example from ",[30,55944,55106],{},[459,55946,55948],{"className":13136,"code":55947,"language":12886,"meta":464,"style":464},"        if os.path.isdir(\"./quasar/dist/pwa\"):\n            s3_deployment.BucketDeployment(\n                self,\n                \"BucketDeployment\",\n                destination_bucket=self.static_site_bucket,\n                sources=[s3_deployment.Source.asset(\"./quasar/dist/pwa\")],\n                distribution=self.cloudfront.distribution,\n            )\n",[30,55949,55950,55962,55967,55973,55980,55992,56007,56019],{"__ignoreMap":464},[151,55951,55952,55954,55957,55960],{"class":469,"line":470},[151,55953,23357],{"class":1869},[151,55955,55956],{"class":503}," os.path.isdir(",[151,55958,55959],{"class":481},"\"./quasar/dist/pwa\"",[151,55961,15264],{"class":503},[151,55963,55964],{"class":469,"line":488},[151,55965,55966],{"class":503},"            s3_deployment.BucketDeployment(\n",[151,55968,55969,55971],{"class":469,"line":500},[151,55970,39867],{"class":15289},[151,55972,9417],{"class":503},[151,55974,55975,55978],{"class":469,"line":509},[151,55976,55977],{"class":481},"                \"BucketDeployment\"",[151,55979,9417],{"class":503},[151,55981,55982,55985,55987,55989],{"class":469,"line":517},[151,55983,55984],{"class":15210},"                destination_bucket",[151,55986,1876],{"class":1869},[151,55988,15277],{"class":15289},[151,55990,55991],{"class":503},".static_site_bucket,\n",[151,55993,55994,55997,55999,56002,56004],{"class":469,"line":534},[151,55995,55996],{"class":15210},"                sources",[151,55998,1876],{"class":1869},[151,56000,56001],{"class":503},"[s3_deployment.Source.asset(",[151,56003,55959],{"class":481},[151,56005,56006],{"class":503},")],\n",[151,56008,56009,56012,56014,56016],{"class":469,"line":1413},[151,56010,56011],{"class":15210},"                distribution",[151,56013,1876],{"class":1869},[151,56015,15277],{"class":15289},[151,56017,56018],{"class":503},".cloudfront.distribution,\n",[151,56020,56021],{"class":469,"line":1418},[151,56022,15381],{"class":503},[11,56024,56025,56026,56029,56030,56033,56034,56037,56038,56040],{},"If the directory ",[30,56027,56028],{},"./quasar/dist/pwa"," exists, CDK will upload the contents of that directory to the ",[30,56031,56032],{},"static_site_bucket"," and then invalidate the cache for ",[30,56035,56036],{},"self.cloudfront.distribution",", all in one shot. ",[30,56039,56028],{}," is the directory where assets from my frontend PWA are placed when compiled. GitLab CI passes these files between the jobs using artifacts:",[459,56042,56044],{"className":14359,"code":56043,"language":14361,"meta":464,"style":464},"artifacts:\n  paths:\n    - quasar/dist/pwa\n",[30,56045,56046,56052,56059],{"__ignoreMap":464},[151,56047,56048,56050],{"class":469,"line":470},[151,56049,40274],{"class":14368},[151,56051,14372],{"class":503},[151,56053,56054,56057],{"class":469,"line":488},[151,56055,56056],{"class":14368},"  paths",[151,56058,14372],{"class":503},[151,56060,56061,56063],{"class":469,"line":500},[151,56062,29541],{"class":503},[151,56064,56065],{"class":481},"quasar/dist/pwa\n",[11,56067,56068,56069,56071,56072,56075],{},"The job to build PWA assets can be set to only run when there are changes in the ",[30,56070,55868],{}," directory, so the ",[30,56073,56074],{},".quasar/dist/pwa"," directory will only exist if the job is executed. This helps speed up our CI/CD pipeline.",[54716,56077,56079],{"id":56078},"ecr-assets","ECR Assets",[11,56081,56082,56083,56086,56087,56089],{},"I use ",[30,56084,56085],{},"cdk boostrap"," and CDK assets in a similar way for building and pushing my application container. In ",[30,56088,55106],{},", I define the following:",[459,56091,56093],{"className":13136,"code":56092,"language":12886,"meta":464,"style":464},"        self.image = ecs.AssetImage(\n            \"./backend\", file=\"scripts/prod/Dockerfile\", target=\"production\",\n        )\n",[30,56094,56095,56107,56134],{"__ignoreMap":464},[151,56096,56097,56099,56102,56104],{"class":469,"line":470},[151,56098,37901],{"class":15289},[151,56100,56101],{"class":503},".image ",[151,56103,1876],{"class":1869},[151,56105,56106],{"class":503}," ecs.AssetImage(\n",[151,56108,56109,56112,56114,56117,56119,56122,56124,56127,56129,56132],{"class":469,"line":488},[151,56110,56111],{"class":481},"            \"./backend\"",[151,56113,106],{"class":503},[151,56115,56116],{"class":15210},"file",[151,56118,1876],{"class":1869},[151,56120,56121],{"class":481},"\"scripts/prod/Dockerfile\"",[151,56123,106],{"class":503},[151,56125,56126],{"class":15210},"target",[151,56128,1876],{"class":1869},[151,56130,56131],{"class":481},"\"production\"",[151,56133,9417],{"class":503},[151,56135,56136],{"class":469,"line":500},[151,56137,16824],{"class":503},[11,56139,56140,56141,56143,56144,106,56147,106,56150,56152,56153,35642],{},"This image is then referenced by multiple ",[30,56142,55629],{},"s that define Fargate services and tasks (",[30,56145,56146],{},"gunicorn",[30,56148,56149],{},"daphne",[30,56151,27556],{}," workers, etc.) This keeps our CDK code DRY.. Don't Repeat Yourself! Similarly, we aren't rebuilding, pushing and pulling the backend container when there are no changes to the code in ",[30,56154,26811],{},[11,56156,56157,56158,56161,56162,643],{},"I'm not setting up an ECR image repository for my application, but I believe there is a way to do this. One question that I have about using ",[30,56159,56160],{},"ecs.AssetImage"," is about image lifecycle management. I know that you can implement rules about how many images you want to keep in an ECR image repository, but ",[15,56163,56164],{},"I'm not sure how this works with CDK Image Assets",[736,56166,56168,56169],{"id":56167},"quick-tour-of-applicationstack","Quick tour of ",[30,56170,55106],{},[11,56172,56173,56174,56176],{},"Here's a very quick look at the structure of my CDK code, focusing on the ",[30,56175,55106],{},", the \"master stack\" or \"skeleton stack\" that contains.",[54716,56178,56180],{"id":56179},"hosted_zone",[30,56181,56179],{},[11,56183,56184,56185,187,56187,56189],{},"We get the hosted zone using the ",[30,56186,33097],{},[30,56188,53821],{},". This is not a nested stack.",[54716,56191,56193],{"id":56192},"site_certificate",[30,56194,56192],{},[11,56196,56197,56198,56200],{},"The ACM Certificate that will be used for the given environment. This references the ",[30,56199,55140],{}," (environment + application).",[54716,56202,56204],{"id":56203},"vpc_stack",[30,56205,56203],{},[11,56207,14155,56208,56210,56211,56214,56215,187,56217,56220,56221,643],{},[30,56209,55629],{}," for defining VPC resources. This construct generates lots of CloudFormation resources. I currently have ",[30,56212,56213],{},"nat_gateways"," set to zero, and I'm ",[30,56216,52908],{},[30,56218,56219],{},"PRIVATE"," subnets spread over 2 AZs. As I mentioned earlier, this is primarily for cost considerations and it is a best practice to use the tiered security model and run our Fargate tasks in private subnets instead of public subnets. I think I need to add NACL resources in this ",[30,56222,55629],{},[54716,56224,56226],{"id":56225},"alb_stack",[30,56227,56225],{},[11,56229,56230,56231,187,56234,56237,56238,56240,56241,643],{},"This defines the load balancer, configures that will send traffic to our Fargate services (such as our Django API). I was a little bit unclear about needing a ",[30,56232,56233],{},"listener",[30,56235,56236],{},"https_listener",". I might be able to get away with removing the ",[30,56239,56233],{}," and only using ",[30,56242,56236],{},[54716,56244,56246],{"id":56245},"static_site_stack",[30,56247,56245],{},[11,56249,56250],{},"This stack defines the S3 bucket and policies that will be used for hosting our static site (Quasar PWA).",[54716,56252,56254],{"id":56253},"backend_assets",[30,56255,56253],{},[11,56257,56258],{},"This stack defines the bucket and policies for managing the bucket that holds static and media assets for Django.",[54716,56260,56262],{"id":56261},"cloudfront",[30,56263,56261],{},[11,56265,56266,56267,56270],{},"This defines the CloudFront distribution that ties together several different parts of the application. It is the \"front desk\" of the application, and acts as a CDN and proxy. There is a separate CloudFront distribution for each environment (dev, staging, production). This stack also defines the Route53 ",[30,56268,56269],{},"ARecord"," that will be used to send traffic to a specific subdomain to the correct CloudFront distribution.",[11,56272,56273,56274,56277],{},"There are three ",[30,56275,56276],{},"origin_configs"," for each distribution:",[700,56279,56280,56286,56291],{},[79,56281,56282,56285],{},[30,56283,56284],{},"CustomOriginConfig"," for the ALB",[79,56287,56288,56290],{},[30,56289,56284],{}," for the S3 bucket website",[79,56292,56293,56296],{},[30,56294,56295],{},"S3OriginConfig"," for the Django static assets",[11,56298,56299,56300,56302,56303,56306],{},"Note that these ",[30,56301,56276],{}," each have different ",[30,56304,56305],{},"behaviors",", and that list comprehension is used to keep this code DRY.",[54716,56308,56310],{"id":56309},"bucketdeployment",[30,56311,56312],{},"BucketDeployment",[11,56314,56315,56316,56318],{},"This will deploy our static site assets to the S3 bucket defined in ",[30,56317,56245],{}," if the static site assets are present at the time of deployment. If they are not present, this means that there were no changes made to the frontend site.",[54716,56320,56321],{"id":30126},[30,56322,30126],{},[11,56324,56325],{},"Defines the ECS Cluster.",[54716,56327,56328],{"id":26801},[30,56329,26801],{},[11,56331,56332,56333,56336,56337,56339,56340,29198,56343,26792,56345,643],{},"There is no L2 construct for ",[30,56334,56335],{},"DBCluster",", so I used ",[30,56338,55613],{}," in order to use the Aurora Postgres ",[30,56341,56342],{},"engine",[30,56344,40767],{},[30,56346,56347],{},"engine_mode",[54716,56349,56351],{"id":56350},"elasticache",[30,56352,56350],{},[11,56354,56355],{},"I also had to use L1 constructs for ElastiCache, but this one is pretty straightforward.",[11,56357,56358,56359,56362,56363,56366],{},"For both RDS and ElastiCache I used the ",[30,56360,56361],{},"vpc_default_security_group"," as the ",[30,56364,56365],{},"source_security_group",". It might be a better idea to define another security group altogether, but this approach works.",[54716,56368,56370],{"id":56369},"assetimage",[30,56371,56372],{},"AssetImage",[11,56374,56375,56376,56378],{},"The docker image that references Django application code in the ",[30,56377,26811],{}," directory. This image is referenced in Fargate services and tasks.",[54716,56380,56382],{"id":56381},"variables",[30,56383,56381],{},[11,56385,56386],{},"This section defines and organizes all of the environment variables and secrets for my application.",[54716,56388,56390],{"id":56389},"backend_service",[30,56391,56389],{},[11,56393,56394,56395,56398],{},"It might be a better idea to replace this with ",[30,56396,56397],{},"NetworkLoadBalancedFargateService",", but instead I implemented this with lower-level constructs just to be clear about what I'm doing. To add a load balanced service, here is what I did:",[700,56400,56401,56404,56410,56417,56420,56423,56431],{},[79,56402,56403],{},"Define the Fargate task",[79,56405,56406,56407,30455],{},"Add the container to this task with other information (secrets, logging, ",[30,56408,56409],{},"command",[79,56411,56412,56413,56416],{},"Give the task role permissions it needs such as access to Secrets, S3 permissions. (It might be a good idea to refactor this into a function that can be called on ",[30,56414,56415],{},"task_role",", but for now I am explicitly granting all permissions)",[79,56418,56419],{},"Create and add a port mapping",[79,56421,56422],{},"Define an ECS Fargate Service that reference the previously defined Fargate task, configure security group",[79,56424,56425,56426,56428,56429,643],{},"Add the service as a target to the ",[30,56427,56236],{}," defined previously in ",[30,56430,56225],{},[79,56432,56433],{},"Optionally configure autoscaling for the Fargate service",[54716,56435,56437],{"id":56436},"flower_service",[30,56438,56436],{},[11,56440,56441,56442],{},"Flower is a monitoring utility for Celery. I had trouble getting this to work correctly, but I managed to make it work by adding a simple nginx container that passes traffic to the flower container running in the same task. ",[20,56443,56444],{"href":56444,"rel":56445},"https://flower.readthedocs.io/en/latest/reverse-proxy.html",[24],[54716,56447,56449],{"id":56448},"celery_default_service",[30,56450,56448],{},[11,56452,56453],{},"This stack defines the default celery queue. This is discussed later in more detail, but the basic idea is to:",[700,56455,56456,56458,56461,56464,56467],{},[79,56457,56403],{},[79,56459,56460],{},"Add the container",[79,56462,56463],{},"Define the Fargate service",[79,56465,56466],{},"Grant permissions",[79,56468,56469],{},"Configure autoscaling",[54716,56471,56473],{"id":56472},"celery_autoscaling",[30,56474,56472],{},[11,56476,56477],{},"This stack defines the Lambda function and schedule on which this Lambda is called. This stack is discussed in more detail later on.",[54716,56479,56481],{"id":56480},"backend_tasks",[30,56482,56480],{},[11,56484,56485,56486,106,56488,187,56490,643],{},"These are administrative tasks that are executed by running manual GitLab CI jobs such as ",[30,56487,27589],{},[30,56489,27586],{},[30,56491,51591],{},[56,56493,56495,56496,56499,56500,10727],{"id":56494},"why-x-why-not-y","Why ",[30,56497,56498],{},"X","? Why not ",[30,56501,56502],{},"Y",[11,56504,56505],{},"This section will compare some of the technology choices I have made in this project to other popular alternatives.",[736,56507,56509],{"id":56508},"why-ecs-why-not-kubernetes","Why ECS? Why not Kubernetes?",[11,56511,56512,56513,56516],{},"I like Kubernetes. I have never used it to support production workloads, but I have explored it in a limited capacity. I have set up this project in Kubernetes locally using ",[30,56514,56515],{},"minikube",", there is an article on this in the documentation. There are also lots of options for how you do Kubernetes, here are a few off of the top of my head:",[76,56518,56519,56522,56525,56528,56531,56534],{},[79,56520,56521],{},"KOPS",[79,56523,56524],{},"EKS",[79,56526,56527],{},"k3s",[79,56529,56530],{},"cdk8s",[79,56532,56533],{},"Kubernetes on Fargate",[79,56535,56536],{},"\"Kuberetes the Hard Way\"",[11,56538,56539],{},"With any of these options you are probably going to want to use Helm to do deployments, which adds another layer of abstraction that also has several different ways to be managed. On the other hand, ECS is \"just ECS\"; there are not a lot of other considerations to make when running workloads in ECS. You have to choose between the two available launch types: EC2 and Fargate. Comparisons of ECS and Kubernetes often mention that ECS integrates nicely with other AWS Services, something I have generally found to be true in setting up this project. Granting permissions to S3 buckets or CloudWatch, or using security groups to give ECS tasks access to certain resources in your VPC are some examples of what this \"tight integration\" has meant for me so far.",[736,56541,56543],{"id":56542},"why-django-why-not-flask","Why Django? Why not Flask?",[11,56545,56546],{},"I like Django for lots of reasons. I get why people say it can be \"overkill\", and there are definitely lots of parts of Django that don't use. I use Django primarily for the ORM, migrations system and the Django Admin. I also use the Django REST Framework, which gives me another big productivity boost when building APIs. I dislike Django Forms and Django templates, but that wasn't always the case. Before that, I disliked JavaScript frameworks and single page applications. That changes when I started working with Vue.js and Quasar.",[736,56548,56550],{"id":56549},"why-quasar-framework-why-not-nuxtjs-or-vanilla-vuejs-why-not-react","Why Quasar Framework? Why not Nuxt.js or vanilla Vue.js? Why not React?",[11,56552,56553],{},"Quasar Framework is a few different things, and just like Vue.js itself, Quasar can be incrementally adopted. Primarily, Quasar is a CLI for creating Vue.js projects. It offers some opinions on how to organize files and folders. It handles SPA, PWA, SSR, Electron, Cordova and other build targets. It implements the MaterialUI spec and it has an awesome and active community (but Django, GitLab and AWS do, too!)",[11,56555,56556,56557,56560,56561,56563],{},"Quasar does things a little bit differently than vanilla Vue.js. There is no ",[30,56558,56559],{},"main.js"," file in a typical Quasar application. Instead, bootfiles are used to initialize things that would typically go into ",[30,56562,56559],{},". I believe this helps Quasar manage multiple build targets easily.",[11,56565,56566],{},"I really haven't worked a lot with Nuxt.js, but I would probably be drawn to it if Quasar was not an option. I like how it helps structure your application. In the same way that Django is \"batteries included\", Quasar is also very much \"batteries included\".",[11,56568,56569,56570,187,56572,13576],{},"I think React is neat, but I have similar feelings between React and Vue that I have between Django and Flask. One requires you make more decisions and therefore has a heavy mental load. The biggest example of this is the tight integration of an official router and state management system for Vue (",[30,56571,54826],{},[30,56573,56574],{},"vuex",[736,56576,56578],{"id":56577},"why-celery-why-not-heuy-or-django-rq","Why Celery? Why not Heuy or django-rq?",[11,56580,56581],{},"I think celery is probably the most heavy-weight option for managing asynchronous tasks in Django. It is very flexible and \"pluggable\" which makes it slightly more challenging to get setup. I would be interested in trying another option, but celery is a mature option that has a large community of users.",[56,56583,56585],{"id":56584},"scaling-celery-workers-to-zero","Scaling Celery workers to zero",[11,56587,56588],{},"Django is used in a few different ways in this application:",[76,56590,56591,56594,56597,56600],{},[79,56592,56593],{},"a backend API supported by Django REST Framework, Postgres and static files stored in AWS S3, served over CloudFront",[79,56595,56596],{},"a websocket server supported by Django Channels",[79,56598,56599],{},"an administrative backend that is automatically generated by Django (Django Admin)",[79,56601,56602],{},"asynchronous task workers supported by Celery",[11,56604,56605,56606,56608,56609,106,56611,187,56614,56616],{},"The API server, websocket server and Django admin can technically be served by the same ",[30,56607,56149],{}," process. In terms of our application and AWS architecture, this means that requests for URLs starting with ",[30,56610,47457],{},[30,56612,56613],{},"/ws/",[30,56615,47266],{}," can all be sent to the same Fargate service target group.",[11,56618,56619],{},"Alternatively, we could split these up into two or three different process that can then be scaled individually.",[11,56621,56622,56623,643],{},"Celery processes should be run as separate processes. If you have multiple queues, you may have workers that are dedicated to processing tasks from certain queues which run as individual Fargate tasks, each with certain CPU and memory allocations and other celery settings, such as ",[30,56624,56625],{},"max_concurrency",[11,56627,56628,56629,56632],{},"To manage the total cost of ownership, I wanted to know how to scale Celery workers between 0 and ",[30,56630,56631],{},"N",". A celery worker is typically an \"always on\" process that watches for new messages that arrive in the queue and then processes message specified (the messages delivered to the queue contain information on which function to call, and what arguments that function should be called with.",[11,56634,56635],{},"Let's image that we have a celery task to process with the following properties:",[76,56637,56638,56641,56644,56647,56650,56653,56656],{},[79,56639,56640],{},"It takes a long time to process (between 15 minutes and 1 or 2 hours)",[79,56642,56643],{},"It has lots of dependencies (such as pandas, scikit-learn, etc.)",[79,56645,56646],{},"It requires a high amount of CPU and memory",[79,56648,56649],{},"It cannot easily be broken down into smaller sub-tasks",[79,56651,56652],{},"The time and frequency at which this task will be called is not predictable",[79,56654,56655],{},"A 2 - 3 minute delay between calling the task and starting work on the task is acceptable",[79,56657,56658],{},"This task might not be very easy to process with AWS Lambda without lots of additional logic, if not impossible.",[11,56660,56661],{},"To manage our project's TCO, we want scale down the number of Fargate tasks that process the queue this task is sent to. If there are no messages queued for this worker and no messages currently being processed by any of the workers for the queue, the number of Fargate tasks (celery workers) should be scaled down.",[11,56663,56664],{},"I haven't done much with autoscaling on AWS before, but I found that CDK provides some very nice abstractions that make scaling with Fargate task very straightforward.",[11,56666,56667,56668,56670],{},"ECS allows you to scale based on some built in metrics, such as CPU Utilization. Scaling between 0 and ",[30,56669,56631],{}," workers based on CPU utilization metrics wouldn't work because there would be no CPU utilization after the Fargate tasks are scaled to zero; messages in the queue would remain unprocessed and no new workers would be brought online.",[11,56672,56673],{},"Using the number of tasks in the queue would be a better option. I tried a few different options to get this to work.",[11,56675,56676,56677,21420,56680,56683],{},"First, I tried using one of the high-level constructs (L3 construct) from ",[30,56678,56679],{},"ecs_patterns",[30,56681,56682],{},"QueueProcessingFargateService",", but this would require that I replace Redis with SQS as the broker to be used with Celery. Some people prefer to use SQS, but I like having the ability to inspect and control, which requires a broker like Redis and is not possible with SQS.",[11,56685,56686],{},"My current solution involves:",[700,56688,56689,56699,56706,56714,56723],{},[79,56690,56691,56692,56695,56696,748],{},"Creating a CloudWatch metric (the namespace is ",[30,56693,56694],{},"FULL_APP_NAME"," and the metric name is the name of the queue, ",[30,56697,56698],{},"default",[79,56700,56701,56702,56705],{},"Calling ",[30,56703,56704],{},"auto_scale_task_count"," on the Fargate service to create a an \"autoscaling Fargate task\"",[79,56707,56701,56708,56711,56712],{},[30,56709,56710],{},"scale_on_metric"," on the \"autoscaling Fargate task\" with the CloudWatch metric created in ",[30,56713,6760],{},[79,56715,49267,56716,56719,56720,56722],{},[30,56717,56718],{},"celery-metrics"," API endpoint on my Django API server that, when ",[30,56721,36573],{},"ed to, collects celery metrics per queue and publishes these metrics to CloudWatch.",[79,56724,56725,56726,56728],{},"Scheduling a Lambda function to post to the ",[30,56727,56718],{}," endpoint every 5 minutes.",[11,56730,56731],{},"Here's the code that takes care of steps 1, 2 and 3:",[459,56733,56735],{"className":13136,"code":56734,"language":12886,"meta":464,"style":464},"        self.default_celery_queue_cw_metric = cw.Metric(\n            namespace=scope.full_app_name, metric_name=\"default\"\n        )\n\n        self.celery_default_queue_asg = self.celery_default_worker_service.auto_scale_task_count(\n            min_capacity=0, max_capacity=2\n        )\n\n        self.celery_default_queue_asg.scale_on_metric(\n            \"CeleryDefaultQueueAutoscaling\",\n            metric=self.default_celery_queue_cw_metric,\n            scaling_steps=[\n                aas.ScalingInterval(change=-1, lower=0),\n                aas.ScalingInterval(change=1, lower=1),\n            ],\n            adjustment_type=aas.AdjustmentType.CHANGE_IN_CAPACITY,\n        )\n",[30,56736,56737,56749,56767,56771,56775,56789,56808,56812,56816,56823,56830,56842,56851,56875,56895,56899,56914],{"__ignoreMap":464},[151,56738,56739,56741,56744,56746],{"class":469,"line":470},[151,56740,37901],{"class":15289},[151,56742,56743],{"class":503},".default_celery_queue_cw_metric ",[151,56745,1876],{"class":1869},[151,56747,56748],{"class":503}," cw.Metric(\n",[151,56750,56751,56754,56756,56759,56762,56764],{"class":469,"line":488},[151,56752,56753],{"class":15210},"            namespace",[151,56755,1876],{"class":1869},[151,56757,56758],{"class":503},"scope.full_app_name, ",[151,56760,56761],{"class":15210},"metric_name",[151,56763,1876],{"class":1869},[151,56765,56766],{"class":481},"\"default\"\n",[151,56768,56769],{"class":469,"line":500},[151,56770,16824],{"class":503},[151,56772,56773],{"class":469,"line":509},[151,56774,1090],{"emptyLinePlaceholder":609},[151,56776,56777,56779,56782,56784,56786],{"class":469,"line":517},[151,56778,37901],{"class":15289},[151,56780,56781],{"class":503},".celery_default_queue_asg ",[151,56783,1876],{"class":1869},[151,56785,15451],{"class":15289},[151,56787,56788],{"class":503},".celery_default_worker_service.auto_scale_task_count(\n",[151,56790,56791,56794,56796,56798,56800,56803,56805],{"class":469,"line":534},[151,56792,56793],{"class":15210},"            min_capacity",[151,56795,1876],{"class":1869},[151,56797,9181],{"class":477},[151,56799,106],{"class":503},[151,56801,56802],{"class":15210},"max_capacity",[151,56804,1876],{"class":1869},[151,56806,56807],{"class":477},"2\n",[151,56809,56810],{"class":469,"line":1413},[151,56811,16824],{"class":503},[151,56813,56814],{"class":469,"line":1418},[151,56815,1090],{"emptyLinePlaceholder":609},[151,56817,56818,56820],{"class":469,"line":2462},[151,56819,37901],{"class":15289},[151,56821,56822],{"class":503},".celery_default_queue_asg.scale_on_metric(\n",[151,56824,56825,56828],{"class":469,"line":2471},[151,56826,56827],{"class":481},"            \"CeleryDefaultQueueAutoscaling\"",[151,56829,9417],{"class":503},[151,56831,56832,56835,56837,56839],{"class":469,"line":2480},[151,56833,56834],{"class":15210},"            metric",[151,56836,1876],{"class":1869},[151,56838,15277],{"class":15289},[151,56840,56841],{"class":503},".default_celery_queue_cw_metric,\n",[151,56843,56844,56847,56849],{"class":469,"line":2489},[151,56845,56846],{"class":15210},"            scaling_steps",[151,56848,1876],{"class":1869},[151,56850,37620],{"class":503},[151,56852,56853,56856,56859,56862,56864,56866,56869,56871,56873],{"class":469,"line":2497},[151,56854,56855],{"class":503},"                aas.ScalingInterval(",[151,56857,56858],{"class":15210},"change",[151,56860,56861],{"class":1869},"=-",[151,56863,6760],{"class":477},[151,56865,106],{"class":503},[151,56867,56868],{"class":15210},"lower",[151,56870,1876],{"class":1869},[151,56872,9181],{"class":477},[151,56874,37985],{"class":503},[151,56876,56877,56879,56881,56883,56885,56887,56889,56891,56893],{"class":469,"line":3140},[151,56878,56855],{"class":503},[151,56880,56858],{"class":15210},[151,56882,1876],{"class":1869},[151,56884,6760],{"class":477},[151,56886,106],{"class":503},[151,56888,56868],{"class":15210},[151,56890,1876],{"class":1869},[151,56892,6760],{"class":477},[151,56894,37985],{"class":503},[151,56896,56897],{"class":469,"line":3149},[151,56898,52319],{"class":503},[151,56900,56901,56904,56906,56909,56912],{"class":469,"line":3158},[151,56902,56903],{"class":15210},"            adjustment_type",[151,56905,1876],{"class":1869},[151,56907,56908],{"class":503},"aas.AdjustmentType.",[151,56910,56911],{"class":477},"CHANGE_IN_CAPACITY",[151,56913,9417],{"class":503},[151,56915,56916],{"class":469,"line":3167},[151,56917,16824],{"class":503},[11,56919,56920,56921,56924,56925,56928,56929,643],{},"Step 4 inspects active and reserved tasks, filters the tasks by the ",[30,56922,56923],{},"routing_key"," (which is the same as the queue name) and the combines this with any queued tasks by calling ",[30,56926,56927],{},"llen(queue_name)",". Finally, the queue names and combined active, reserved and queued task totals are sent to CloudWatch via boto3. The code for this is in ",[30,56930,56931],{},"backend/apps/core/utils/celery_utils.py",[11,56933,56934],{},"Here's the code for the Lambda function in step 5:",[459,56936,56938],{"className":13136,"code":56937,"language":12886,"meta":464,"style":464},"import json\nimport os\nimport urllib.request\n\nFULL_DOMAIN_NAME = os.environ.get(\"FULL_DOMAIN_NAME\")\nCELERY_METRICS_PATH = \"api/celery-metrics/\"\n\nCELERY_METRICS_URL = f\"https://{FULL_DOMAIN_NAME}/{CELERY_METRICS_PATH}\"\nCELERY_METRICS_TOKEN = os.environ.get(\"CELERY_METRICS_TOKEN\")\n\n\ndef lambda_handler(event, context):\n    data = {\"celery_metrics_token\": CELERY_METRICS_TOKEN}\n    params = json.dumps(data).encode('utf8')\n    req = urllib.request.Request(\n        CELERY_METRICS_URL,\n        data=params,\n        headers={'content-type': 'application/json'},\n    )\n    response = urllib.request.urlopen(req)\n    return response.read()\n",[30,56939,56940,56946,56952,56959,56963,56977,56987,56991,57012,57026,57030,57034,57050,57067,57081,57091,57098,57108,57128,57132,57142],{"__ignoreMap":464},[151,56941,56942,56944],{"class":469,"line":470},[151,56943,16859],{"class":1869},[151,56945,24063],{"class":503},[151,56947,56948,56950],{"class":469,"line":488},[151,56949,16859],{"class":1869},[151,56951,24070],{"class":503},[151,56953,56954,56956],{"class":469,"line":500},[151,56955,16859],{"class":1869},[151,56957,56958],{"class":503}," urllib.request\n",[151,56960,56961],{"class":469,"line":509},[151,56962,1090],{"emptyLinePlaceholder":609},[151,56964,56965,56968,56970,56972,56975],{"class":469,"line":517},[151,56966,56967],{"class":477},"FULL_DOMAIN_NAME",[151,56969,19865],{"class":1869},[151,56971,36806],{"class":503},[151,56973,56974],{"class":481},"\"FULL_DOMAIN_NAME\"",[151,56976,3640],{"class":503},[151,56978,56979,56982,56984],{"class":469,"line":534},[151,56980,56981],{"class":477},"CELERY_METRICS_PATH",[151,56983,19865],{"class":1869},[151,56985,56986],{"class":481}," \"api/celery-metrics/\"\n",[151,56988,56989],{"class":469,"line":1413},[151,56990,1090],{"emptyLinePlaceholder":609},[151,56992,56993,56996,56998,57000,57002,57005,57007,57010],{"class":469,"line":1418},[151,56994,56995],{"class":477},"CELERY_METRICS_URL",[151,56997,19865],{"class":1869},[151,56999,36853],{"class":12347},[151,57001,53248],{"class":481},[151,57003,57004],{"class":477},"{FULL_DOMAIN_NAME}",[151,57006,19883],{"class":481},[151,57008,57009],{"class":477},"{CELERY_METRICS_PATH}",[151,57011,16406],{"class":481},[151,57013,57014,57017,57019,57021,57024],{"class":469,"line":2462},[151,57015,57016],{"class":477},"CELERY_METRICS_TOKEN",[151,57018,19865],{"class":1869},[151,57020,36806],{"class":503},[151,57022,57023],{"class":481},"\"CELERY_METRICS_TOKEN\"",[151,57025,3640],{"class":503},[151,57027,57028],{"class":469,"line":2471},[151,57029,1090],{"emptyLinePlaceholder":609},[151,57031,57032],{"class":469,"line":2480},[151,57033,1090],{"emptyLinePlaceholder":609},[151,57035,57036,57038,57040,57042,57044,57046,57048],{"class":469,"line":2489},[151,57037,16925],{"class":12347},[151,57039,51957],{"class":473},[151,57041,12386],{"class":503},[151,57043,39519],{"class":15232},[151,57045,106],{"class":503},[151,57047,39524],{"class":15232},[151,57049,15264],{"class":503},[151,57051,57052,57054,57056,57058,57061,57063,57065],{"class":469,"line":2497},[151,57053,53501],{"class":503},[151,57055,1876],{"class":1869},[151,57057,52023],{"class":503},[151,57059,57060],{"class":481},"\"celery_metrics_token\"",[151,57062,6208],{"class":503},[151,57064,57016],{"class":477},[151,57066,6274],{"class":503},[151,57068,57069,57072,57074,57077,57079],{"class":469,"line":3140},[151,57070,57071],{"class":503},"    params ",[151,57073,1876],{"class":1869},[151,57075,57076],{"class":503}," json.dumps(data).encode(",[151,57078,29328],{"class":481},[151,57080,3640],{"class":503},[151,57082,57083,57086,57088],{"class":469,"line":3149},[151,57084,57085],{"class":503},"    req ",[151,57087,1876],{"class":1869},[151,57089,57090],{"class":503}," urllib.request.Request(\n",[151,57092,57093,57096],{"class":469,"line":3158},[151,57094,57095],{"class":477},"        CELERY_METRICS_URL",[151,57097,9417],{"class":503},[151,57099,57100,57103,57105],{"class":469,"line":3167},[151,57101,57102],{"class":15210},"        data",[151,57104,1876],{"class":1869},[151,57106,57107],{"class":503},"params,\n",[151,57109,57110,57113,57115,57117,57120,57122,57125],{"class":469,"line":3175},[151,57111,57112],{"class":15210},"        headers",[151,57114,1876],{"class":1869},[151,57116,5729],{"class":503},[151,57118,57119],{"class":481},"'content-type'",[151,57121,6208],{"class":503},[151,57123,57124],{"class":481},"'application/json'",[151,57126,57127],{"class":503},"},\n",[151,57129,57130],{"class":469,"line":3184},[151,57131,39567],{"class":503},[151,57133,57134,57137,57139],{"class":469,"line":3193},[151,57135,57136],{"class":503},"    response ",[151,57138,1876],{"class":1869},[151,57140,57141],{"class":503}," urllib.request.urlopen(req)\n",[151,57143,57144,57146],{"class":469,"line":3720},[151,57145,17496],{"class":1869},[151,57147,57148],{"class":503}," response.read()\n",[11,57150,57151],{},"Finally, here is the code for defining the lambda function and scheduled invocation of the lambda function:",[459,57153,57155],{"className":13136,"code":57154,"language":12886,"meta":464,"style":464},"class CeleryAutoscalingStack(cloudformation.NestedStack):\n    def __init__(self, scope: core.Construct, id: str, **kwargs) -> None:\n        super().__init__(\n            scope, id, **kwargs,\n        )\n\n        self.lambda_function = aws_lambda.Function(\n            self,\n            \"CeleryMetricsLambdaFunction\",\n            code=aws_lambda.Code.asset(\"awslambda\"),\n            handler=\"publish_celery_metrics.lambda_handler\",\n            runtime=aws_lambda.Runtime.PYTHON_3_7,\n            environment=scope.variables.regular_variables,\n        )\n\n        self.celery_default_cw_metric_schedule = events.Rule(\n            self,\n            \"CeleryDefaultCWMetricSchedule\",\n            schedule=events.Schedule.rate(core.Duration.minutes(5)),\n            targets=[\n                events_targets.LambdaFunction(handler=self.lambda_function)\n            ],\n        )\n\n        # TODO: refactor this to loop through CloudWatch metrics multiple celery queues\n        scope.celery_default_service.default_celery_queue_cw_metric.grant_put_metric_data(\n            scope.backend_service.backend_task.task_role\n        )\n",[30,57156,57157,57174,57209,57219,57233,57237,57241,57253,57259,57266,57280,57291,57305,57314,57318,57322,57334,57340,57347,57362,57371,57385,57389,57393,57397,57407,57412,57417],{"__ignoreMap":464},[151,57158,57159,57161,57164,57166,57168,57170,57172],{"class":469,"line":470},[151,57160,16519],{"class":12347},[151,57162,57163],{"class":15254}," CeleryAutoscalingStack",[151,57165,12386],{"class":503},[151,57167,30124],{"class":15260},[151,57169,643],{"class":503},[151,57171,55629],{"class":15260},[151,57173,15264],{"class":503},[151,57175,57176,57178,57180,57182,57184,57186,57188,57191,57193,57195,57197,57199,57201,57203,57205,57207],{"class":469,"line":488},[151,57177,16566],{"class":12347},[151,57179,15272],{"class":2226},[151,57181,12386],{"class":503},[151,57183,15277],{"class":15232},[151,57185,106],{"class":503},[151,57187,37849],{"class":15232},[151,57189,57190],{"class":503},": core.Construct, ",[151,57192,47409],{"class":15232},[151,57194,6208],{"class":503},[151,57196,15343],{"class":6205},[151,57198,106],{"class":503},[151,57200,24677],{"class":1869},[151,57202,37866],{"class":15232},[151,57204,17374],{"class":503},[151,57206,15437],{"class":477},[151,57208,14372],{"class":503},[151,57210,57211,57213,57215,57217],{"class":469,"line":500},[151,57212,37877],{"class":6205},[151,57214,37880],{"class":503},[151,57216,37883],{"class":2226},[151,57218,15410],{"class":503},[151,57220,57221,57224,57226,57228,57230],{"class":469,"line":509},[151,57222,57223],{"class":503},"            scope, ",[151,57225,47409],{"class":2226},[151,57227,106],{"class":503},[151,57229,24677],{"class":1869},[151,57231,57232],{"class":503},"kwargs,\n",[151,57234,57235],{"class":469,"line":517},[151,57236,16824],{"class":503},[151,57238,57239],{"class":469,"line":534},[151,57240,1090],{"emptyLinePlaceholder":609},[151,57242,57243,57245,57248,57250],{"class":469,"line":1413},[151,57244,37901],{"class":15289},[151,57246,57247],{"class":503},".lambda_function ",[151,57249,1876],{"class":1869},[151,57251,57252],{"class":503}," aws_lambda.Function(\n",[151,57254,57255,57257],{"class":469,"line":1418},[151,57256,15290],{"class":15289},[151,57258,9417],{"class":503},[151,57260,57261,57264],{"class":469,"line":2462},[151,57262,57263],{"class":481},"            \"CeleryMetricsLambdaFunction\"",[151,57265,9417],{"class":503},[151,57267,57268,57270,57272,57275,57278],{"class":469,"line":2471},[151,57269,39688],{"class":15210},[151,57271,1876],{"class":1869},[151,57273,57274],{"class":503},"aws_lambda.Code.asset(",[151,57276,57277],{"class":481},"\"awslambda\"",[151,57279,37985],{"class":503},[151,57281,57282,57284,57286,57289],{"class":469,"line":2480},[151,57283,39791],{"class":15210},[151,57285,1876],{"class":1869},[151,57287,57288],{"class":481},"\"publish_celery_metrics.lambda_handler\"",[151,57290,9417],{"class":503},[151,57292,57293,57295,57297,57300,57303],{"class":469,"line":2489},[151,57294,39752],{"class":15210},[151,57296,1876],{"class":1869},[151,57298,57299],{"class":503},"aws_lambda.Runtime.",[151,57301,57302],{"class":477},"PYTHON_3_7",[151,57304,9417],{"class":503},[151,57306,57307,57309,57311],{"class":469,"line":2497},[151,57308,38021],{"class":15210},[151,57310,1876],{"class":1869},[151,57312,57313],{"class":503},"scope.variables.regular_variables,\n",[151,57315,57316],{"class":469,"line":3140},[151,57317,16824],{"class":503},[151,57319,57320],{"class":469,"line":3149},[151,57321,1090],{"emptyLinePlaceholder":609},[151,57323,57324,57326,57329,57331],{"class":469,"line":3158},[151,57325,37901],{"class":15289},[151,57327,57328],{"class":503},".celery_default_cw_metric_schedule ",[151,57330,1876],{"class":1869},[151,57332,57333],{"class":503}," events.Rule(\n",[151,57335,57336,57338],{"class":469,"line":3167},[151,57337,15290],{"class":15289},[151,57339,9417],{"class":503},[151,57341,57342,57345],{"class":469,"line":3175},[151,57343,57344],{"class":481},"            \"CeleryDefaultCWMetricSchedule\"",[151,57346,9417],{"class":503},[151,57348,57349,57352,57354,57357,57359],{"class":469,"line":3184},[151,57350,57351],{"class":15210},"            schedule",[151,57353,1876],{"class":1869},[151,57355,57356],{"class":503},"events.Schedule.rate(core.Duration.minutes(",[151,57358,24380],{"class":477},[151,57360,57361],{"class":503},")),\n",[151,57363,57364,57367,57369],{"class":469,"line":3193},[151,57365,57366],{"class":15210},"            targets",[151,57368,1876],{"class":1869},[151,57370,37620],{"class":503},[151,57372,57373,57376,57378,57380,57382],{"class":469,"line":3720},[151,57374,57375],{"class":503},"                events_targets.LambdaFunction(",[151,57377,51756],{"class":15210},[151,57379,1876],{"class":1869},[151,57381,15277],{"class":15289},[151,57383,57384],{"class":503},".lambda_function)\n",[151,57386,57387],{"class":469,"line":3729},[151,57388,52319],{"class":503},[151,57390,57391],{"class":469,"line":3735},[151,57392,16824],{"class":503},[151,57394,57395],{"class":469,"line":3745},[151,57396,1090],{"emptyLinePlaceholder":609},[151,57398,57399,57402,57404],{"class":469,"line":3754},[151,57400,57401],{"class":1527},"        # ",[151,57403,21038],{"class":1869},[151,57405,57406],{"class":1527},": refactor this to loop through CloudWatch metrics multiple celery queues\n",[151,57408,57409],{"class":469,"line":3760},[151,57410,57411],{"class":503},"        scope.celery_default_service.default_celery_queue_cw_metric.grant_put_metric_data(\n",[151,57413,57414],{"class":469,"line":3773},[151,57415,57416],{"class":503},"            scope.backend_service.backend_task.task_role\n",[151,57418,57419],{"class":469,"line":3782},[151,57420,16824],{"class":503},[56,57422,57424],{"id":57423},"miscellaneous-grievances","Miscellaneous Grievances",[76,57426,57427,57430,57436],{},[79,57428,57429],{},"Fargate tasks cannot access Secrets that use JSON template",[79,57431,57432,57433,57435],{},"You cannot select ",[30,57434,27359],{}," on a CDK-created ECS cluster. This option seems to only be available from the ECS Wizard in the AWS Console",[79,57437,57438,57439,57443],{},"It's not super clear how to package dependencies in Lambda with CDK. Here's an interesting approach that I would like to try: ",[20,57440,57441],{"href":57441,"rel":57442},"https://github.com/aws-samples/aws-cdk-examples/issues/130#issuecomment-554097487",[24],". The Lambda functions I'm using don't require anything outside of the standard library, but if they did it would require some additional work to make sure dependencies can be added as Lambda Layers.",[56,57445,21038],{"id":21037},[76,57447,57448,57451,57454,57457],{},[79,57449,57450],{},"Create an IAM role template that can be used to create an IAM role that has access to setup infrastructure through CDK. Currently I'm using an admin account which is not a best practice.",[79,57452,57453],{},"Move this Wiki Article to the project documentation site and main README",[79,57455,57456],{},"Expand on each topic, replace existing documentation",[79,57458,57459,57460,57462],{},"Add a summary of each ",[30,57461,55629],{}," from CDK code",[589,57464,57465],{},"html pre.shiki code .s8-w5, html code.shiki .s8-w5{--shiki-default:#6A737D;--shiki-dark:#6A737D;--shiki-sepia:#88846F}html pre.shiki code .sC2Qs, html code.shiki .sC2Qs{--shiki-default:#D73A49;--shiki-dark:#F97583;--shiki-sepia:#F92672}html pre.shiki code .sMOD_, html code.shiki .sMOD_{--shiki-default:#24292E;--shiki-dark:#E1E4E8;--shiki-sepia:#F8F8F2}html pre.shiki code .sq6CD, html code.shiki .sq6CD{--shiki-default:#D73A49;--shiki-default-font-style:inherit;--shiki-dark:#F97583;--shiki-dark-font-style:inherit;--shiki-sepia:#66D9EF;--shiki-sepia-font-style:italic}html pre.shiki code .sstjo, html code.shiki .sstjo{--shiki-default:#032F62;--shiki-dark:#9ECBFF;--shiki-sepia:#E6DB74}html pre.shiki code .s7F3e, html code.shiki .s7F3e{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#AE81FF}html pre.shiki code .sTHNf, html code.shiki .sTHNf{--shiki-default:#E36209;--shiki-default-font-style:inherit;--shiki-dark:#FFAB70;--shiki-dark-font-style:inherit;--shiki-sepia:#FD971F;--shiki-sepia-font-style:italic}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html .sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html.sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html pre.shiki code .s5clZ, html code.shiki .s5clZ{--shiki-default:#22863A;--shiki-dark:#85E89D;--shiki-sepia:#F92672}html pre.shiki code .sP7S_, html code.shiki .sP7S_{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#FD971F}html pre.shiki code .srTi1, html code.shiki .srTi1{--shiki-default:#6F42C1;--shiki-dark:#B392F0;--shiki-sepia:#A6E22E}html pre.shiki code .so59x, html code.shiki .so59x{--shiki-default:#24292E;--shiki-default-font-style:inherit;--shiki-dark:#E1E4E8;--shiki-dark-font-style:inherit;--shiki-sepia:#FD971F;--shiki-sepia-font-style:italic}html pre.shiki code .sz2Vg, html code.shiki .sz2Vg{--shiki-default:#6F42C1;--shiki-default-text-decoration:inherit;--shiki-dark:#B392F0;--shiki-dark-text-decoration:inherit;--shiki-sepia:#A6E22E;--shiki-sepia-text-decoration:underline}html pre.shiki code .s30JN, html code.shiki .s30JN{--shiki-default:#6F42C1;--shiki-default-font-style:inherit;--shiki-default-text-decoration:inherit;--shiki-dark:#B392F0;--shiki-dark-font-style:inherit;--shiki-dark-text-decoration:inherit;--shiki-sepia:#A6E22E;--shiki-sepia-font-style:italic;--shiki-sepia-text-decoration:underline}html pre.shiki code .sTrkL, html code.shiki .sTrkL{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#66D9EF}html pre.shiki code .s-m8C, html code.shiki .s-m8C{--shiki-default:#005CC5;--shiki-default-font-style:inherit;--shiki-dark:#79B8FF;--shiki-dark-font-style:inherit;--shiki-sepia:#66D9EF;--shiki-sepia-font-style:italic}",{"title":464,"searchDepth":488,"depth":488,"links":57467},[57468,57469,57481,57495,57502,57503,57504],{"id":54488,"depth":488,"text":54489},{"id":54536,"depth":488,"text":54537,"children":57470},[57471,57472,57473,57474,57475,57476,57477,57478,57479,57480],{"id":46183,"depth":500,"text":46184},{"id":29583,"depth":500,"text":29584},{"id":54673,"depth":500,"text":54674},{"id":27264,"depth":500,"text":51877},{"id":54761,"depth":500,"text":54762},{"id":54808,"depth":500,"text":54809},{"id":54834,"depth":500,"text":54835},{"id":54936,"depth":500,"text":54937},{"id":54995,"depth":500,"text":54507},{"id":55025,"depth":500,"text":55026},{"id":55072,"depth":488,"text":55073,"children":57482},[57483,57485,57486,57488,57490,57492,57493],{"id":55079,"depth":500,"text":57484},"Start here: https://cdkworkshop.com",{"id":55090,"depth":500,"text":55091},{"id":55586,"depth":500,"text":57487},"Using cdk synth in development",{"id":55626,"depth":500,"text":57489},"NestedStacks",{"id":55689,"depth":500,"text":57491},"cdk deploy and CDK in GitLab CI pipelines",{"id":55927,"depth":500,"text":55930},{"id":56167,"depth":500,"text":57494},"Quick tour of ApplicationStack",{"id":56494,"depth":488,"text":57496,"children":57497},"Why X? Why not Y?",[57498,57499,57500,57501],{"id":56508,"depth":500,"text":56509},{"id":56542,"depth":500,"text":56543},{"id":56549,"depth":500,"text":56550},{"id":56577,"depth":500,"text":56578},{"id":56584,"depth":488,"text":56585},{"id":57423,"depth":488,"text":57424},{"id":21037,"depth":488,"text":21038},"2020-06-02",{"layout":48045},"/2020/06/02/django-postgres-vue-gitlab-ecs",{"title":54432,"description":464},"2020/06/02/django-postgres-vue-gitlab-ecs",[30127,30122,47672,12646,30123],"7gxmSpzofdtgfIFcf8YvfpW3sxIY_o5Y1KTfUMP98Gg",{"id":57513,"title":57514,"body":57515,"comments":609,"date":57571,"description":57572,"draft":602,"extension":605,"external":606,"image":57573,"meta":57574,"navigation":609,"path":57575,"seo":57576,"stem":57577,"tags":57578,"__hash__":57579},"blog/2019/01/09/django-docker-vue-nginx-drf-traefik-celery.md","Setup, Development and Deployment of a web app using Django, VueJS, VuePress, Docker, nginx, traefik and GitLab",{"type":8,"value":57516,"toc":57569},[57517,57529,57532,57554,57557,57562],[11,57518,57519,57520,57524,57525,643],{},"This project is currently deployed on ",[20,57521,57522],{"href":57522,"rel":57523},"https://verbose-equals-true.tk",[24],". The source code is available at ",[20,57526,57527],{"href":57527,"rel":57528},"https://gitlab.com/briancaffey/verbose-equals-true",[24],[11,57530,57531],{},"The goal of this project is to explain how to setup a project starting with a fresh installation of 16.04. Setup includes local development environment, GitLab CI/CD, VSCode settings and configuration of the different containers that make up the application:",[76,57533,57534,57536,57539,57541,57543,57546,57548,57551],{},[79,57535,54466],{},[79,57537,57538],{},"Node (for local development with VueJS)",[79,57540,49765],{},[79,57542,51166],{},[79,57544,57545],{},"Postgres",[79,57547,27557],{},[79,57549,57550],{},"flower",[79,57552,57553],{},"portainer",[11,57555,57556],{},"Here's an overview of the architecture used in the application:",[11,57558,57559],{},[2718,57560],{"alt":20386,"src":57561},"/static/architecture.png",[11,57563,57564,57565,643],{},"Extensive documentation for this project can be found at ",[20,57566,57567],{"href":57567,"rel":57568},"https://verbose-equals-true.tk/docs",[24],{"title":464,"searchDepth":488,"depth":488,"links":57570},[],"2019-01-09","This project is currently deployed on https://verbose-equals-true.tk. The source code is available at https://gitlab.com/briancaffey/verbose-equals-true.","/static/technologies.png",{"layout":48045},"/2019/01/09/django-docker-vue-nginx-drf-traefik-celery",{"title":57514,"description":57572},"2019/01/09/django-docker-vue-nginx-drf-traefik-celery",[30122,12646,30129,47672],"chUpawMMoieGHXW3jOUznjOHny1Ne3eaXE-H0D6iZxE",{"id":57581,"title":57582,"body":57583,"comments":609,"date":59376,"description":59377,"draft":602,"extension":605,"external":606,"image":59378,"meta":59379,"navigation":609,"path":59380,"seo":59381,"stem":59382,"tags":59383,"__hash__":59387},"blog/2018/04/26/generating-music-from-guitar-tabs-with-python.md","Generating MIDI files from guitar tablatures with Python and regular expressions",{"type":8,"value":57584,"toc":59369},[57585,57592,57598,57601,57604,57607,57614,57617,57621,57624,57628,57634,57640,57646,57652,57658,57664,57667,57737,57741,57744,57754,57760,57764,57767,57775,57780,57783,58022,58025,58032,58035,58044,58047,58053,58059,58065,58071,58076,58082,58085,58091,58094,58097,58100,58105,58111,58313,58319,58322,59321,59324,59363,59366],[11,57586,57587,57588,57591],{},"Recently I have been playing lots of classical guitar. I use ",[20,57589,57590],{"href":57590},"classtab.org",", a website with hundreds of classical guitar pieces in tab form. Here's an example of a guitar tab:",[459,57593,57596],{"className":57594,"code":57595,"language":997},[995],"|---------------0---|---3-2-3-----------|-----0-------------|\n|-------0-3-1-1---3-|-3-------3-0-----0-|-1-3---3-1-0---0---|\n|---0-2-0-----------|-------------0-2---|-------------2---0-|\n|-------------2-----|-------2-----------|-------------------|\n|-------------------|-2-----------------|-0-----2-----3-----|\n|-3-----------------|-------------0-----|-------------------|\n",[30,57597,57595],{"__ignoreMap":464},[11,57599,57600],{},"It is a representation of where and when strings are played. The six lines here represent the six strings of the guitar e, A, D, G, B and E and the numbers indicate where on the fretboard to press for each note. The vertical bars represent one bar of music.",[11,57602,57603],{},"My goal is to write an application that can read in guitar tabs and then produce a corresponding MIDI file for the song.",[11,57605,57606],{},"To start, I will use regular expressions to parse the bars of music in a guitar tab text file.",[11,57608,57609,57610,57613],{},"Next, I will use a Python package called MIDIUtil to generate ",[30,57611,57612],{},".mid"," files from the guitar tabs.",[11,57615,57616],{},"Finally, I'll put this into a simple web app that lets users copy paste guitar tab text or enter a URL from a site like classtab.org that will similarly generate a MIDI file.",[56,57618,57620],{"id":57619},"regex","Regex",[11,57622,57623],{},"First, let's look at a few different examples of guitar tabs to get a better sense of what our goal should be in using regular expressions to parse notes.",[736,57625,57627],{"id":57626},"tab-examples","Tab examples",[11,57629,57630],{},[20,57631,57632],{"href":57632,"rel":57633},"http://www.classtab.org/barrios_la_catedral_3_allegro.txt",[24],[459,57635,57638],{"className":57636,"code":57637,"language":997},[995],"La Catedral\nAuthor: Agustin Barrios\nTablature by: Gianpiero Ciammaricone\n              giancian@yahoo.com\n              http://www.geocities.com/vienna/6619\n\nTablature Explanation at the end.\n\n[key corrected from \"D Major\" Jan 99]\n\nThis is Revision 3 (5/2/98), I have added a modification made by\nBarrios in line 5 of the TAB, both ways of playing it are included,\nso you can choose the best one for you.\nRevision 2: Made a few corrections.\nRevision 1: The first release of the tab.\n\n3rd Movement (Allegro):\nKey: B Minor\nTime signature: 6/8\n\nE-|-------------------------|-----------------------------|\nB-|-----------------3-------|-----------------3-----------|\nG-|---------0---4-------4---|---------0---4-------4-------|\nD-|---4-3h4---4---4---4---4-|---4-3h4---4---4---4---4-----|\nA-|-2-----------------------|-2---------------------------|\nE-|-------------------------|-----------------------------|\n    p m i   m i m i a i m i\n\nE-|-------------------------|-----------------------------|\nB-|-----------------3-------|-----------------3-----------|\nG-|---------2---4-------4---|---------2---4-------4-------|\nD-|---5-4h5---5---5---5---5-|---5-4h5---5---5---5---5-----|\nA-|-2-----------------------|-2---------------------------|\nE-|-------------------------|-----------------------------|\n    p m i   m i m i a i m i\n\nE-|-------------------------|-----------------------------|\nB-|-----------------5-------|-----------------5-----------|\nG-|---------4---6-------6---|---------4---6-------6-------|\nD-|---7-6h7---7---7---7---7-|---7-6h7---7---7---7---7-----|\nA-|-4-----------------------|-4---------------------------|\nE-|-------------------------|-----------------------------|\n    p m i   m i m i a i m i\n",[30,57639,57637],{"__ignoreMap":464},[11,57641,57642],{},[20,57643,57644],{"href":57644,"rel":57645},"http://www.classtab.org/moreno_torroba_siete_piezas_de_album_6_chisperada.txt",[24],[459,57647,57650],{"className":57648,"code":57649,"language":997},[995],"SIETE PIEZAS DE ALBUM: VI. CHISPERADA\nAs recorded by Federico Moreno Torroba\n\nMusic by Federico Moreno Torroba\n\n|-----0-----0-|-----------0-|-----0-----0-|\n|---1-----0---|-----3-----0-|---1-----0---|\n|---2-----0---|---2-------1-|---2-----0---|\n|---2---------|---3---------|---2---------|\n|-0-----------|---------2---|-0-----------|\n|-------3-----|-1-----0-----|-------3-----|\n\n|-----------0-|-----0-----0-|-----------0-|-----------0--|\n|-----3-----0-|---1-----0---|-----3-----0-|-----3-----0--|\n|---2-------1-|---2-----0---|---2-------1-|---2-------1--|\n|---3---------|---2---------|---3---------|---3----------|\n|---------2---|-0-----------|-0-------2---|-0-------2----|\n|-1-----0-----|-------3-----|-------0-----|-------0------|\n\n|---------------0--|-0-----0-----|-1-----1-3p1-0---|\n|---1-0---------0--|-1-----------|---------------3-|\n|-2-----2-0-----1--|-2-----1-----|-2-----2---------|\n|-----------3------|-------------|-------0---------|\n|-3----------------|---0-0---2-2-|---3-3-----------|\n|-------------0----|-------------|-----------------|\n",[30,57651,57649],{"__ignoreMap":464},[11,57653,57654],{},[20,57655,57656],{"href":57656,"rel":57657},"http://www.classtab.org/moreno_torroba_sonatina_in_a_2_andante.txt",[24],[459,57659,57662],{"className":57660,"code":57661,"language":997},[995],"Sonatina in A - II - Andante\n\nF. Moreno Torroba (1891-1982)\n\ntuning DADGBE\n\nE|-----2--3---5----------10--8--7-------|-----8----7^8^7-----------------------|\nB|-----3--5---7----------7--------------|-----8-----------10---------8---------|\nG|-----2---------------------9--7-------|-----9--------------------------11----|\nD|-----0--------------4-----------------|--------------------------10----------|\nA|-----0----------0---------------------|----------------------7---------------|\nD|-----0--------------------------------|-----0--------------------------------|\n\nE|--------------------------------------|--------------------------------------|\nB|------------------7--8^10-8---7^5-----|---5---------7-8-7-5-----5------------|\nG|--9^11^9----7--9------------------7---|---6-----------------7---6------------|\nD|-------8------------------------------|---5---------------------5--7----19°--|\nA|-------10-----------------------------|------0--12°--------------------------|\nD|-------0------------------------------|---0---------------------0------------|\n",[30,57663,57661],{"__ignoreMap":464},[11,57665,57666],{},"Here are a few things that we need to consider:",[76,57668,57669,57692,57709,57712,57724],{},[79,57670,57671,57672,30583,57675,106,57678,106,57680,57682,57683,57685,57686,57688,57689,643],{},"The start of each line sometimes starts with a the note of the string (",[30,57673,57674],{},"E",[30,57676,57677],{},"e",[30,57679,105],{},[30,57681,115],{},", etc.) followed by ",[30,57684,3947],{}," or just simply start with ",[30,57687,3947],{},". Some lines may start with ",[30,57690,57691],{},"E-|",[79,57693,57694,57695,57697,57698,106,57700,30583,57703,57706,57707,643],{},"There is generally a ",[30,57696,12445],{}," character between each note. When this is not the case, we may have a ",[30,57699,11],{},[30,57701,57702],{},"h",[30,57704,57705],{},"^"," which indicates \"pull off\" or \"hammer on\". This means that you are playing the string with the hand on the fretboard, not the hand over the sound hole. We should replace these with ",[30,57708,12445],{},[79,57710,57711],{},"A tab may have an inconsistant number of characters in each bar. I'm not too concerned about this. I'm not going for perfection just yet, but it could make this task very difficult if you want to get the timing of your sounds just right.",[79,57713,57714,57715,57718,57719,57721,57722,643],{},"Notes played on the 10th fret and higher must be read together. For example, ",[30,57716,57717],{},"---12---"," should be played on the 12 fret; it does not mean play ",[30,57720,6760],{}," and then play ",[30,57723,6619],{},[79,57725,57726,57727,30583,57730,19883,57732,57734,57735,55176],{},"There are some other special characters such as ",[30,57728,57729],{},"°",[30,57731,3613],{},[30,57733,3663],{},". I think these both denote harmonics. This is when you place your finger on the string at a position along the fretboard, but do not press down. We probably want to replace these characters with ",[30,57736,12445],{},[736,57738,57740],{"id":57739},"tab-text-file-processing","Tab text file processing",[11,57742,57743],{},"Here are some things that may be helpful to do as soon as we read a text file:",[76,57745,57746,57749],{},[79,57747,57748],{},"convert everything to upper or lower case",[79,57750,57751,57752],{},"replace non-numeric characters with ",[30,57753,12445],{},[11,57755,57756,57757,643],{},"When I work with regex, I usually go straight to ",[20,57758,57759],{"href":57759},"regex101.com",[56,57761,57763],{"id":57762},"midiutil","MIDIUtil",[11,57765,57766],{},"At this point, I think it will be helpful to examine MIDIUtil and create and play a basic MIDI file.",[11,57768,57769,57770,643],{},"Before you do that, you might want to read over the ",[20,57771,57774],{"href":57772,"rel":57773},"https://wiki.archlinux.org/index.php/MIDI",[24],"MIDI Arch Wiki article",[210,57776,57777],{},[11,57778,57779],{},"MIDIUtil is a pure Python library that allows one to write multi-track Musical Instrument Digital Interface (MIDI) files from within Python programs (both format 1 and format 2 files are now supported). It is object-oriented and allows one to create and write these files with a minimum of fuss.",[11,57781,57782],{},"Here's a basic example of MIDIUtil. This script generates a MIDI file that contains a major scale.",[459,57784,57786],{"className":13136,"code":57785,"language":12886,"meta":464,"style":464},"#!/usr/bin/env python\n\nfrom midiutil import MIDIFile\n\ndegrees  = [60, 62, 64, 65, 67, 69, 71, 72]  # MIDI note number\ntrack    = 0\nchannel  = 0\ntime     = 0    # In beats\nduration = 1    # In beats\ntempo    = 60   # In BPM\nvolume   = 100  # 0-127, as per the MIDI standard\n\nMyMIDI = MIDIFile(1)  # One track, defaults to format 1 (tempo track is created\n                      # automatically)\nMyMIDI.addTempo(track, time, tempo)\n\nfor i, pitch in enumerate(degrees):\n    MyMIDI.addNote(track, channel, pitch, time + i, duration, volume)\n\nwith open(\"major-scale.mid\", \"wb\") as output_file:\n    MyMIDI.writeFile(output_file)\n",[30,57787,57788,57793,57797,57809,57813,57862,57872,57881,57894,57905,57918,57931,57935,57953,57958,57963,57967,57981,57991,57995,58017],{"__ignoreMap":464},[151,57789,57790],{"class":469,"line":470},[151,57791,57792],{"class":1527},"#!/usr/bin/env python\n",[151,57794,57795],{"class":469,"line":488},[151,57796,1090],{"emptyLinePlaceholder":609},[151,57798,57799,57801,57804,57806],{"class":469,"line":500},[151,57800,16853],{"class":1869},[151,57802,57803],{"class":503}," midiutil ",[151,57805,16859],{"class":1869},[151,57807,57808],{"class":503}," MIDIFile\n",[151,57810,57811],{"class":469,"line":509},[151,57812,1090],{"emptyLinePlaceholder":609},[151,57814,57815,57818,57820,57822,57824,57826,57828,57830,57833,57835,57838,57840,57843,57845,57848,57850,57852,57854,57856,57859],{"class":469,"line":517},[151,57816,57817],{"class":503},"degrees  ",[151,57819,1876],{"class":1869},[151,57821,6604],{"class":503},[151,57823,39825],{"class":477},[151,57825,106],{"class":503},[151,57827,9218],{"class":477},[151,57829,106],{"class":503},[151,57831,57832],{"class":477},"64",[151,57834,106],{"class":503},[151,57836,57837],{"class":477},"65",[151,57839,106],{"class":503},[151,57841,57842],{"class":477},"67",[151,57844,106],{"class":503},[151,57846,57847],{"class":477},"69",[151,57849,106],{"class":503},[151,57851,41742],{"class":477},[151,57853,106],{"class":503},[151,57855,9232],{"class":477},[151,57857,57858],{"class":503},"]  ",[151,57860,57861],{"class":1527},"# MIDI note number\n",[151,57863,57864,57867,57869],{"class":469,"line":534},[151,57865,57866],{"class":503},"track    ",[151,57868,1876],{"class":1869},[151,57870,57871],{"class":477}," 0\n",[151,57873,57874,57877,57879],{"class":469,"line":1413},[151,57875,57876],{"class":503},"channel  ",[151,57878,1876],{"class":1869},[151,57880,57871],{"class":477},[151,57882,57883,57886,57888,57891],{"class":469,"line":1418},[151,57884,57885],{"class":503},"time     ",[151,57887,1876],{"class":1869},[151,57889,57890],{"class":477}," 0",[151,57892,57893],{"class":1527},"    # In beats\n",[151,57895,57896,57899,57901,57903],{"class":469,"line":2462},[151,57897,57898],{"class":503},"duration ",[151,57900,1876],{"class":1869},[151,57902,12448],{"class":477},[151,57904,57893],{"class":1527},[151,57906,57907,57910,57912,57915],{"class":469,"line":2471},[151,57908,57909],{"class":503},"tempo    ",[151,57911,1876],{"class":1869},[151,57913,57914],{"class":477}," 60",[151,57916,57917],{"class":1527},"   # In BPM\n",[151,57919,57920,57923,57925,57928],{"class":469,"line":2480},[151,57921,57922],{"class":503},"volume   ",[151,57924,1876],{"class":1869},[151,57926,57927],{"class":477}," 100",[151,57929,57930],{"class":1527},"  # 0-127, as per the MIDI standard\n",[151,57932,57933],{"class":469,"line":2489},[151,57934,1090],{"emptyLinePlaceholder":609},[151,57936,57937,57940,57942,57945,57947,57950],{"class":469,"line":2497},[151,57938,57939],{"class":503},"MyMIDI ",[151,57941,1876],{"class":1869},[151,57943,57944],{"class":503}," MIDIFile(",[151,57946,6760],{"class":477},[151,57948,57949],{"class":503},")  ",[151,57951,57952],{"class":1527},"# One track, defaults to format 1 (tempo track is created\n",[151,57954,57955],{"class":469,"line":3140},[151,57956,57957],{"class":1527},"                      # automatically)\n",[151,57959,57960],{"class":469,"line":3149},[151,57961,57962],{"class":503},"MyMIDI.addTempo(track, time, tempo)\n",[151,57964,57965],{"class":469,"line":3158},[151,57966,1090],{"emptyLinePlaceholder":609},[151,57968,57969,57971,57974,57976,57978],{"class":469,"line":3167},[151,57970,16732],{"class":1869},[151,57972,57973],{"class":503}," i, pitch ",[151,57975,16417],{"class":1869},[151,57977,17042],{"class":2226},[151,57979,57980],{"class":503},"(degrees):\n",[151,57982,57983,57986,57988],{"class":469,"line":3175},[151,57984,57985],{"class":503},"    MyMIDI.addNote(track, channel, pitch, time ",[151,57987,22885],{"class":1869},[151,57989,57990],{"class":503}," i, duration, volume)\n",[151,57992,57993],{"class":469,"line":3184},[151,57994,1090],{"emptyLinePlaceholder":609},[151,57996,57997,57999,58001,58003,58006,58008,58010,58012,58014],{"class":469,"line":3193},[151,57998,24959],{"class":1869},[151,58000,16970],{"class":2226},[151,58002,12386],{"class":503},[151,58004,58005],{"class":481},"\"major-scale.mid\"",[151,58007,106],{"class":503},[151,58009,41144],{"class":481},[151,58011,16995],{"class":503},[151,58013,16998],{"class":1869},[151,58015,58016],{"class":503}," output_file:\n",[151,58018,58019],{"class":469,"line":3720},[151,58020,58021],{"class":503},"    MyMIDI.writeFile(output_file)\n",[11,58023,58024],{},"Now let's try to play this file. Well, if you read the MIDI article, you would know that this isn't really what we are doing. We want to see what our major scale sounds like. In order to do that, we need to install and configure both a MIDI synthesizer as well as a soundfont.",[11,58026,58027,58028,643],{},"We can use timidity++ for our MIDI synth. There are very simple instructions on how to do this ",[20,58029,13074],{"href":58030,"rel":58031},"https://wiki.archlinux.org/index.php/timidity",[24],[11,58033,58034],{},"Here are the steps I followed:",[11,58036,58037,58038,187,58041,643],{},"From the AUR, install ",[30,58039,58040],{},"timidity++",[30,58042,58043],{},"timidity-freepats",[11,58045,58046],{},"Add yourself to the audio group:",[459,58048,58051],{"className":58049,"code":58050,"language":997},[995],"# gpasswd -a brian audio\n",[30,58052,58050],{"__ignoreMap":464},[11,58054,58055,58056,208],{},"Add the following line to ",[30,58057,58058],{},"/etc/timidity++/timidity.cfg",[459,58060,58063],{"className":58061,"code":58062,"language":997},[995],"soundfont /usr/share/soundfonts/timidity-freepats.sf2\n",[30,58064,58062],{"__ignoreMap":464},[459,58066,58069],{"className":58067,"code":58068,"language":997},[995],"sudo systemctl start timidity.service\nsudo systemctl enable timidity.service\n",[30,58070,58068],{"__ignoreMap":464},[11,58072,10635,58073,208],{},[30,58074,58075],{},"aplaymidi -l",[459,58077,58080],{"className":58078,"code":58079,"language":997},[995]," $ aplaymidi -l\n Port    Client name                      Port name\n 14:0    Midi Through                     Midi Through Port-0\n130:0    TiMidity                         TiMidity port 0\n130:1    TiMidity                         TiMidity port 1\n130:2    TiMidity                         TiMidity port 2\n130:3    TiMidity                         TiMidity port 3\n",[30,58081,58079],{"__ignoreMap":464},[11,58083,58084],{},"Finally, we can play our MIDI file that we generated by running the python script above:",[459,58086,58089],{"className":58087,"code":58088,"language":997},[995]," $ aplaymidi major-scale.mid --port 130:0\n\u003Cmusic plays...>\n",[30,58090,58088],{"__ignoreMap":464},[11,58092,58093],{},"For this project, playing files using a synthesizer is useful to verify that the MIDI was successfully created. If/when we get to the point of making a web app to convert midi files to music files, we will generate the midifile in-memory, convert it to a WAV file and then return that to the user.",[11,58095,58096],{},"Let's try to process a very simple tab file with just one string and a few notes, and then generate a simple MIDI file based on the tab.",[11,58098,58099],{},"Our sample tab can be:",[11,58101,58102],{},[51,58103,58104],{},"simple-tab.txt",[459,58106,58109],{"className":58107,"code":58108,"language":997},[995],"--0--2--4--\n",[30,58110,58108],{"__ignoreMap":464},[459,58112,58113],{"className":13136,"code":57785,"language":12886,"meta":464,"style":464},[30,58114,58115,58119,58123,58133,58137,58179,58187,58195,58205,58215,58225,58235,58239,58253,58257,58261,58265,58277,58285,58289,58309],{"__ignoreMap":464},[151,58116,58117],{"class":469,"line":470},[151,58118,57792],{"class":1527},[151,58120,58121],{"class":469,"line":488},[151,58122,1090],{"emptyLinePlaceholder":609},[151,58124,58125,58127,58129,58131],{"class":469,"line":500},[151,58126,16853],{"class":1869},[151,58128,57803],{"class":503},[151,58130,16859],{"class":1869},[151,58132,57808],{"class":503},[151,58134,58135],{"class":469,"line":509},[151,58136,1090],{"emptyLinePlaceholder":609},[151,58138,58139,58141,58143,58145,58147,58149,58151,58153,58155,58157,58159,58161,58163,58165,58167,58169,58171,58173,58175,58177],{"class":469,"line":517},[151,58140,57817],{"class":503},[151,58142,1876],{"class":1869},[151,58144,6604],{"class":503},[151,58146,39825],{"class":477},[151,58148,106],{"class":503},[151,58150,9218],{"class":477},[151,58152,106],{"class":503},[151,58154,57832],{"class":477},[151,58156,106],{"class":503},[151,58158,57837],{"class":477},[151,58160,106],{"class":503},[151,58162,57842],{"class":477},[151,58164,106],{"class":503},[151,58166,57847],{"class":477},[151,58168,106],{"class":503},[151,58170,41742],{"class":477},[151,58172,106],{"class":503},[151,58174,9232],{"class":477},[151,58176,57858],{"class":503},[151,58178,57861],{"class":1527},[151,58180,58181,58183,58185],{"class":469,"line":534},[151,58182,57866],{"class":503},[151,58184,1876],{"class":1869},[151,58186,57871],{"class":477},[151,58188,58189,58191,58193],{"class":469,"line":1413},[151,58190,57876],{"class":503},[151,58192,1876],{"class":1869},[151,58194,57871],{"class":477},[151,58196,58197,58199,58201,58203],{"class":469,"line":1418},[151,58198,57885],{"class":503},[151,58200,1876],{"class":1869},[151,58202,57890],{"class":477},[151,58204,57893],{"class":1527},[151,58206,58207,58209,58211,58213],{"class":469,"line":2462},[151,58208,57898],{"class":503},[151,58210,1876],{"class":1869},[151,58212,12448],{"class":477},[151,58214,57893],{"class":1527},[151,58216,58217,58219,58221,58223],{"class":469,"line":2471},[151,58218,57909],{"class":503},[151,58220,1876],{"class":1869},[151,58222,57914],{"class":477},[151,58224,57917],{"class":1527},[151,58226,58227,58229,58231,58233],{"class":469,"line":2480},[151,58228,57922],{"class":503},[151,58230,1876],{"class":1869},[151,58232,57927],{"class":477},[151,58234,57930],{"class":1527},[151,58236,58237],{"class":469,"line":2489},[151,58238,1090],{"emptyLinePlaceholder":609},[151,58240,58241,58243,58245,58247,58249,58251],{"class":469,"line":2497},[151,58242,57939],{"class":503},[151,58244,1876],{"class":1869},[151,58246,57944],{"class":503},[151,58248,6760],{"class":477},[151,58250,57949],{"class":503},[151,58252,57952],{"class":1527},[151,58254,58255],{"class":469,"line":3140},[151,58256,57957],{"class":1527},[151,58258,58259],{"class":469,"line":3149},[151,58260,57962],{"class":503},[151,58262,58263],{"class":469,"line":3158},[151,58264,1090],{"emptyLinePlaceholder":609},[151,58266,58267,58269,58271,58273,58275],{"class":469,"line":3167},[151,58268,16732],{"class":1869},[151,58270,57973],{"class":503},[151,58272,16417],{"class":1869},[151,58274,17042],{"class":2226},[151,58276,57980],{"class":503},[151,58278,58279,58281,58283],{"class":469,"line":3175},[151,58280,57985],{"class":503},[151,58282,22885],{"class":1869},[151,58284,57990],{"class":503},[151,58286,58287],{"class":469,"line":3184},[151,58288,1090],{"emptyLinePlaceholder":609},[151,58290,58291,58293,58295,58297,58299,58301,58303,58305,58307],{"class":469,"line":3193},[151,58292,24959],{"class":1869},[151,58294,16970],{"class":2226},[151,58296,12386],{"class":503},[151,58298,58005],{"class":481},[151,58300,106],{"class":503},[151,58302,41144],{"class":481},[151,58304,16995],{"class":503},[151,58306,16998],{"class":1869},[151,58308,58016],{"class":503},[151,58310,58311],{"class":469,"line":3720},[151,58312,58021],{"class":503},[459,58314,58317],{"className":58315,"code":58316,"language":997},[995]," $ echo \"--0--2--4--\" > simple-tab.txt\n",[30,58318,58316],{"__ignoreMap":464},[11,58320,58321],{},"Here's the script I used to generate some fairly accurate MIDI files. There are still some bugs in the script that generates the MIDI file, but this meets the goal I had last weekend of writing a simple script to generate music from guitar tabs.",[459,58323,58325],{"className":13136,"code":58324,"language":12886,"meta":464,"style":464},"#!/usr/bin/env python\n\nfrom midiutil import MIDIFile\nimport sys\nimport re\n\nstring_notes = [\"high_e\", \"B\", \"G\", \"D\", \"A\", \"E\"]\n\nguitar_strings = {\n    'E':{'note_val':52, 'track_num':0},\n    'A':{'note_val':57, 'track_num':1},\n    'D':{'note_val':62, 'track_num':2},\n    'G':{'note_val':67, 'track_num':3},\n    'B':{'note_val':71, 'track_num':4},\n    'high_e':{'note_val':76, 'track_num':5},\n}\n\n# read the tab file\nfile_name = sys.argv[1]\nif file_name.split(\".\")[-1] != 'txt':\n    print(\"Please select a text file\")\n\nwith open(file_name) as f:\n    contents = f.read()\n\ncontents = contents.replace(\"h\", \"-\")\ncontents = contents.replace(\"p\", \"-\")\ncontents = contents.replace(\"/\", \"-\")\ncontents = contents.replace(\"*\", \"-\")\ncontents = contents.upper()\nbar_group = re.findall(r\"(?:[E,B,G,D,A,-]+\\|[0-9-h|]+\\n){6}\",contents)\n\n#bar_group = re.findall(r\"(?:\\|[0-9-\\*h\\|]+\\n){6}\",contents)\n\n\ntrack    = 0\nchannel  = 0\ntime     = 0    # In beats\nduration = 1    # In beats\ntempo    = 1000   # In BPM\nvolume   = 100  # 0-127, as per the MIDI standard\n\nMyMIDI = MIDIFile(6)  # One track, defaults to format 1 (tempo track is created\n                      # automatically)\nMyMIDI.addTempo(track, time, tempo)\n\ninterval = len(bar_group[0].split(\"\\n\")) - 1\n\nfor b in bar_group:\n\n    strings = b.split(\"\\n\")\n    strings = [x for x in strings if x != '']\n    e_count = 0\n    for i,s in enumerate(strings):\n\n        current_string = strings[i][0]\n        if current_string not in guitar_strings.keys():\n            current_string = string_notes[i]\n        if current_string == \"E\":\n            e_count += 1\n        if e_count == 2:\n            current_string == \"high_e\"\n\n        track = guitar_strings[current_string]['track_num']\n\n        s = s[1:]\n        s = s.replace('|', '')\n        s = list(s)\n\n        for i, pitch in enumerate(s):\n            volume = 100\n\n            if pitch == \"\\n\":\n                break\n            if pitch == \"-\":\n                volume = 0\n                pitch = 50\n            print(\"adding note\")\n            pitch = int(pitch) + guitar_strings[current_string]['note_val']\n            MyMIDI.addNote(track, channel, pitch, time + i, duration, volume)\n\n    time += interval*8\n\nwith open(\"major-scale.mid\", \"wb\") as output_file:\n    MyMIDI.writeFile(output_file)\n",[30,58326,58327,58331,58335,58345,58352,58359,58363,58402,58406,58415,58440,58464,58487,58510,58533,58557,58561,58565,58570,58583,58608,58619,58623,58636,58646,58650,58670,58687,58703,58720,58729,58774,58778,58783,58787,58791,58799,58807,58817,58827,58838,58848,58852,58866,58870,58874,58878,58906,58910,58922,58926,58944,58972,58981,58995,58999,59013,59027,59037,59050,59059,59073,59082,59086,59100,59104,59119,59137,59149,59153,59166,59176,59180,59197,59202,59215,59224,59234,59246,59266,59275,59279,59293,59297,59317],{"__ignoreMap":464},[151,58328,58329],{"class":469,"line":470},[151,58330,57792],{"class":1527},[151,58332,58333],{"class":469,"line":488},[151,58334,1090],{"emptyLinePlaceholder":609},[151,58336,58337,58339,58341,58343],{"class":469,"line":500},[151,58338,16853],{"class":1869},[151,58340,57803],{"class":503},[151,58342,16859],{"class":1869},[151,58344,57808],{"class":503},[151,58346,58347,58349],{"class":469,"line":509},[151,58348,16859],{"class":1869},[151,58350,58351],{"class":503}," sys\n",[151,58353,58354,58356],{"class":469,"line":517},[151,58355,16859],{"class":1869},[151,58357,58358],{"class":503}," re\n",[151,58360,58361],{"class":469,"line":534},[151,58362,1090],{"emptyLinePlaceholder":609},[151,58364,58365,58368,58370,58372,58375,58377,58380,58382,58385,58387,58390,58392,58395,58397,58400],{"class":469,"line":1413},[151,58366,58367],{"class":503},"string_notes ",[151,58369,1876],{"class":1869},[151,58371,6604],{"class":503},[151,58373,58374],{"class":481},"\"high_e\"",[151,58376,106],{"class":503},[151,58378,58379],{"class":481},"\"B\"",[151,58381,106],{"class":503},[151,58383,58384],{"class":481},"\"G\"",[151,58386,106],{"class":503},[151,58388,58389],{"class":481},"\"D\"",[151,58391,106],{"class":503},[151,58393,58394],{"class":481},"\"A\"",[151,58396,106],{"class":503},[151,58398,58399],{"class":481},"\"E\"",[151,58401,3691],{"class":503},[151,58403,58404],{"class":469,"line":1418},[151,58405,1090],{"emptyLinePlaceholder":609},[151,58407,58408,58411,58413],{"class":469,"line":2462},[151,58409,58410],{"class":503},"guitar_strings ",[151,58412,1876],{"class":1869},[151,58414,19833],{"class":503},[151,58416,58417,58420,58422,58425,58427,58429,58431,58434,58436,58438],{"class":469,"line":2471},[151,58418,58419],{"class":481},"    'E'",[151,58421,9795],{"class":503},[151,58423,58424],{"class":481},"'note_val'",[151,58426,208],{"class":503},[151,58428,45428],{"class":477},[151,58430,106],{"class":503},[151,58432,58433],{"class":481},"'track_num'",[151,58435,208],{"class":503},[151,58437,9181],{"class":477},[151,58439,57127],{"class":503},[151,58441,58442,58445,58447,58449,58451,58454,58456,58458,58460,58462],{"class":469,"line":2480},[151,58443,58444],{"class":481},"    'A'",[151,58446,9795],{"class":503},[151,58448,58424],{"class":481},[151,58450,208],{"class":503},[151,58452,58453],{"class":477},"57",[151,58455,106],{"class":503},[151,58457,58433],{"class":481},[151,58459,208],{"class":503},[151,58461,6760],{"class":477},[151,58463,57127],{"class":503},[151,58465,58466,58469,58471,58473,58475,58477,58479,58481,58483,58485],{"class":469,"line":2489},[151,58467,58468],{"class":481},"    'D'",[151,58470,9795],{"class":503},[151,58472,58424],{"class":481},[151,58474,208],{"class":503},[151,58476,9218],{"class":477},[151,58478,106],{"class":503},[151,58480,58433],{"class":481},[151,58482,208],{"class":503},[151,58484,6619],{"class":477},[151,58486,57127],{"class":503},[151,58488,58489,58492,58494,58496,58498,58500,58502,58504,58506,58508],{"class":469,"line":2497},[151,58490,58491],{"class":481},"    'G'",[151,58493,9795],{"class":503},[151,58495,58424],{"class":481},[151,58497,208],{"class":503},[151,58499,57842],{"class":477},[151,58501,106],{"class":503},[151,58503,58433],{"class":481},[151,58505,208],{"class":503},[151,58507,6557],{"class":477},[151,58509,57127],{"class":503},[151,58511,58512,58515,58517,58519,58521,58523,58525,58527,58529,58531],{"class":469,"line":3140},[151,58513,58514],{"class":481},"    'B'",[151,58516,9795],{"class":503},[151,58518,58424],{"class":481},[151,58520,208],{"class":503},[151,58522,41742],{"class":477},[151,58524,106],{"class":503},[151,58526,58433],{"class":481},[151,58528,208],{"class":503},[151,58530,9187],{"class":477},[151,58532,57127],{"class":503},[151,58534,58535,58538,58540,58542,58544,58547,58549,58551,58553,58555],{"class":469,"line":3149},[151,58536,58537],{"class":481},"    'high_e'",[151,58539,9795],{"class":503},[151,58541,58424],{"class":481},[151,58543,208],{"class":503},[151,58545,58546],{"class":477},"76",[151,58548,106],{"class":503},[151,58550,58433],{"class":481},[151,58552,208],{"class":503},[151,58554,24380],{"class":477},[151,58556,57127],{"class":503},[151,58558,58559],{"class":469,"line":3158},[151,58560,6274],{"class":503},[151,58562,58563],{"class":469,"line":3167},[151,58564,1090],{"emptyLinePlaceholder":609},[151,58566,58567],{"class":469,"line":3175},[151,58568,58569],{"class":1527},"# read the tab file\n",[151,58571,58572,58574,58576,58579,58581],{"class":469,"line":3184},[151,58573,45707],{"class":503},[151,58575,1876],{"class":1869},[151,58577,58578],{"class":503}," sys.argv[",[151,58580,6760],{"class":477},[151,58582,3691],{"class":503},[151,58584,58585,58587,58590,58592,58594,58596,58598,58600,58603,58606],{"class":469,"line":3193},[151,58586,17218],{"class":1869},[151,58588,58589],{"class":503}," file_name.split(",[151,58591,44221],{"class":481},[151,58593,40832],{"class":503},[151,58595,12445],{"class":1869},[151,58597,6760],{"class":477},[151,58599,16654],{"class":503},[151,58601,58602],{"class":1869},"!=",[151,58604,58605],{"class":481}," 'txt'",[151,58607,14372],{"class":503},[151,58609,58610,58612,58614,58617],{"class":469,"line":3720},[151,58611,24285],{"class":2226},[151,58613,12386],{"class":503},[151,58615,58616],{"class":481},"\"Please select a text file\"",[151,58618,3640],{"class":503},[151,58620,58621],{"class":469,"line":3729},[151,58622,1090],{"emptyLinePlaceholder":609},[151,58624,58625,58627,58629,58632,58634],{"class":469,"line":3735},[151,58626,24959],{"class":1869},[151,58628,16970],{"class":2226},[151,58630,58631],{"class":503},"(file_name) ",[151,58633,16998],{"class":1869},[151,58635,17001],{"class":503},[151,58637,58638,58641,58643],{"class":469,"line":3745},[151,58639,58640],{"class":503},"    contents ",[151,58642,1876],{"class":1869},[151,58644,58645],{"class":503}," f.read()\n",[151,58647,58648],{"class":469,"line":3754},[151,58649,1090],{"emptyLinePlaceholder":609},[151,58651,58652,58655,58657,58660,58663,58665,58668],{"class":469,"line":3760},[151,58653,58654],{"class":503},"contents ",[151,58656,1876],{"class":1869},[151,58658,58659],{"class":503}," contents.replace(",[151,58661,58662],{"class":481},"\"h\"",[151,58664,106],{"class":503},[151,58666,58667],{"class":481},"\"-\"",[151,58669,3640],{"class":503},[151,58671,58672,58674,58676,58678,58681,58683,58685],{"class":469,"line":3773},[151,58673,58654],{"class":503},[151,58675,1876],{"class":1869},[151,58677,58659],{"class":503},[151,58679,58680],{"class":481},"\"p\"",[151,58682,106],{"class":503},[151,58684,58667],{"class":481},[151,58686,3640],{"class":503},[151,58688,58689,58691,58693,58695,58697,58699,58701],{"class":469,"line":3782},[151,58690,58654],{"class":503},[151,58692,1876],{"class":1869},[151,58694,58659],{"class":503},[151,58696,45143],{"class":481},[151,58698,106],{"class":503},[151,58700,58667],{"class":481},[151,58702,3640],{"class":503},[151,58704,58705,58707,58709,58711,58714,58716,58718],{"class":469,"line":3791},[151,58706,58654],{"class":503},[151,58708,1876],{"class":1869},[151,58710,58659],{"class":503},[151,58712,58713],{"class":481},"\"*\"",[151,58715,106],{"class":503},[151,58717,58667],{"class":481},[151,58719,3640],{"class":503},[151,58721,58722,58724,58726],{"class":469,"line":3803},[151,58723,58654],{"class":503},[151,58725,1876],{"class":1869},[151,58727,58728],{"class":503}," contents.upper()\n",[151,58730,58731,58734,58736,58739,58742,58744,58748,58751,58753,58757,58760,58762,58764,58766,58769,58771],{"class":469,"line":3811},[151,58732,58733],{"class":503},"bar_group ",[151,58735,1876],{"class":1869},[151,58737,58738],{"class":503}," re.findall(",[151,58740,58741],{"class":12347},"r",[151,58743,8592],{"class":481},[151,58745,58747],{"class":58746},"sLkwE","(?:",[151,58749,58750],{"class":477},"[E,B,G,D,A,-]",[151,58752,22885],{"class":1869},[151,58754,58756],{"class":58755},"sHuvb","\\|",[151,58758,58759],{"class":477},"[0-9-h|]",[151,58761,22885],{"class":1869},[151,58763,8043],{"class":58755},[151,58765,748],{"class":58746},[151,58767,58768],{"class":1869},"{6}",[151,58770,8592],{"class":481},[151,58772,58773],{"class":503},",contents)\n",[151,58775,58776],{"class":469,"line":3820},[151,58777,1090],{"emptyLinePlaceholder":609},[151,58779,58780],{"class":469,"line":7084},[151,58781,58782],{"class":1527},"#bar_group = re.findall(r\"(?:\\|[0-9-\\*h\\|]+\\n){6}\",contents)\n",[151,58784,58785],{"class":469,"line":7148},[151,58786,1090],{"emptyLinePlaceholder":609},[151,58788,58789],{"class":469,"line":7211},[151,58790,1090],{"emptyLinePlaceholder":609},[151,58792,58793,58795,58797],{"class":469,"line":7273},[151,58794,57866],{"class":503},[151,58796,1876],{"class":1869},[151,58798,57871],{"class":477},[151,58800,58801,58803,58805],{"class":469,"line":7335},[151,58802,57876],{"class":503},[151,58804,1876],{"class":1869},[151,58806,57871],{"class":477},[151,58808,58809,58811,58813,58815],{"class":469,"line":7398},[151,58810,57885],{"class":503},[151,58812,1876],{"class":1869},[151,58814,57890],{"class":477},[151,58816,57893],{"class":1527},[151,58818,58819,58821,58823,58825],{"class":469,"line":7462},[151,58820,57898],{"class":503},[151,58822,1876],{"class":1869},[151,58824,12448],{"class":477},[151,58826,57893],{"class":1527},[151,58828,58829,58831,58833,58836],{"class":469,"line":7467},[151,58830,57909],{"class":503},[151,58832,1876],{"class":1869},[151,58834,58835],{"class":477}," 1000",[151,58837,57917],{"class":1527},[151,58839,58840,58842,58844,58846],{"class":469,"line":7532},[151,58841,57922],{"class":503},[151,58843,1876],{"class":1869},[151,58845,57927],{"class":477},[151,58847,57930],{"class":1527},[151,58849,58850],{"class":469,"line":7537},[151,58851,1090],{"emptyLinePlaceholder":609},[151,58853,58854,58856,58858,58860,58862,58864],{"class":469,"line":7603},[151,58855,57939],{"class":503},[151,58857,1876],{"class":1869},[151,58859,57944],{"class":503},[151,58861,25038],{"class":477},[151,58863,57949],{"class":503},[151,58865,57952],{"class":1527},[151,58867,58868],{"class":469,"line":7608},[151,58869,57957],{"class":1527},[151,58871,58872],{"class":469,"line":7673},[151,58873,57962],{"class":503},[151,58875,58876],{"class":469,"line":7678},[151,58877,1090],{"emptyLinePlaceholder":609},[151,58879,58880,58883,58885,58887,58890,58892,58894,58896,58898,58900,58902,58904],{"class":469,"line":7708},[151,58881,58882],{"class":503},"interval ",[151,58884,1876],{"class":1869},[151,58886,45035],{"class":2226},[151,58888,58889],{"class":503},"(bar_group[",[151,58891,9181],{"class":477},[151,58893,52005],{"class":503},[151,58895,8592],{"class":481},[151,58897,8043],{"class":477},[151,58899,8592],{"class":481},[151,58901,34074],{"class":503},[151,58903,12445],{"class":1869},[151,58905,3181],{"class":477},[151,58907,58908],{"class":469,"line":7713},[151,58909,1090],{"emptyLinePlaceholder":609},[151,58911,58912,58914,58917,58919],{"class":469,"line":7746},[151,58913,16732],{"class":1869},[151,58915,58916],{"class":503}," b ",[151,58918,16417],{"class":1869},[151,58920,58921],{"class":503}," bar_group:\n",[151,58923,58924],{"class":469,"line":7751},[151,58925,1090],{"emptyLinePlaceholder":609},[151,58927,58928,58931,58933,58936,58938,58940,58942],{"class":469,"line":7816},[151,58929,58930],{"class":503},"    strings ",[151,58932,1876],{"class":1869},[151,58934,58935],{"class":503}," b.split(",[151,58937,8592],{"class":481},[151,58939,8043],{"class":477},[151,58941,8592],{"class":481},[151,58943,3640],{"class":503},[151,58945,58946,58948,58950,58952,58954,58956,58958,58961,58963,58965,58967,58970],{"class":469,"line":7821},[151,58947,58930],{"class":503},[151,58949,1876],{"class":1869},[151,58951,45004],{"class":503},[151,58953,16732],{"class":1869},[151,58955,44552],{"class":503},[151,58957,16417],{"class":1869},[151,58959,58960],{"class":503}," strings ",[151,58962,17218],{"class":1869},[151,58964,44552],{"class":503},[151,58966,58602],{"class":1869},[151,58968,58969],{"class":481}," ''",[151,58971,3691],{"class":503},[151,58973,58974,58977,58979],{"class":469,"line":7847},[151,58975,58976],{"class":503},"    e_count ",[151,58978,1876],{"class":1869},[151,58980,57871],{"class":477},[151,58982,58983,58985,58988,58990,58992],{"class":469,"line":7852},[151,58984,16411],{"class":1869},[151,58986,58987],{"class":503}," i,s ",[151,58989,16417],{"class":1869},[151,58991,17042],{"class":2226},[151,58993,58994],{"class":503},"(strings):\n",[151,58996,58997],{"class":469,"line":7887},[151,58998,1090],{"emptyLinePlaceholder":609},[151,59000,59001,59004,59006,59009,59011],{"class":469,"line":7892},[151,59002,59003],{"class":503},"        current_string ",[151,59005,1876],{"class":1869},[151,59007,59008],{"class":503}," strings[i][",[151,59010,9181],{"class":477},[151,59012,3691],{"class":503},[151,59014,59015,59017,59020,59022,59024],{"class":469,"line":7924},[151,59016,23357],{"class":1869},[151,59018,59019],{"class":503}," current_string ",[151,59021,241],{"class":1869},[151,59023,2820],{"class":1869},[151,59025,59026],{"class":503}," guitar_strings.keys():\n",[151,59028,59029,59032,59034],{"class":469,"line":7929},[151,59030,59031],{"class":503},"            current_string ",[151,59033,1876],{"class":1869},[151,59035,59036],{"class":503}," string_notes[i]\n",[151,59038,59039,59041,59043,59045,59048],{"class":469,"line":7991},[151,59040,23357],{"class":1869},[151,59042,59019],{"class":503},[151,59044,17223],{"class":1869},[151,59046,59047],{"class":481}," \"E\"",[151,59049,14372],{"class":503},[151,59051,59052,59055,59057],{"class":469,"line":7996},[151,59053,59054],{"class":503},"            e_count ",[151,59056,24780],{"class":1869},[151,59058,3181],{"class":477},[151,59060,59061,59063,59066,59068,59071],{"class":469,"line":8078},[151,59062,23357],{"class":1869},[151,59064,59065],{"class":503}," e_count ",[151,59067,17223],{"class":1869},[151,59069,59070],{"class":477}," 2",[151,59072,14372],{"class":503},[151,59074,59075,59077,59079],{"class":469,"line":8140},[151,59076,59031],{"class":503},[151,59078,17223],{"class":1869},[151,59080,59081],{"class":481}," \"high_e\"\n",[151,59083,59084],{"class":469,"line":8145},[151,59085,1090],{"emptyLinePlaceholder":609},[151,59087,59088,59091,59093,59096,59098],{"class":469,"line":8259},[151,59089,59090],{"class":503},"        track ",[151,59092,1876],{"class":1869},[151,59094,59095],{"class":503}," guitar_strings[current_string][",[151,59097,58433],{"class":481},[151,59099,3691],{"class":503},[151,59101,59102],{"class":469,"line":8264},[151,59103,1090],{"emptyLinePlaceholder":609},[151,59105,59106,59109,59111,59114,59116],{"class":469,"line":8613},[151,59107,59108],{"class":503},"        s ",[151,59110,1876],{"class":1869},[151,59112,59113],{"class":503}," s[",[151,59115,6760],{"class":477},[151,59117,59118],{"class":503},":]\n",[151,59120,59121,59123,59125,59128,59131,59133,59135],{"class":469,"line":8678},[151,59122,59108],{"class":503},[151,59124,1876],{"class":1869},[151,59126,59127],{"class":503}," s.replace(",[151,59129,59130],{"class":481},"'|'",[151,59132,106],{"class":503},[151,59134,2301],{"class":481},[151,59136,3640],{"class":503},[151,59138,59139,59141,59143,59146],{"class":469,"line":8742},[151,59140,59108],{"class":503},[151,59142,1876],{"class":1869},[151,59144,59145],{"class":6205}," list",[151,59147,59148],{"class":503},"(s)\n",[151,59150,59151],{"class":469,"line":8806},[151,59152,1090],{"emptyLinePlaceholder":609},[151,59154,59155,59157,59159,59161,59163],{"class":469,"line":8870},[151,59156,16616],{"class":1869},[151,59158,57973],{"class":503},[151,59160,16417],{"class":1869},[151,59162,17042],{"class":2226},[151,59164,59165],{"class":503},"(s):\n",[151,59167,59168,59171,59173],{"class":469,"line":8875},[151,59169,59170],{"class":503},"            volume ",[151,59172,1876],{"class":1869},[151,59174,59175],{"class":477}," 100\n",[151,59177,59178],{"class":469,"line":8881},[151,59179,1090],{"emptyLinePlaceholder":609},[151,59181,59182,59184,59187,59189,59191,59193,59195],{"class":469,"line":8886},[151,59183,40442],{"class":1869},[151,59185,59186],{"class":503}," pitch ",[151,59188,17223],{"class":1869},[151,59190,16722],{"class":481},[151,59192,8043],{"class":477},[151,59194,8592],{"class":481},[151,59196,14372],{"class":503},[151,59198,59199],{"class":469,"line":8892},[151,59200,59201],{"class":1869},"                break\n",[151,59203,59204,59206,59208,59210,59213],{"class":469,"line":8963},[151,59205,40442],{"class":1869},[151,59207,59186],{"class":503},[151,59209,17223],{"class":1869},[151,59211,59212],{"class":481}," \"-\"",[151,59214,14372],{"class":503},[151,59216,59217,59220,59222],{"class":469,"line":8969},[151,59218,59219],{"class":503},"                volume ",[151,59221,1876],{"class":1869},[151,59223,57871],{"class":477},[151,59225,59226,59229,59231],{"class":469,"line":15001},[151,59227,59228],{"class":503},"                pitch ",[151,59230,1876],{"class":1869},[151,59232,59233],{"class":477}," 50\n",[151,59235,59236,59239,59241,59244],{"class":469,"line":15009},[151,59237,59238],{"class":2226},"            print",[151,59240,12386],{"class":503},[151,59242,59243],{"class":481},"\"adding note\"",[151,59245,3640],{"class":503},[151,59247,59248,59251,59253,59255,59258,59260,59262,59264],{"class":469,"line":15019},[151,59249,59250],{"class":503},"            pitch ",[151,59252,1876],{"class":1869},[151,59254,16673],{"class":6205},[151,59256,59257],{"class":503},"(pitch) ",[151,59259,22885],{"class":1869},[151,59261,59095],{"class":503},[151,59263,58424],{"class":481},[151,59265,3691],{"class":503},[151,59267,59268,59271,59273],{"class":469,"line":15027},[151,59269,59270],{"class":503},"            MyMIDI.addNote(track, channel, pitch, time ",[151,59272,22885],{"class":1869},[151,59274,57990],{"class":503},[151,59276,59277],{"class":469,"line":15037},[151,59278,1090],{"emptyLinePlaceholder":609},[151,59280,59281,59284,59286,59289,59291],{"class":469,"line":15045},[151,59282,59283],{"class":503},"    time ",[151,59285,24780],{"class":1869},[151,59287,59288],{"class":503}," interval",[151,59290,23268],{"class":1869},[151,59292,3726],{"class":477},[151,59294,59295],{"class":469,"line":15055},[151,59296,1090],{"emptyLinePlaceholder":609},[151,59298,59299,59301,59303,59305,59307,59309,59311,59313,59315],{"class":469,"line":15060},[151,59300,24959],{"class":1869},[151,59302,16970],{"class":2226},[151,59304,12386],{"class":503},[151,59306,58005],{"class":481},[151,59308,106],{"class":503},[151,59310,41144],{"class":481},[151,59312,16995],{"class":503},[151,59314,16998],{"class":1869},[151,59316,58016],{"class":503},[151,59318,59319],{"class":469,"line":15068},[151,59320,58021],{"class":503},[11,59322,59323],{},"Here's me playing one of my favorite songs, you might recognize it!",[210,59325,59330],{"className":59326,"dataInstgrmCaptioned":464,"dataInstgrmPermalink":59328,"dataInstgrmVersion":24369,"style":59329},[59327],"instagram-media","https://www.instagram.com/p/rg2gmyyFdC/"," background:#FFF; border:0; border-radius:3px; box-shadow:0 0 1px 0 rgba(0,0,0,0.5),0 1px 10px 0 rgba(0,0,0,0.15); margin: 1px; max-width:658px; padding:0; width:99.375%; width:-webkit-calc(100% - 2px); width:calc(100% - 2px);",[23950,59331,26792,59333,26792,59339,26792,59347],{"style":59332},"padding:8px;",[23950,59334,26792,59336],{"style":59335}," background:#F8F8F8; line-height:0; margin-top:40px; padding:50% 0; text-align:center; width:100%;",[23950,59337],{"style":59338}," background:url(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACwAAAAsCAMAAAApWqozAAAABGdBTUEAALGPC/xhBQAAAAFzUkdCAK7OHOkAAAAMUExURczMzPf399fX1+bm5mzY9AMAAADiSURBVDjLvZXbEsMgCES5/P8/t9FuRVCRmU73JWlzosgSIIZURCjo/ad+EQJJB4Hv8BFt+IDpQoCx1wjOSBFhh2XssxEIYn3ulI/6MNReE07UIWJEv8UEOWDS88LY97kqyTliJKKtuYBbruAyVh5wOHiXmpi5we58Ek028czwyuQdLKPG1Bkb4NnM+VeAnfHqn1k4+GPT6uGQcvu2h2OVuIf/gWUFyy8OWEpdyZSa3aVCqpVoVvzZZ2VTnn2wU8qzVjDDetO90GSy9mVLqtgYSy231MxrY6I2gGqjrTY0L8fxCxfCBbhWrsYYAAAAAElFTkSuQmCC); display:block; height:44px; margin:0 auto -44px; position:relative; top:-22px; width:44px;",[11,59340,26792,59342],{"style":59341}," margin:8px 0 0 0; padding:0 4px;",[20,59343,59346],{"href":59328,"style":59344,"target":59345}," color:#000; font-family:Arial,sans-serif; font-size:14px; font-style:normal; font-weight:normal; line-height:17px; text-decoration:none; word-wrap:break-word;","_blank","Written in 1902, this catchy tune from Fransico Tárrega's 'Gran Vals' is now heard by an estimated 22,000 per second worldwide.. it is also a registered sound trademark of a billion dollar Finnish company 🎵 #franciscotárrega #tarrega #tárrega #classicalguitar #spanishguitarist #spain #instamusic #granvals #grandevalse #grandwaltz #crownmolding",[11,59348,59350,59351,59356,59357],{"style":59349}," color:#c9c8cd; font-family:Arial,sans-serif; font-size:14px; line-height:17px; margin-bottom:0; margin-top:8px; overflow:hidden; padding:8px 0 7px; text-align:center; text-overflow:ellipsis; white-space:nowrap;","A post shared by ",[20,59352,59355],{"href":59353,"style":59354,"target":59345},"https://www.instagram.com/briancaffey/"," color:#c9c8cd; font-family:Arial,sans-serif; font-size:14px; font-style:normal; font-weight:normal; line-height:17px;"," Brian"," (@briancaffey) on ",[59358,59359,59362],"time",{"style":59360,"dateTime":59361}," font-family:Arial,sans-serif; font-size:14px; line-height:17px;","2014-08-10T09:48:59+00:00","Aug 10, 2014 at 2:48am PDT",[19822,59364],{"async":609,"defer":609,"src":59365},"//www.instagram.com/embed.js",[589,59367,59368],{},"html pre.shiki code .s8-w5, html code.shiki .s8-w5{--shiki-default:#6A737D;--shiki-dark:#6A737D;--shiki-sepia:#88846F}html pre.shiki code .sC2Qs, html code.shiki .sC2Qs{--shiki-default:#D73A49;--shiki-dark:#F97583;--shiki-sepia:#F92672}html pre.shiki code .sMOD_, html code.shiki .sMOD_{--shiki-default:#24292E;--shiki-dark:#E1E4E8;--shiki-sepia:#F8F8F2}html pre.shiki code .s7F3e, html code.shiki .s7F3e{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#AE81FF}html pre.shiki code .sTrkL, html code.shiki .sTrkL{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#66D9EF}html pre.shiki code .sstjo, html code.shiki .sstjo{--shiki-default:#032F62;--shiki-dark:#9ECBFF;--shiki-sepia:#E6DB74}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html .sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html.sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html pre.shiki code .sq6CD, html code.shiki .sq6CD{--shiki-default:#D73A49;--shiki-default-font-style:inherit;--shiki-dark:#F97583;--shiki-dark-font-style:inherit;--shiki-sepia:#66D9EF;--shiki-sepia-font-style:italic}html pre.shiki code .sLkwE, html code.shiki .sLkwE{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#E6DB74}html pre.shiki code .sHuvb, html code.shiki .sHuvb{--shiki-default:#22863A;--shiki-default-font-weight:bold;--shiki-dark:#85E89D;--shiki-dark-font-weight:bold;--shiki-sepia:#AE81FF;--shiki-sepia-font-weight:inherit}html pre.shiki code .s-m8C, html code.shiki .s-m8C{--shiki-default:#005CC5;--shiki-default-font-style:inherit;--shiki-dark:#79B8FF;--shiki-dark-font-style:inherit;--shiki-sepia:#66D9EF;--shiki-sepia-font-style:italic}",{"title":464,"searchDepth":488,"depth":488,"links":59370},[59371,59375],{"id":57619,"depth":488,"text":57620,"children":59372},[59373,59374],{"id":57626,"depth":500,"text":57627},{"id":57739,"depth":500,"text":57740},{"id":57762,"depth":488,"text":57763},"2018-04-26","Recently I have been playing lots of classical guitar. I use classtab.org, a website with hundreds of classical guitar pieces in tab form. Here's an example of a guitar tab:","/static/guitar.jpg",{"layout":48045},"/2018/04/26/generating-music-from-guitar-tabs-with-python",{"title":57582,"description":59377},"2018/04/26/generating-music-from-guitar-tabs-with-python",[59384,12886,57619,59385,59386],"midi","guitar","music","UM5gRYCvBPtdFu3vOJYl1nJzHdN6ZoRUbvKaRD5BzBU",{"id":59389,"title":59390,"body":59391,"comments":609,"date":64594,"description":464,"draft":602,"extension":605,"external":606,"image":64595,"meta":64596,"navigation":609,"path":64598,"seo":64599,"stem":64600,"tags":64601,"__hash__":64607},"blog/2018/02/19/leaflet-maps-with-django.md","Display, filter and export geographical data in a Django app with Leaflet, Mapbox, DataTables, Bootstrap 4 and Travis-CI",{"type":8,"value":59392,"toc":64582},[59393,59398,59406,59414,59419,59426,59439,59442,59526,59529,59532,59536,59539,59545,59648,59651,59655,59661,59667,59670,59909,59916,59943,59949,60107,60110,60113,60115,60729,60731,60735,60742,60745,60951,60957,60959,61113,61115,61127,61138,61147,61156,61160,61167,61216,61225,61235,61495,61498,61505,61507,61564,61566,61573,61577,61593,61596,61884,61898,61916,62346,62352,62459,62463,62466,62485,62658,62664,62666,63170,63172,63190,63200,63206,63216,63219,63362,63365,63379,63383,63386,63391,63396,63399,63617,63620,63625,63632,63697,63703,63958,64008,64011,64015,64018,64028,64142,64145,64160,64170,64174,64177,64190,64195,64424,64429,64560,64563,64569,64572,64579],[11,59394,59395],{},[2718,59396],{"alt":20386,"src":59397},"/static/map_entire.png",[736,59399,59401],{"id":59400},"live-demo-on-digitalocean",[20,59402,59405],{"href":59403,"rel":59404},"http://159.89.235.193/books/",[24],"Live Demo on DigitalOcean",[11,59407,59408,59409,208],{},"This post is a review of my first attempt at using geographical data in a Django project. I have been interested in working with map APIs, and I once looked into the Google Maps API. For this project I chose to use ",[20,59410,59413],{"href":59411,"rel":59412},"http://leafletjs.com/",[24],"Leaflet",[210,59415,59416],{},[11,59417,59418],{},"an open-source JavaScript library for mobile-friendly interactive maps",[11,59420,59421,59422,643],{},"Getting started with Leaflet is easy. All you need to do is request a public Mapbox API key which is free (with no credit card required). You can get a key from ",[20,59423,59424],{"href":59424,"rel":59425},"https://www.mapbox.com/account/access-tokens/",[24],[11,59427,59428,59429,59434,59435,59438],{},"Then you will follow steps on the ",[20,59430,59433],{"href":59431,"rel":59432},"http://leafletjs.com/examples/quick-start/",[24],"quickstart guide"," and replace ",[30,59436,59437],{},"your.mapbox.access.token"," with your Mapbox API key.",[11,59440,59441],{},"{% raw %}",[459,59443,59445],{"className":19459,"code":59444,"language":19461,"meta":464,"style":464},"L.tileLayer(\n  'https://api.tiles.mapbox.com/v4/{id}/{z}/{x}/{y}.png?access_token={accessToken}',\n  {\n    attribution:\n      'Map data © \u003Ca href=\"http://openstreetmap.org\">OpenStreetMap\u003C/a> contributors, \u003Ca href=\"http://creativecommons.org/licenses/by-sa/2.0/\">CC-BY-SA\u003C/a>, Imagery © \u003Ca href=\"http://mapbox.com\">Mapbox\u003C/a>',\n    maxZoom: 18,\n    id: 'mapbox.streets',\n    accessToken: 'your.mapbox.access.token',\n  }\n).addTo(mymap)\n",[30,59446,59447,59459,59466,59471,59476,59483,59492,59502,59512,59516],{"__ignoreMap":464},[151,59448,59449,59452,59454,59457],{"class":469,"line":470},[151,59450,59451],{"class":12360},"L",[151,59453,643],{"class":503},[151,59455,59456],{"class":473},"tileLayer",[151,59458,15410],{"class":503},[151,59460,59461,59464],{"class":469,"line":488},[151,59462,59463],{"class":481},"  'https://api.tiles.mapbox.com/v4/{id}/{z}/{x}/{y}.png?access_token={accessToken}'",[151,59465,9417],{"class":503},[151,59467,59468],{"class":469,"line":500},[151,59469,59470],{"class":503},"  {\n",[151,59472,59473],{"class":469,"line":509},[151,59474,59475],{"class":503},"    attribution:\n",[151,59477,59478,59481],{"class":469,"line":517},[151,59479,59480],{"class":481},"      'Map data © \u003Ca href=\"http://openstreetmap.org\">OpenStreetMap\u003C/a> contributors, \u003Ca href=\"http://creativecommons.org/licenses/by-sa/2.0/\">CC-BY-SA\u003C/a>, Imagery © \u003Ca href=\"http://mapbox.com\">Mapbox\u003C/a>'",[151,59482,9417],{"class":503},[151,59484,59485,59488,59490],{"class":469,"line":534},[151,59486,59487],{"class":503},"    maxZoom: ",[151,59489,7696],{"class":477},[151,59491,9417],{"class":503},[151,59493,59494,59497,59500],{"class":469,"line":1413},[151,59495,59496],{"class":503},"    id: ",[151,59498,59499],{"class":481},"'mapbox.streets'",[151,59501,9417],{"class":503},[151,59503,59504,59507,59510],{"class":469,"line":1418},[151,59505,59506],{"class":503},"    accessToken: ",[151,59508,59509],{"class":481},"'your.mapbox.access.token'",[151,59511,9417],{"class":503},[151,59513,59514],{"class":469,"line":2462},[151,59515,19957],{"class":503},[151,59517,59518,59520,59523],{"class":469,"line":2471},[151,59519,13576],{"class":503},[151,59521,59522],{"class":473},"addTo",[151,59524,59525],{"class":503},"(mymap)\n",[11,59527,59528],{},"{% endraw %}",[11,59530,59531],{},"I'll come back to using Leaflet later on, and also show how to use Leaflet plugins that can be used to add functionality such as aggregating markers on a map.",[56,59533,59535],{"id":59534},"django-model","Django model",[11,59537,59538],{},"Next, let's look at the Django setup. For now I'm simply working with a local project and SQLite3 backend.",[11,59540,59541,59542,208],{},"To store latitude and longitude data, we can use ",[30,59543,59544],{},"DecimalField",[459,59546,59548],{"className":13136,"code":59547,"language":12886,"meta":464,"style":464},"class Book(models.Model):\n    title = models.CharField(max_length=300)\n    lat = models.DecimalField(max_digits=9, decimal_places=6)\n    lon = models.DecimalField(max_digits=9, decimal_places=6)\n    [...]\n",[30,59549,59550,59568,59587,59615,59640],{"__ignoreMap":464},[151,59551,59552,59554,59557,59559,59562,59564,59566],{"class":469,"line":470},[151,59553,16519],{"class":12347},[151,59555,59556],{"class":15254}," Book",[151,59558,12386],{"class":503},[151,59560,59561],{"class":15260},"models",[151,59563,643],{"class":503},[151,59565,1612],{"class":15260},[151,59567,15264],{"class":503},[151,59569,59570,59573,59575,59578,59580,59582,59585],{"class":469,"line":488},[151,59571,59572],{"class":503},"    title ",[151,59574,1876],{"class":1869},[151,59576,59577],{"class":503}," models.CharField(",[151,59579,305],{"class":15210},[151,59581,1876],{"class":1869},[151,59583,59584],{"class":477},"300",[151,59586,3640],{"class":503},[151,59588,59589,59592,59594,59597,59600,59602,59604,59606,59609,59611,59613],{"class":469,"line":500},[151,59590,59591],{"class":503},"    lat ",[151,59593,1876],{"class":1869},[151,59595,59596],{"class":503}," models.DecimalField(",[151,59598,59599],{"class":15210},"max_digits",[151,59601,1876],{"class":1869},[151,59603,7918],{"class":477},[151,59605,106],{"class":503},[151,59607,59608],{"class":15210},"decimal_places",[151,59610,1876],{"class":1869},[151,59612,25038],{"class":477},[151,59614,3640],{"class":503},[151,59616,59617,59620,59622,59624,59626,59628,59630,59632,59634,59636,59638],{"class":469,"line":509},[151,59618,59619],{"class":503},"    lon ",[151,59621,1876],{"class":1869},[151,59623,59596],{"class":503},[151,59625,59599],{"class":15210},[151,59627,1876],{"class":1869},[151,59629,7918],{"class":477},[151,59631,106],{"class":503},[151,59633,59608],{"class":15210},[151,59635,1876],{"class":1869},[151,59637,25038],{"class":477},[151,59639,3640],{"class":503},[151,59641,59642,59644,59646],{"class":469,"line":517},[151,59643,33774],{"class":503},[151,59645,27455],{"class":477},[151,59647,3691],{"class":503},[11,59649,59650],{},"This allows us to store numbers to six decimal places. The difference between 0 and 0.00001 is about 3 feet, so this gives us plenty of accuracy for our geographical coordinate data.",[56,59652,59654],{"id":59653},"plotting-books-on-a-map","Plotting books on a map",[11,59656,59657,59658,59660],{},"Plotting our books on a map is fairly straightforward, but there were a few obstacles to work around. Initially I setup an additional endpoint that returned an AJAX response with the longitude, latitude, title and link for each book. The ",[30,59659,47833],{}," function in the AJAX call then passed the returned JSON into a function that populated the book data in the map.",[11,59662,55909,59663,59666],{},[30,59664,59665],{},"books"," page displays book data in two places: on the map and on the DataTable below the map. Since I was already making one database query for DataTables, I decided to reuse this data for the map and not use a separate request and database query. There are a few helpful idioms in Python and JavaScript to accomplish this.",[11,59668,59669],{},"First, let's look at the request with some added comments:",[459,59671,59673],{"className":13136,"code":59672,"language":12886,"meta":464,"style":464},"def all_books(request):\n    \"\"\"\n    Main view for books. request.GET parameters are used to filter books\n    \"\"\"\n    books = Book.objects.all()\n    form = QueryForm(request.GET or None)\n    paramDict = request.GET\n\n    books = filter_books(books, paramDict)\n\n    page_count = books.aggregate(Sum('pages'))\n\n    # This takes the first book query an reformats the data so it can be read\n    # by the map script on the frontend.\n    map_books = [{'loc':[float(book.lon), float(book.lat)],\n                  'title':book.title,\n                  'url':book.get_absolute_url()} for book in books]\n    context = {\n        'books':books,\n        # Here, we apply `json.dumps`, `escapejs` and `marksafe` for security\n        # and proper formatting\n        'map_books': mark_safe(escapejs(json.dumps(map_books))),\n        'page_count':page_count['pages__sum'],\n        'form':form}\n    return render(request, 'books/books.html', context)\n\n",[30,59674,59675,59689,59693,59698,59702,59712,59730,59741,59745,59754,59758,59773,59777,59782,59787,59814,59822,59840,59849,59857,59862,59867,59875,59888,59896],{"__ignoreMap":464},[151,59676,59677,59679,59682,59684,59687],{"class":469,"line":470},[151,59678,16925],{"class":12347},[151,59680,59681],{"class":473}," all_books",[151,59683,12386],{"class":503},[151,59685,59686],{"class":15232},"request",[151,59688,15264],{"class":503},[151,59690,59691],{"class":469,"line":488},[151,59692,17384],{"class":481},[151,59694,59695],{"class":469,"line":500},[151,59696,59697],{"class":481},"    Main view for books. request.GET parameters are used to filter books\n",[151,59699,59700],{"class":469,"line":509},[151,59701,17384],{"class":481},[151,59703,59704,59707,59709],{"class":469,"line":517},[151,59705,59706],{"class":503},"    books ",[151,59708,1876],{"class":1869},[151,59710,59711],{"class":503}," Book.objects.all()\n",[151,59713,59714,59717,59719,59722,59724,59726,59728],{"class":469,"line":534},[151,59715,59716],{"class":503},"    form ",[151,59718,1876],{"class":1869},[151,59720,59721],{"class":503}," QueryForm(request.",[151,59723,47765],{"class":477},[151,59725,2161],{"class":1869},[151,59727,40451],{"class":477},[151,59729,3640],{"class":503},[151,59731,59732,59735,59737,59739],{"class":469,"line":1413},[151,59733,59734],{"class":503},"    paramDict ",[151,59736,1876],{"class":1869},[151,59738,40684],{"class":503},[151,59740,14433],{"class":477},[151,59742,59743],{"class":469,"line":1418},[151,59744,1090],{"emptyLinePlaceholder":609},[151,59746,59747,59749,59751],{"class":469,"line":2462},[151,59748,59706],{"class":503},[151,59750,1876],{"class":1869},[151,59752,59753],{"class":503}," filter_books(books, paramDict)\n",[151,59755,59756],{"class":469,"line":2471},[151,59757,1090],{"emptyLinePlaceholder":609},[151,59759,59760,59763,59765,59768,59771],{"class":469,"line":2480},[151,59761,59762],{"class":503},"    page_count ",[151,59764,1876],{"class":1869},[151,59766,59767],{"class":503}," books.aggregate(Sum(",[151,59769,59770],{"class":481},"'pages'",[151,59772,12451],{"class":503},[151,59774,59775],{"class":469,"line":2489},[151,59776,1090],{"emptyLinePlaceholder":609},[151,59778,59779],{"class":469,"line":2497},[151,59780,59781],{"class":1527},"    # This takes the first book query an reformats the data so it can be read\n",[151,59783,59784],{"class":469,"line":3140},[151,59785,59786],{"class":1527},"    # by the map script on the frontend.\n",[151,59788,59789,59792,59794,59797,59800,59803,59806,59809,59811],{"class":469,"line":3149},[151,59790,59791],{"class":503},"    map_books ",[151,59793,1876],{"class":1869},[151,59795,59796],{"class":503}," [{",[151,59798,59799],{"class":481},"'loc'",[151,59801,59802],{"class":503},":[",[151,59804,59805],{"class":6205},"float",[151,59807,59808],{"class":503},"(book.lon), ",[151,59810,59805],{"class":6205},[151,59812,59813],{"class":503},"(book.lat)],\n",[151,59815,59816,59819],{"class":469,"line":3158},[151,59817,59818],{"class":481},"                  'title'",[151,59820,59821],{"class":503},":book.title,\n",[151,59823,59824,59827,59830,59832,59835,59837],{"class":469,"line":3167},[151,59825,59826],{"class":481},"                  'url'",[151,59828,59829],{"class":503},":book.get_absolute_url()} ",[151,59831,16732],{"class":1869},[151,59833,59834],{"class":503}," book ",[151,59836,16417],{"class":1869},[151,59838,59839],{"class":503}," books]\n",[151,59841,59842,59845,59847],{"class":469,"line":3175},[151,59843,59844],{"class":503},"    context ",[151,59846,1876],{"class":1869},[151,59848,19833],{"class":503},[151,59850,59851,59854],{"class":469,"line":3184},[151,59852,59853],{"class":481},"        'books'",[151,59855,59856],{"class":503},":books,\n",[151,59858,59859],{"class":469,"line":3193},[151,59860,59861],{"class":1527},"        # Here, we apply `json.dumps`, `escapejs` and `marksafe` for security\n",[151,59863,59864],{"class":469,"line":3720},[151,59865,59866],{"class":1527},"        # and proper formatting\n",[151,59868,59869,59872],{"class":469,"line":3729},[151,59870,59871],{"class":481},"        'map_books'",[151,59873,59874],{"class":503},": mark_safe(escapejs(json.dumps(map_books))),\n",[151,59876,59877,59880,59883,59886],{"class":469,"line":3735},[151,59878,59879],{"class":481},"        'page_count'",[151,59881,59882],{"class":503},":page_count[",[151,59884,59885],{"class":481},"'pages__sum'",[151,59887,18746],{"class":503},[151,59889,59890,59893],{"class":469,"line":3745},[151,59891,59892],{"class":481},"        'form'",[151,59894,59895],{"class":503},":form}\n",[151,59897,59898,59900,59903,59906],{"class":469,"line":3754},[151,59899,17496],{"class":1869},[151,59901,59902],{"class":503}," render(request, ",[151,59904,59905],{"class":481},"'books/books.html'",[151,59907,59908],{"class":503},", context)\n",[11,59910,59911,59912,59915],{},"On the frontend, I passed the ",[30,59913,59914],{},"map_books"," variable to the template like this:",[459,59917,59919],{"className":19459,"code":59918,"language":19461,"meta":464,"style":464},"var map_books = JSON.parse('{{ map_books }}')\n",[30,59920,59921],{"__ignoreMap":464},[151,59922,59923,59925,59928,59930,59932,59934,59936,59938,59941],{"class":469,"line":470},[151,59924,29289],{"class":12347},[151,59926,59927],{"class":503}," map_books ",[151,59929,1876],{"class":1869},[151,59931,29297],{"class":12360},[151,59933,643],{"class":503},[151,59935,29302],{"class":473},[151,59937,12386],{"class":503},[151,59939,59940],{"class":481},"'{{ map_books }}'",[151,59942,3640],{"class":503},[11,59944,59945,59946,59948],{},"Now that I have ",[30,59947,59914],{}," as a JavaScript object, I can simply pass it into the map function that populates data in the Leaflet map. Here's the function that does that:",[459,59950,59952],{"className":19459,"code":59951,"language":19461,"meta":464,"style":464},"function populateMap(data) {\n  for (i in data) {\n    var title = data[i].title, //value searched\n      loc = data[i].loc, //position found\n      url = data[i].url,\n      marker = new L.Marker(new L.latLng(loc), { title: title, icon: bookIcon }) //se property searched\n    marker.bindPopup('\u003Cp>\u003Ca href=\"' + url + '\">' + title + '\u003C/a>\u003C/p>')\n    markersLayer.addLayer(marker)\n  }\n}\n",[30,59953,59954,59968,59981,59997,60010,60020,60054,60088,60099,60103],{"__ignoreMap":464},[151,59955,59956,59959,59962,59964,59966],{"class":469,"line":470},[151,59957,59958],{"class":12347},"function",[151,59960,59961],{"class":473}," populateMap",[151,59963,12386],{"class":503},[151,59965,12355],{"class":15210},[151,59967,23288],{"class":503},[151,59969,59970,59973,59976,59978],{"class":469,"line":488},[151,59971,59972],{"class":1869},"  for",[151,59974,59975],{"class":503}," (i ",[151,59977,16417],{"class":1869},[151,59979,59980],{"class":503}," data) {\n",[151,59982,59983,59986,59989,59991,59994],{"class":469,"line":500},[151,59984,59985],{"class":12347},"    var",[151,59987,59988],{"class":503}," title ",[151,59990,1876],{"class":1869},[151,59992,59993],{"class":503}," data[i].title, ",[151,59995,59996],{"class":1527},"//value searched\n",[151,59998,59999,60002,60004,60007],{"class":469,"line":509},[151,60000,60001],{"class":503},"      loc ",[151,60003,1876],{"class":1869},[151,60005,60006],{"class":503}," data[i].loc, ",[151,60008,60009],{"class":1527},"//position found\n",[151,60011,60012,60015,60017],{"class":469,"line":517},[151,60013,60014],{"class":503},"      url ",[151,60016,1876],{"class":1869},[151,60018,60019],{"class":503}," data[i].url,\n",[151,60021,60022,60025,60027,60029,60032,60034,60037,60039,60041,60043,60045,60048,60051],{"class":469,"line":534},[151,60023,60024],{"class":503},"      marker ",[151,60026,1876],{"class":1869},[151,60028,4236],{"class":1869},[151,60030,60031],{"class":12360}," L",[151,60033,643],{"class":503},[151,60035,60036],{"class":473},"Marker",[151,60038,12386],{"class":503},[151,60040,34199],{"class":1869},[151,60042,60031],{"class":12360},[151,60044,643],{"class":503},[151,60046,60047],{"class":473},"latLng",[151,60049,60050],{"class":503},"(loc), { title: title, icon: bookIcon }) ",[151,60052,60053],{"class":1527},"//se property searched\n",[151,60055,60056,60059,60062,60064,60067,60069,60072,60074,60077,60079,60081,60083,60086],{"class":469,"line":1413},[151,60057,60058],{"class":503},"    marker.",[151,60060,60061],{"class":473},"bindPopup",[151,60063,12386],{"class":503},[151,60065,60066],{"class":481},"'\u003Cp>\u003Ca href=\"'",[151,60068,23378],{"class":1869},[151,60070,60071],{"class":503}," url ",[151,60073,22885],{"class":1869},[151,60075,60076],{"class":481}," '\">'",[151,60078,23378],{"class":1869},[151,60080,59988],{"class":503},[151,60082,22885],{"class":1869},[151,60084,60085],{"class":481}," '\u003C/a>\u003C/p>'",[151,60087,3640],{"class":503},[151,60089,60090,60093,60096],{"class":469,"line":1418},[151,60091,60092],{"class":503},"    markersLayer.",[151,60094,60095],{"class":473},"addLayer",[151,60097,60098],{"class":503},"(marker)\n",[151,60100,60101],{"class":469,"line":2462},[151,60102,19957],{"class":503},[151,60104,60105],{"class":469,"line":2471},[151,60106,6274],{"class":503},[11,60108,60109],{},"This adds our markers to the map, and also passes link and title data to the popup box when a marker is clicked.",[11,60111,60112],{},"Here's the entire script that is used to populate map data:",[11,60114,59441],{},[459,60116,60118],{"className":19811,"code":60117,"language":19813,"meta":464,"style":464},"\u003Cscript>\n  var mymap = new L.Map('mapid', { zoom: 9, center: new L.latLng([40, 13]) }) //set center from first location\n\n  L.tileLayer(\n    'https://api.tiles.mapbox.com/v4/{id}/{z}/{x}/{y}.png?access_token={accessToken}',\n    {\n      attribution:\n        'Map data © \u003Ca href=\"http://openstreetmap.org\">OpenStreetMap\u003C/a> contributors, \u003Ca href=\"http://creativecommons.org/licenses/by-sa/2.0/\">CC-BY-SA\u003C/a>, Imagery © \u003Ca href=\"http://mapbox.com\">Mapbox\u003C/a>',\n      maxZoom: 18,\n      id: 'mapbox.streets',\n      accessToken:\n        'pk.eyJ1IjoiYnJpYW5jYWZmZXkiLCJhIjoiY2pkczJycjl5MmhqbTMzbzQ3bHJuaHA3aiJ9.cvRYYPNjQJVpFjdUcmIHzA',\n    }\n  ).addTo(mymap)\n\n  var bookIcon = L.icon({\n    iconUrl: '/static/images/marker-icon.png',\n    iconSize: [35, 35], // size of the icon\n    iconAnchor: [17, 35], // point of the icon which will correspond to marker's location\n    popupAnchor: [0, -30], // point from which the popup should open relative to the iconAnchor\n  })\n\n  var markersLayer = L.markerClusterGroup()\n\n  mymap.addLayer(markersLayer)\n\n  var controlSearch = new L.Control.Search({\n    position: 'topright',\n    layer: markersLayer,\n    initial: false,\n    zoom: 12,\n    marker: false,\n  })\n\n  mymap.addControl(controlSearch)\n\n  function populateMap(data) {\n    for (i in data) {\n      var title = data[i].title, //value searched\n        loc = data[i].loc, //position found\n        url = data[i].url,\n        marker = new L.Marker(new L.latLng(loc), {\n          title: title,\n          icon: bookIcon,\n        }) //se property searched\n      marker.bindPopup('\u003Cp>\u003Ca href=\"' + url + '\">' + title + '\u003C/a>\u003C/p>')\n      markersLayer.addLayer(marker)\n    }\n  }\n\n  // use context variable instead of making AJAX call\n  var map_books = JSON.parse('{{ map_books }}')\n  var new_lat = map_books[0].loc[0]\n  var new_lon = map_books[0].loc[1]\n  mymap.setView([0, 0], 2)\n  populateMap(map_books)\n\u003C/script>\n",[30,60119,60120,60128,60183,60187,60198,60205,60209,60214,60221,60230,60238,60243,60250,60254,60263,60267,60285,60295,60312,60328,60346,60351,60355,60373,60377,60387,60391,60412,60422,60427,60436,60445,60454,60458,60462,60472,60476,60489,60499,60512,60523,60532,60560,60565,60570,60577,60606,60615,60619,60623,60627,60632,60652,60673,60692,60713,60721],{"__ignoreMap":464},[151,60121,60122,60124,60126],{"class":469,"line":470},[151,60123,3613],{"class":503},[151,60125,19822],{"class":14368},[151,60127,3742],{"class":503},[151,60129,60130,60133,60136,60138,60140,60142,60144,60147,60149,60152,60155,60157,60160,60162,60164,60166,60168,60171,60173,60175,60177,60180],{"class":469,"line":488},[151,60131,60132],{"class":12347},"  var",[151,60134,60135],{"class":503}," mymap ",[151,60137,1876],{"class":1869},[151,60139,4236],{"class":1869},[151,60141,60031],{"class":12360},[151,60143,643],{"class":503},[151,60145,60146],{"class":473},"Map",[151,60148,12386],{"class":503},[151,60150,60151],{"class":481},"'mapid'",[151,60153,60154],{"class":503},", { zoom: ",[151,60156,7918],{"class":477},[151,60158,60159],{"class":503},", center: ",[151,60161,34199],{"class":1869},[151,60163,60031],{"class":12360},[151,60165,643],{"class":503},[151,60167,60047],{"class":473},[151,60169,60170],{"class":503},"([",[151,60172,44365],{"class":477},[151,60174,106],{"class":503},[151,60176,42327],{"class":477},[151,60178,60179],{"class":503},"]) }) ",[151,60181,60182],{"class":1527},"//set center from first location\n",[151,60184,60185],{"class":469,"line":500},[151,60186,1090],{"emptyLinePlaceholder":609},[151,60188,60189,60192,60194,60196],{"class":469,"line":509},[151,60190,60191],{"class":12360},"  L",[151,60193,643],{"class":503},[151,60195,59456],{"class":473},[151,60197,15410],{"class":503},[151,60199,60200,60203],{"class":469,"line":517},[151,60201,60202],{"class":481},"    'https://api.tiles.mapbox.com/v4/{id}/{z}/{x}/{y}.png?access_token={accessToken}'",[151,60204,9417],{"class":503},[151,60206,60207],{"class":469,"line":534},[151,60208,9404],{"class":503},[151,60210,60211],{"class":469,"line":1413},[151,60212,60213],{"class":503},"      attribution:\n",[151,60215,60216,60219],{"class":469,"line":1418},[151,60217,60218],{"class":481},"        'Map data © \u003Ca href=\"http://openstreetmap.org\">OpenStreetMap\u003C/a> contributors, \u003Ca href=\"http://creativecommons.org/licenses/by-sa/2.0/\">CC-BY-SA\u003C/a>, Imagery © \u003Ca href=\"http://mapbox.com\">Mapbox\u003C/a>'",[151,60220,9417],{"class":503},[151,60222,60223,60226,60228],{"class":469,"line":2462},[151,60224,60225],{"class":503},"      maxZoom: ",[151,60227,7696],{"class":477},[151,60229,9417],{"class":503},[151,60231,60232,60234,60236],{"class":469,"line":2471},[151,60233,33785],{"class":503},[151,60235,59499],{"class":481},[151,60237,9417],{"class":503},[151,60239,60240],{"class":469,"line":2480},[151,60241,60242],{"class":503},"      accessToken:\n",[151,60244,60245,60248],{"class":469,"line":2489},[151,60246,60247],{"class":481},"        'pk.eyJ1IjoiYnJpYW5jYWZmZXkiLCJhIjoiY2pkczJycjl5MmhqbTMzbzQ3bHJuaHA3aiJ9.cvRYYPNjQJVpFjdUcmIHzA'",[151,60249,9417],{"class":503},[151,60251,60252],{"class":469,"line":2497},[151,60253,9461],{"class":503},[151,60255,60256,60259,60261],{"class":469,"line":3140},[151,60257,60258],{"class":503},"  ).",[151,60260,59522],{"class":473},[151,60262,59525],{"class":503},[151,60264,60265],{"class":469,"line":3149},[151,60266,1090],{"emptyLinePlaceholder":609},[151,60268,60269,60271,60274,60276,60278,60280,60283],{"class":469,"line":3158},[151,60270,60132],{"class":12347},[151,60272,60273],{"class":503}," bookIcon ",[151,60275,1876],{"class":1869},[151,60277,60031],{"class":12360},[151,60279,643],{"class":503},[151,60281,60282],{"class":473},"icon",[151,60284,19476],{"class":503},[151,60286,60287,60290,60293],{"class":469,"line":3167},[151,60288,60289],{"class":503},"    iconUrl: ",[151,60291,60292],{"class":481},"'/static/images/marker-icon.png'",[151,60294,9417],{"class":503},[151,60296,60297,60300,60302,60304,60306,60309],{"class":469,"line":3175},[151,60298,60299],{"class":503},"    iconSize: [",[151,60301,41984],{"class":477},[151,60303,106],{"class":503},[151,60305,41984],{"class":477},[151,60307,60308],{"class":503},"], ",[151,60310,60311],{"class":1527},"// size of the icon\n",[151,60313,60314,60317,60319,60321,60323,60325],{"class":469,"line":3184},[151,60315,60316],{"class":503},"    iconAnchor: [",[151,60318,42293],{"class":477},[151,60320,106],{"class":503},[151,60322,41984],{"class":477},[151,60324,60308],{"class":503},[151,60326,60327],{"class":1527},"// point of the icon which will correspond to marker's location\n",[151,60329,60330,60333,60335,60337,60339,60341,60343],{"class":469,"line":3193},[151,60331,60332],{"class":503},"    popupAnchor: [",[151,60334,9181],{"class":477},[151,60336,106],{"class":503},[151,60338,12445],{"class":1869},[151,60340,42017],{"class":477},[151,60342,60308],{"class":503},[151,60344,60345],{"class":1527},"// point from which the popup should open relative to the iconAnchor\n",[151,60347,60348],{"class":469,"line":3720},[151,60349,60350],{"class":503},"  })\n",[151,60352,60353],{"class":469,"line":3729},[151,60354,1090],{"emptyLinePlaceholder":609},[151,60356,60357,60359,60362,60364,60366,60368,60371],{"class":469,"line":3735},[151,60358,60132],{"class":12347},[151,60360,60361],{"class":503}," markersLayer ",[151,60363,1876],{"class":1869},[151,60365,60031],{"class":12360},[151,60367,643],{"class":503},[151,60369,60370],{"class":473},"markerClusterGroup",[151,60372,12461],{"class":503},[151,60374,60375],{"class":469,"line":3745},[151,60376,1090],{"emptyLinePlaceholder":609},[151,60378,60379,60382,60384],{"class":469,"line":3754},[151,60380,60381],{"class":503},"  mymap.",[151,60383,60095],{"class":473},[151,60385,60386],{"class":503},"(markersLayer)\n",[151,60388,60389],{"class":469,"line":3760},[151,60390,1090],{"emptyLinePlaceholder":609},[151,60392,60393,60395,60398,60400,60402,60404,60407,60410],{"class":469,"line":3773},[151,60394,60132],{"class":12347},[151,60396,60397],{"class":503}," controlSearch ",[151,60399,1876],{"class":1869},[151,60401,4236],{"class":1869},[151,60403,60031],{"class":12360},[151,60405,60406],{"class":503},".Control.",[151,60408,60409],{"class":473},"Search",[151,60411,19476],{"class":503},[151,60413,60414,60417,60420],{"class":469,"line":3782},[151,60415,60416],{"class":503},"    position: ",[151,60418,60419],{"class":481},"'topright'",[151,60421,9417],{"class":503},[151,60423,60424],{"class":469,"line":3791},[151,60425,60426],{"class":503},"    layer: markersLayer,\n",[151,60428,60429,60432,60434],{"class":469,"line":3803},[151,60430,60431],{"class":503},"    initial: ",[151,60433,9522],{"class":477},[151,60435,9417],{"class":503},[151,60437,60438,60441,60443],{"class":469,"line":3811},[151,60439,60440],{"class":503},"    zoom: ",[151,60442,42360],{"class":477},[151,60444,9417],{"class":503},[151,60446,60447,60450,60452],{"class":469,"line":3820},[151,60448,60449],{"class":503},"    marker: ",[151,60451,9522],{"class":477},[151,60453,9417],{"class":503},[151,60455,60456],{"class":469,"line":7084},[151,60457,60350],{"class":503},[151,60459,60460],{"class":469,"line":7148},[151,60461,1090],{"emptyLinePlaceholder":609},[151,60463,60464,60466,60469],{"class":469,"line":7211},[151,60465,60381],{"class":503},[151,60467,60468],{"class":473},"addControl",[151,60470,60471],{"class":503},"(controlSearch)\n",[151,60473,60474],{"class":469,"line":7273},[151,60475,1090],{"emptyLinePlaceholder":609},[151,60477,60478,60481,60483,60485,60487],{"class":469,"line":7335},[151,60479,60480],{"class":12347},"  function",[151,60482,59961],{"class":473},[151,60484,12386],{"class":503},[151,60486,12355],{"class":15210},[151,60488,23288],{"class":503},[151,60490,60491,60493,60495,60497],{"class":469,"line":7398},[151,60492,16411],{"class":1869},[151,60494,59975],{"class":503},[151,60496,16417],{"class":1869},[151,60498,59980],{"class":503},[151,60500,60501,60504,60506,60508,60510],{"class":469,"line":7462},[151,60502,60503],{"class":12347},"      var",[151,60505,59988],{"class":503},[151,60507,1876],{"class":1869},[151,60509,59993],{"class":503},[151,60511,59996],{"class":1527},[151,60513,60514,60517,60519,60521],{"class":469,"line":7467},[151,60515,60516],{"class":503},"        loc ",[151,60518,1876],{"class":1869},[151,60520,60006],{"class":503},[151,60522,60009],{"class":1527},[151,60524,60525,60528,60530],{"class":469,"line":7532},[151,60526,60527],{"class":503},"        url ",[151,60529,1876],{"class":1869},[151,60531,60019],{"class":503},[151,60533,60534,60537,60539,60541,60543,60545,60547,60549,60551,60553,60555,60557],{"class":469,"line":7537},[151,60535,60536],{"class":503},"        marker ",[151,60538,1876],{"class":1869},[151,60540,4236],{"class":1869},[151,60542,60031],{"class":12360},[151,60544,643],{"class":503},[151,60546,60036],{"class":473},[151,60548,12386],{"class":503},[151,60550,34199],{"class":1869},[151,60552,60031],{"class":12360},[151,60554,643],{"class":503},[151,60556,60047],{"class":473},[151,60558,60559],{"class":503},"(loc), {\n",[151,60561,60562],{"class":469,"line":7603},[151,60563,60564],{"class":503},"          title: title,\n",[151,60566,60567],{"class":469,"line":7608},[151,60568,60569],{"class":503},"          icon: bookIcon,\n",[151,60571,60572,60575],{"class":469,"line":7673},[151,60573,60574],{"class":503},"        }) ",[151,60576,60053],{"class":1527},[151,60578,60579,60582,60584,60586,60588,60590,60592,60594,60596,60598,60600,60602,60604],{"class":469,"line":7678},[151,60580,60581],{"class":503},"      marker.",[151,60583,60061],{"class":473},[151,60585,12386],{"class":503},[151,60587,60066],{"class":481},[151,60589,23378],{"class":1869},[151,60591,60071],{"class":503},[151,60593,22885],{"class":1869},[151,60595,60076],{"class":481},[151,60597,23378],{"class":1869},[151,60599,59988],{"class":503},[151,60601,22885],{"class":1869},[151,60603,60085],{"class":481},[151,60605,3640],{"class":503},[151,60607,60608,60611,60613],{"class":469,"line":7708},[151,60609,60610],{"class":503},"      markersLayer.",[151,60612,60095],{"class":473},[151,60614,60098],{"class":503},[151,60616,60617],{"class":469,"line":7713},[151,60618,9461],{"class":503},[151,60620,60621],{"class":469,"line":7746},[151,60622,19957],{"class":503},[151,60624,60625],{"class":469,"line":7751},[151,60626,1090],{"emptyLinePlaceholder":609},[151,60628,60629],{"class":469,"line":7816},[151,60630,60631],{"class":1527},"  // use context variable instead of making AJAX call\n",[151,60633,60634,60636,60638,60640,60642,60644,60646,60648,60650],{"class":469,"line":7821},[151,60635,60132],{"class":12347},[151,60637,59927],{"class":503},[151,60639,1876],{"class":1869},[151,60641,29297],{"class":12360},[151,60643,643],{"class":503},[151,60645,29302],{"class":473},[151,60647,12386],{"class":503},[151,60649,59940],{"class":481},[151,60651,3640],{"class":503},[151,60653,60654,60656,60659,60661,60664,60666,60669,60671],{"class":469,"line":7847},[151,60655,60132],{"class":12347},[151,60657,60658],{"class":503}," new_lat ",[151,60660,1876],{"class":1869},[151,60662,60663],{"class":503}," map_books[",[151,60665,9181],{"class":477},[151,60667,60668],{"class":503},"].loc[",[151,60670,9181],{"class":477},[151,60672,3691],{"class":503},[151,60674,60675,60677,60680,60682,60684,60686,60688,60690],{"class":469,"line":7852},[151,60676,60132],{"class":12347},[151,60678,60679],{"class":503}," new_lon ",[151,60681,1876],{"class":1869},[151,60683,60663],{"class":503},[151,60685,9181],{"class":477},[151,60687,60668],{"class":503},[151,60689,6760],{"class":477},[151,60691,3691],{"class":503},[151,60693,60694,60696,60699,60701,60703,60705,60707,60709,60711],{"class":469,"line":7887},[151,60695,60381],{"class":503},[151,60697,60698],{"class":473},"setView",[151,60700,60170],{"class":503},[151,60702,9181],{"class":477},[151,60704,106],{"class":503},[151,60706,9181],{"class":477},[151,60708,60308],{"class":503},[151,60710,6619],{"class":477},[151,60712,3640],{"class":503},[151,60714,60715,60718],{"class":469,"line":7892},[151,60716,60717],{"class":473},"  populateMap",[151,60719,60720],{"class":503},"(map_books)\n",[151,60722,60723,60725,60727],{"class":469,"line":7924},[151,60724,19966],{"class":503},[151,60726,19822],{"class":14368},[151,60728,3742],{"class":503},[11,60730,59528],{},[56,60732,60734],{"id":60733},"login-redirect","Login Redirect",[11,60736,60737,60738,60741],{},"I wanted to include a small note on a simple issue that I previously had trouble with, which is using the ",[30,60739,60740],{},"next"," parameter to do a redirect to a page that an unauthenticated user will be redirected to after they successfully login.",[11,60743,60744],{},"Here's the login view:",[459,60746,60748],{"className":13136,"code":60747,"language":12886,"meta":464,"style":464},"def login_view(request):\n    next_redirect = request.GET.get('next')\n    form = UserLoginForm(request.POST or None)\n    if form.is_valid():\n        next_redirect = request.POST.get('next')\n        username = form.cleaned_data.get('username')\n        password = form.cleaned_data.get('password')\n        user = authenticate(username=username, password=password)\n        login(request, user)\n        print(next_redirect)\n        if next_redirect != 'None':\n            return redirect(next_redirect)\n        return redirect('books:all')\n    context = { 'form':form, 'next':next_redirect }\n    return render(request, 'accounts/login_form.html', context)\n",[30,60749,60750,60763,60782,60799,60806,60823,60838,60851,60876,60881,60888,60902,60909,60921,60940],{"__ignoreMap":464},[151,60751,60752,60754,60757,60759,60761],{"class":469,"line":470},[151,60753,16925],{"class":12347},[151,60755,60756],{"class":473}," login_view",[151,60758,12386],{"class":503},[151,60760,59686],{"class":15232},[151,60762,15264],{"class":503},[151,60764,60765,60768,60770,60772,60774,60777,60780],{"class":469,"line":488},[151,60766,60767],{"class":503},"    next_redirect ",[151,60769,1876],{"class":1869},[151,60771,40684],{"class":503},[151,60773,47765],{"class":477},[151,60775,60776],{"class":503},".get(",[151,60778,60779],{"class":481},"'next'",[151,60781,3640],{"class":503},[151,60783,60784,60786,60788,60791,60793,60795,60797],{"class":469,"line":500},[151,60785,59716],{"class":503},[151,60787,1876],{"class":1869},[151,60789,60790],{"class":503}," UserLoginForm(request.",[151,60792,36573],{"class":477},[151,60794,2161],{"class":1869},[151,60796,40451],{"class":477},[151,60798,3640],{"class":503},[151,60800,60801,60803],{"class":469,"line":509},[151,60802,23327],{"class":1869},[151,60804,60805],{"class":503}," form.is_valid():\n",[151,60807,60808,60811,60813,60815,60817,60819,60821],{"class":469,"line":517},[151,60809,60810],{"class":503},"        next_redirect ",[151,60812,1876],{"class":1869},[151,60814,40684],{"class":503},[151,60816,36573],{"class":477},[151,60818,60776],{"class":503},[151,60820,60779],{"class":481},[151,60822,3640],{"class":503},[151,60824,60825,60828,60830,60833,60836],{"class":469,"line":534},[151,60826,60827],{"class":503},"        username ",[151,60829,1876],{"class":1869},[151,60831,60832],{"class":503}," form.cleaned_data.get(",[151,60834,60835],{"class":481},"'username'",[151,60837,3640],{"class":503},[151,60839,60840,60843,60845,60847,60849],{"class":469,"line":1413},[151,60841,60842],{"class":503},"        password ",[151,60844,1876],{"class":1869},[151,60846,60832],{"class":503},[151,60848,27158],{"class":481},[151,60850,3640],{"class":503},[151,60852,60853,60856,60858,60861,60864,60866,60869,60871,60873],{"class":469,"line":1418},[151,60854,60855],{"class":503},"        user ",[151,60857,1876],{"class":1869},[151,60859,60860],{"class":503}," authenticate(",[151,60862,60863],{"class":15210},"username",[151,60865,1876],{"class":1869},[151,60867,60868],{"class":503},"username, ",[151,60870,27213],{"class":15210},[151,60872,1876],{"class":1869},[151,60874,60875],{"class":503},"password)\n",[151,60877,60878],{"class":469,"line":2462},[151,60879,60880],{"class":503},"        login(request, user)\n",[151,60882,60883,60885],{"class":469,"line":2471},[151,60884,18355],{"class":2226},[151,60886,60887],{"class":503},"(next_redirect)\n",[151,60889,60890,60892,60895,60897,60900],{"class":469,"line":2480},[151,60891,23357],{"class":1869},[151,60893,60894],{"class":503}," next_redirect ",[151,60896,58602],{"class":1869},[151,60898,60899],{"class":481}," 'None'",[151,60901,14372],{"class":503},[151,60903,60904,60906],{"class":469,"line":2489},[151,60905,15386],{"class":1869},[151,60907,60908],{"class":503}," redirect(next_redirect)\n",[151,60910,60911,60913,60916,60919],{"class":469,"line":2497},[151,60912,16833],{"class":1869},[151,60914,60915],{"class":503}," redirect(",[151,60917,60918],{"class":481},"'books:all'",[151,60920,3640],{"class":503},[151,60922,60923,60925,60927,60929,60932,60935,60937],{"class":469,"line":3140},[151,60924,59844],{"class":503},[151,60926,1876],{"class":1869},[151,60928,12351],{"class":503},[151,60930,60931],{"class":481},"'form'",[151,60933,60934],{"class":503},":form, ",[151,60936,60779],{"class":481},[151,60938,60939],{"class":503},":next_redirect }\n",[151,60941,60942,60944,60946,60949],{"class":469,"line":3149},[151,60943,17496],{"class":1869},[151,60945,59902],{"class":503},[151,60947,60948],{"class":481},"'accounts/login_form.html'",[151,60950,59908],{"class":503},[11,60952,60953,60954,60956],{},"Notice that I pass ",[30,60955,60740],{}," as a context variable. I use this in the login form here:",[11,60958,59441],{},[459,60960,60962],{"className":19811,"code":60961,"language":19813,"meta":464,"style":464},"\u003Cform method=\"POST\" action=\".\">\n  {% csrf_token %} {{ form | crispy }}\n  \u003Cinput type=\"hidden\" value=\"{{ next }}\" name=\"next\" />\n  \u003Cdiv class=\"login-center\">\n    \u003Cinput class=\"btn btn-success login-center\" type=\"submit\" value=\"Login\" />\n  \u003C/div>\n  \u003Cbr />\n  or \u003Ca href=\"{% url 'accounts:register' %}\">create an account\u003C/a>\n\u003C/form>\n",[30,60963,60964,60987,60992,61023,61038,61067,61075,61083,61105],{"__ignoreMap":464},[151,60965,60966,60968,60970,60973,60975,60978,60981,60983,60985],{"class":469,"line":470},[151,60967,3613],{"class":503},[151,60969,48710],{"class":14368},[151,60971,60972],{"class":473}," method",[151,60974,1876],{"class":503},[151,60976,60977],{"class":481},"\"POST\"",[151,60979,60980],{"class":473}," action",[151,60982,1876],{"class":503},[151,60984,44221],{"class":481},[151,60986,3742],{"class":503},[151,60988,60989],{"class":469,"line":488},[151,60990,60991],{"class":503},"  {% csrf_token %} {{ form | crispy }}\n",[151,60993,60994,60996,60998,61001,61003,61006,61008,61010,61013,61016,61018,61021],{"class":469,"line":500},[151,60995,33991],{"class":503},[151,60997,29860],{"class":14368},[151,60999,61000],{"class":473}," type",[151,61002,1876],{"class":503},[151,61004,61005],{"class":481},"\"hidden\"",[151,61007,2186],{"class":473},[151,61009,1876],{"class":503},[151,61011,61012],{"class":481},"\"{{ next }}\"",[151,61014,61015],{"class":473}," name",[151,61017,1876],{"class":503},[151,61019,61020],{"class":481},"\"next\"",[151,61022,34675],{"class":503},[151,61024,61025,61027,61029,61031,61033,61036],{"class":469,"line":509},[151,61026,33991],{"class":503},[151,61028,23950],{"class":14368},[151,61030,48323],{"class":473},[151,61032,1876],{"class":503},[151,61034,61035],{"class":481},"\"login-center\"",[151,61037,3742],{"class":503},[151,61039,61040,61042,61044,61046,61048,61051,61053,61055,61058,61060,61062,61065],{"class":469,"line":517},[151,61041,34669],{"class":503},[151,61043,29860],{"class":14368},[151,61045,48323],{"class":473},[151,61047,1876],{"class":503},[151,61049,61050],{"class":481},"\"btn btn-success login-center\"",[151,61052,61000],{"class":473},[151,61054,1876],{"class":503},[151,61056,61057],{"class":481},"\"submit\"",[151,61059,2186],{"class":473},[151,61061,1876],{"class":503},[151,61063,61064],{"class":481},"\"Login\"",[151,61066,34675],{"class":503},[151,61068,61069,61071,61073],{"class":469,"line":534},[151,61070,34741],{"class":503},[151,61072,23950],{"class":14368},[151,61074,3742],{"class":503},[151,61076,61077,61079,61081],{"class":469,"line":1413},[151,61078,33991],{"class":503},[151,61080,1205],{"class":14368},[151,61082,34675],{"class":503},[151,61084,61085,61088,61090,61093,61095,61098,61101,61103],{"class":469,"line":1418},[151,61086,61087],{"class":503},"  or \u003C",[151,61089,20],{"class":14368},[151,61091,61092],{"class":473}," href",[151,61094,1876],{"class":503},[151,61096,61097],{"class":481},"\"{% url 'accounts:register' %}\"",[151,61099,61100],{"class":503},">create an account\u003C/",[151,61102,20],{"class":14368},[151,61104,3742],{"class":503},[151,61106,61107,61109,61111],{"class":469,"line":2462},[151,61108,19966],{"class":503},[151,61110,48710],{"class":14368},[151,61112,3742],{"class":503},[11,61114,59528],{},[11,61116,61117,61118,61120,61121,61123,61124,61126],{},"By passing this as a hidden value to the form, when a ",[30,61119,36573],{}," request is made, we are redirected to the value of ",[30,61122,60740],{}," if it is not ",[30,61125,15437],{},", or if there is no redirect.",[11,61128,61129,61130,61133,61134,61137],{},"When a ",[30,61131,61132],{},"@login_required"," decorator is used, and we want to access a url such as ",[30,61135,61136],{},"/books/add",", we are redirected to a url that looks like this:",[459,61139,61141],{"className":19811,"code":61140,"language":19813,"meta":464,"style":464},"http://localhost:8000/accounts/login/?next=/books/new/\n",[30,61142,61143],{"__ignoreMap":464},[151,61144,61145],{"class":469,"line":470},[151,61146,61140],{"class":503},[11,61148,61149,61150,61152,61153,61155],{},"In this way we can pass the next parameter from the ",[30,61151,47765],{}," request to the ",[30,61154,36573],{}," request that is made when the login view is hit.",[56,61157,61159],{"id":61158},"authors","Authors",[11,61161,61162,61163,61166],{},"For illustration purposes, I included an ",[30,61164,61165],{},"Author"," model. Since a book can have zero, one or many Authors, I used a many-to-many relationship to link authors to books. Here is how we do this in our Book model:",[459,61168,61170],{"className":13136,"code":61169,"language":12886,"meta":464,"style":464},"class Book(models.Model):\n    [...other fields...]\n    authors = models.ManyToManyField('authors.Author')\n",[30,61171,61172,61188,61201],{"__ignoreMap":464},[151,61173,61174,61176,61178,61180,61182,61184,61186],{"class":469,"line":470},[151,61175,16519],{"class":12347},[151,61177,59556],{"class":15254},[151,61179,12386],{"class":503},[151,61181,59561],{"class":15260},[151,61183,643],{"class":503},[151,61185,1612],{"class":15260},[151,61187,15264],{"class":503},[151,61189,61190,61192,61194,61197,61199],{"class":469,"line":488},[151,61191,33774],{"class":503},[151,61193,27455],{"class":477},[151,61195,61196],{"class":503},"other fields",[151,61198,27455],{"class":477},[151,61200,3691],{"class":503},[151,61202,61203,61206,61208,61211,61214],{"class":469,"line":500},[151,61204,61205],{"class":503},"    authors ",[151,61207,1876],{"class":1869},[151,61209,61210],{"class":503}," models.ManyToManyField(",[151,61212,61213],{"class":481},"'authors.Author'",[151,61215,3640],{"class":503},[11,61217,61218,61219,61221,61222,61224],{},"This dot notation with ",[30,61220,61213],{}," is important. Previously I would have just imported the ",[30,61223,61165],{},"model and then referenced that in the ManyToManyField. However, this can lead to circular import errors which are hard to debug unless you know what a circular import is.",[11,61226,61227,61228,61231,61232,61234],{},"Now let's look at how to find all ",[30,61229,61230],{},"Books"," by a certain author, as well as all ",[30,61233,61159],{}," of a given book. Uing the Django shell, we can do the following:",[459,61236,61238],{"className":13136,"code":61237,"language":12886,"meta":464,"style":464},"from authors.models import Author\nfrom books.models import Book\na = Author.objects.all().first()\na.book_set.values()\n\u003CQuerySet [{'id': 583, 'title': 'None Or Other', 'lat': Decimal('-118.035345'), 'lon': Decimal('34.139729'), 'pages': 1881, 'publish_date': datetime.date(2018, 3, 18), 'website': 'https://www.fleming-mitchell.com/', 'synopsis': 'Option ever large throw dinner worker ahead realize clearly congress as smile size spend expert chair well. Brother item win follow hope coach garden later arrive who if ago voice analysis simply reflect. Amount under data drug kind book fish still information president minute stage dog. Focus full green society parent door according my management sell arrive only send international tonight. Player character financial detail oil check bring pressure possible former. Learn politics compare position large loss exactly probably approach physical international machine as model. ', 'slug': 'none-or-other', 'status': True}, {'id': 601, 'title': 'Teacher Them Step', 'lat': Decimal('-80.268357'), 'lon': Decimal('26.661763'), 'pages': 206, 'publish_date': datetime.date(2018, 2, 27), 'website': 'http://www.rodriguez.net/', 'synopsis': 'Drug fact behavior environment green try account where training brother building particular window even reach. Pressure lawyer dog world thought near institution we get force market can guy receive matter structure research foot of small. Professor production very this practice car wind wish relationship after follow professor news card concern media property have. According majority owner go mention reach store computer project kitchen group quality present several another time school and will. Before during central artist page only health region happen share traditional section well out human. Mention quite short race only education heavy book up recent official here spring oil buy which language information practice. Time mother better peace girl defense rock mr never feeling tax city stock bar tv others right conference skin. ', 'slug': 'teacher-them-step', 'status': True}]>\n",[30,61239,61240,61252,61264,61274,61279],{"__ignoreMap":464},[151,61241,61242,61244,61247,61249],{"class":469,"line":470},[151,61243,16853],{"class":1869},[151,61245,61246],{"class":503}," authors.models ",[151,61248,16859],{"class":1869},[151,61250,61251],{"class":503}," Author\n",[151,61253,61254,61256,61259,61261],{"class":469,"line":488},[151,61255,16853],{"class":1869},[151,61257,61258],{"class":503}," books.models ",[151,61260,16859],{"class":1869},[151,61262,61263],{"class":503}," Book\n",[151,61265,61266,61269,61271],{"class":469,"line":500},[151,61267,61268],{"class":503},"a ",[151,61270,1876],{"class":1869},[151,61272,61273],{"class":503}," Author.objects.all().first()\n",[151,61275,61276],{"class":469,"line":509},[151,61277,61278],{"class":503},"a.book_set.values()\n",[151,61280,61281,61283,61286,61289,61291,61294,61296,61299,61301,61304,61306,61309,61312,61315,61317,61320,61322,61325,61327,61329,61331,61334,61336,61339,61342,61345,61347,61349,61351,61353,61355,61358,61360,61363,61365,61368,61370,61373,61375,61378,61380,61383,61385,61388,61390,61392,61395,61397,61399,61402,61404,61406,61408,61411,61413,61415,61417,61420,61422,61424,61426,61429,61431,61433,61435,61438,61440,61442,61444,61446,61448,61450,61452,61455,61457,61459,61461,61464,61466,61468,61470,61473,61475,61477,61479,61482,61484,61486,61488,61490,61493],{"class":469,"line":517},[151,61282,3613],{"class":1869},[151,61284,61285],{"class":503},"QuerySet [{",[151,61287,61288],{"class":481},"'id'",[151,61290,6208],{"class":503},[151,61292,61293],{"class":477},"583",[151,61295,106],{"class":503},[151,61297,61298],{"class":481},"'title'",[151,61300,6208],{"class":503},[151,61302,61303],{"class":481},"'None Or Other'",[151,61305,106],{"class":503},[151,61307,61308],{"class":481},"'lat'",[151,61310,61311],{"class":503},": Decimal(",[151,61313,61314],{"class":481},"'-118.035345'",[151,61316,24817],{"class":503},[151,61318,61319],{"class":481},"'lon'",[151,61321,61311],{"class":503},[151,61323,61324],{"class":481},"'34.139729'",[151,61326,24817],{"class":503},[151,61328,59770],{"class":481},[151,61330,6208],{"class":503},[151,61332,61333],{"class":477},"1881",[151,61335,106],{"class":503},[151,61337,61338],{"class":481},"'publish_date'",[151,61340,61341],{"class":503},": datetime.date(",[151,61343,61344],{"class":477},"2018",[151,61346,106],{"class":503},[151,61348,6557],{"class":477},[151,61350,106],{"class":503},[151,61352,7696],{"class":477},[151,61354,24817],{"class":503},[151,61356,61357],{"class":481},"'website'",[151,61359,6208],{"class":503},[151,61361,61362],{"class":481},"'https://www.fleming-mitchell.com/'",[151,61364,106],{"class":503},[151,61366,61367],{"class":481},"'synopsis'",[151,61369,6208],{"class":503},[151,61371,61372],{"class":481},"'Option ever large throw dinner worker ahead realize clearly congress as smile size spend expert chair well. Brother item win follow hope coach garden later arrive who if ago voice analysis simply reflect. Amount under data drug kind book fish still information president minute stage dog. Focus full green society parent door according my management sell arrive only send international tonight. Player character financial detail oil check bring pressure possible former. Learn politics compare position large loss exactly probably approach physical international machine as model. '",[151,61374,106],{"class":503},[151,61376,61377],{"class":481},"'slug'",[151,61379,6208],{"class":503},[151,61381,61382],{"class":481},"'none-or-other'",[151,61384,106],{"class":503},[151,61386,61387],{"class":481},"'status'",[151,61389,6208],{"class":503},[151,61391,36962],{"class":477},[151,61393,61394],{"class":503},"}, {",[151,61396,61288],{"class":481},[151,61398,6208],{"class":503},[151,61400,61401],{"class":477},"601",[151,61403,106],{"class":503},[151,61405,61298],{"class":481},[151,61407,6208],{"class":503},[151,61409,61410],{"class":481},"'Teacher Them Step'",[151,61412,106],{"class":503},[151,61414,61308],{"class":481},[151,61416,61311],{"class":503},[151,61418,61419],{"class":481},"'-80.268357'",[151,61421,24817],{"class":503},[151,61423,61319],{"class":481},[151,61425,61311],{"class":503},[151,61427,61428],{"class":481},"'26.661763'",[151,61430,24817],{"class":503},[151,61432,59770],{"class":481},[151,61434,6208],{"class":503},[151,61436,61437],{"class":477},"206",[151,61439,106],{"class":503},[151,61441,61338],{"class":481},[151,61443,61341],{"class":503},[151,61445,61344],{"class":477},[151,61447,106],{"class":503},[151,61449,6619],{"class":477},[151,61451,106],{"class":503},[151,61453,61454],{"class":477},"27",[151,61456,24817],{"class":503},[151,61458,61357],{"class":481},[151,61460,6208],{"class":503},[151,61462,61463],{"class":481},"'http://www.rodriguez.net/'",[151,61465,106],{"class":503},[151,61467,61367],{"class":481},[151,61469,6208],{"class":503},[151,61471,61472],{"class":481},"'Drug fact behavior environment green try account where training brother building particular window even reach. Pressure lawyer dog world thought near institution we get force market can guy receive matter structure research foot of small. Professor production very this practice car wind wish relationship after follow professor news card concern media property have. According majority owner go mention reach store computer project kitchen group quality present several another time school and will. Before during central artist page only health region happen share traditional section well out human. Mention quite short race only education heavy book up recent official here spring oil buy which language information practice. Time mother better peace girl defense rock mr never feeling tax city stock bar tv others right conference skin. '",[151,61474,106],{"class":503},[151,61476,61377],{"class":481},[151,61478,6208],{"class":503},[151,61480,61481],{"class":481},"'teacher-them-step'",[151,61483,106],{"class":503},[151,61485,61387],{"class":481},[151,61487,6208],{"class":503},[151,61489,36962],{"class":477},[151,61491,61492],{"class":503},"}]",[151,61494,3742],{"class":1869},[11,61496,61497],{},"This gave us two books by an author.",[11,61499,61500,61501,61504],{},"Now, let's look at how to find all authors of a given book. For this example, we can get authors directly in the template from a book object. This example come from the ",[30,61502,61503],{},"book_detail"," template:",[11,61506,59441],{},[459,61508,61510],{"className":19811,"code":61509,"language":19813,"meta":464,"style":464},"\u003Ch3>\n  Authors: {% for author in book.authors.all %}\n  \u003Ca href=\"{% url 'authors:author_detail' id=author.id %}\"\n    >{{ author.full_name }}\u003C/a\n  >\n  {% endfor %}\n\u003C/h3>\n",[30,61511,61512,61520,61525,61538,61546,61551,61556],{"__ignoreMap":464},[151,61513,61514,61516,61518],{"class":469,"line":470},[151,61515,3613],{"class":503},[151,61517,736],{"class":14368},[151,61519,3742],{"class":503},[151,61521,61522],{"class":469,"line":488},[151,61523,61524],{"class":503},"  Authors: {% for author in book.authors.all %}\n",[151,61526,61527,61529,61531,61533,61535],{"class":469,"line":500},[151,61528,33991],{"class":503},[151,61530,20],{"class":14368},[151,61532,61092],{"class":473},[151,61534,1876],{"class":503},[151,61536,61537],{"class":481},"\"{% url 'authors:author_detail' id=author.id %}\"\n",[151,61539,61540,61543],{"class":469,"line":509},[151,61541,61542],{"class":503},"    >{{ author.full_name }}\u003C/",[151,61544,61545],{"class":14368},"a\n",[151,61547,61548],{"class":469,"line":517},[151,61549,61550],{"class":503},"  >\n",[151,61552,61553],{"class":469,"line":534},[151,61554,61555],{"class":503},"  {% endfor %}\n",[151,61557,61558,61560,61562],{"class":469,"line":1413},[151,61559,19966],{"class":503},[151,61561,736],{"class":14368},[151,61563,3742],{"class":503},[11,61565,59528],{},[11,61567,61568,61569,61572],{},"We could just as well access ",[30,61570,61571],{},"books.authors.all"," in the view and pass authors in a variable that we can iterate over in the template, but this keeps things simpler without much more code to write.",[56,61574,61576],{"id":61575},"filtering-data","Filtering Data",[11,61578,61579,61580,61582,61583,61585,61586,61589,61590,61592],{},"One other question I wanted to answer with this project is how to do relatively complex data filtering. To do this, I used Django forms to create a form that calls a ",[30,61581,47765],{}," request to the same view it came from, which is to say, a filter form on the template for the ",[30,61584,12458],{}," books view has a form action with an ",[30,61587,61588],{},"action"," value of the url that returns the ",[30,61591,12458],{}," view.",[11,61594,61595],{},"Here's a look at the form I put together for filtering data:",[459,61597,61599],{"className":13136,"code":61598,"language":12886,"meta":464,"style":464},"class QueryForm(forms.Form):\n\n    publish_date_before = forms.DateField(\n        label='',\n        required=False,\n        # initial = datetime.datetime.now(),\n        widget = forms.TextInput(\n            attrs={\n                'class': 'form-control',\n                'id':'datepicker1',\n                'placeholder':'published before'\n            }))\n\n    publish_date_after = forms.DateField(\n        label='',\n        required=False,\n        # initial = datetime.datetime.now(),\n        widget = forms.TextInput(\n            attrs={\n                'class': 'form-control',\n                'id':'datepicker2',\n                'placeholder':'published after'\n            }))\n\n    keywords = forms.CharField(\n        required=False,\n        label='',\n        widget=forms.TextInput(\n\n            attrs={\n                'class':'form-control',\n                'placeholder':'space-separated words matching title, synopsis, website or tags'\n            }))\n",[30,61600,61601,61620,61624,61634,61645,61656,61661,61671,61680,61692,61704,61714,61719,61723,61732,61742,61752,61756,61764,61772,61782,61793,61802,61806,61810,61820,61830,61840,61849,61853,61861,61871,61880],{"__ignoreMap":464},[151,61602,61603,61605,61608,61610,61613,61615,61618],{"class":469,"line":470},[151,61604,16519],{"class":12347},[151,61606,61607],{"class":15254}," QueryForm",[151,61609,12386],{"class":503},[151,61611,61612],{"class":15260},"forms",[151,61614,643],{"class":503},[151,61616,61617],{"class":15260},"Form",[151,61619,15264],{"class":503},[151,61621,61622],{"class":469,"line":488},[151,61623,1090],{"emptyLinePlaceholder":609},[151,61625,61626,61629,61631],{"class":469,"line":500},[151,61627,61628],{"class":503},"    publish_date_before ",[151,61630,1876],{"class":1869},[151,61632,61633],{"class":503}," forms.DateField(\n",[151,61635,61636,61639,61641,61643],{"class":469,"line":509},[151,61637,61638],{"class":15210},"        label",[151,61640,1876],{"class":1869},[151,61642,2301],{"class":481},[151,61644,9417],{"class":503},[151,61646,61647,61650,61652,61654],{"class":469,"line":517},[151,61648,61649],{"class":15210},"        required",[151,61651,1876],{"class":1869},[151,61653,39461],{"class":477},[151,61655,9417],{"class":503},[151,61657,61658],{"class":469,"line":534},[151,61659,61660],{"class":1527},"        # initial = datetime.datetime.now(),\n",[151,61662,61663,61666,61668],{"class":469,"line":1413},[151,61664,61665],{"class":15210},"        widget",[151,61667,19865],{"class":1869},[151,61669,61670],{"class":503}," forms.TextInput(\n",[151,61672,61673,61676,61678],{"class":469,"line":1418},[151,61674,61675],{"class":15210},"            attrs",[151,61677,1876],{"class":1869},[151,61679,12966],{"class":503},[151,61681,61682,61685,61687,61690],{"class":469,"line":2462},[151,61683,61684],{"class":481},"                'class'",[151,61686,6208],{"class":503},[151,61688,61689],{"class":481},"'form-control'",[151,61691,9417],{"class":503},[151,61693,61694,61697,61699,61702],{"class":469,"line":2471},[151,61695,61696],{"class":481},"                'id'",[151,61698,208],{"class":503},[151,61700,61701],{"class":481},"'datepicker1'",[151,61703,9417],{"class":503},[151,61705,61706,61709,61711],{"class":469,"line":2480},[151,61707,61708],{"class":481},"                'placeholder'",[151,61710,208],{"class":503},[151,61712,61713],{"class":481},"'published before'\n",[151,61715,61716],{"class":469,"line":2489},[151,61717,61718],{"class":503},"            }))\n",[151,61720,61721],{"class":469,"line":2497},[151,61722,1090],{"emptyLinePlaceholder":609},[151,61724,61725,61728,61730],{"class":469,"line":3140},[151,61726,61727],{"class":503},"    publish_date_after ",[151,61729,1876],{"class":1869},[151,61731,61633],{"class":503},[151,61733,61734,61736,61738,61740],{"class":469,"line":3149},[151,61735,61638],{"class":15210},[151,61737,1876],{"class":1869},[151,61739,2301],{"class":481},[151,61741,9417],{"class":503},[151,61743,61744,61746,61748,61750],{"class":469,"line":3158},[151,61745,61649],{"class":15210},[151,61747,1876],{"class":1869},[151,61749,39461],{"class":477},[151,61751,9417],{"class":503},[151,61753,61754],{"class":469,"line":3167},[151,61755,61660],{"class":1527},[151,61757,61758,61760,61762],{"class":469,"line":3175},[151,61759,61665],{"class":15210},[151,61761,19865],{"class":1869},[151,61763,61670],{"class":503},[151,61765,61766,61768,61770],{"class":469,"line":3184},[151,61767,61675],{"class":15210},[151,61769,1876],{"class":1869},[151,61771,12966],{"class":503},[151,61773,61774,61776,61778,61780],{"class":469,"line":3193},[151,61775,61684],{"class":481},[151,61777,6208],{"class":503},[151,61779,61689],{"class":481},[151,61781,9417],{"class":503},[151,61783,61784,61786,61788,61791],{"class":469,"line":3720},[151,61785,61696],{"class":481},[151,61787,208],{"class":503},[151,61789,61790],{"class":481},"'datepicker2'",[151,61792,9417],{"class":503},[151,61794,61795,61797,61799],{"class":469,"line":3729},[151,61796,61708],{"class":481},[151,61798,208],{"class":503},[151,61800,61801],{"class":481},"'published after'\n",[151,61803,61804],{"class":469,"line":3735},[151,61805,61718],{"class":503},[151,61807,61808],{"class":469,"line":3745},[151,61809,1090],{"emptyLinePlaceholder":609},[151,61811,61812,61815,61817],{"class":469,"line":3754},[151,61813,61814],{"class":503},"    keywords ",[151,61816,1876],{"class":1869},[151,61818,61819],{"class":503}," forms.CharField(\n",[151,61821,61822,61824,61826,61828],{"class":469,"line":3760},[151,61823,61649],{"class":15210},[151,61825,1876],{"class":1869},[151,61827,39461],{"class":477},[151,61829,9417],{"class":503},[151,61831,61832,61834,61836,61838],{"class":469,"line":3773},[151,61833,61638],{"class":15210},[151,61835,1876],{"class":1869},[151,61837,2301],{"class":481},[151,61839,9417],{"class":503},[151,61841,61842,61844,61846],{"class":469,"line":3782},[151,61843,61665],{"class":15210},[151,61845,1876],{"class":1869},[151,61847,61848],{"class":503},"forms.TextInput(\n",[151,61850,61851],{"class":469,"line":3791},[151,61852,1090],{"emptyLinePlaceholder":609},[151,61854,61855,61857,61859],{"class":469,"line":3803},[151,61856,61675],{"class":15210},[151,61858,1876],{"class":1869},[151,61860,12966],{"class":503},[151,61862,61863,61865,61867,61869],{"class":469,"line":3811},[151,61864,61684],{"class":481},[151,61866,208],{"class":503},[151,61868,61689],{"class":481},[151,61870,9417],{"class":503},[151,61872,61873,61875,61877],{"class":469,"line":3820},[151,61874,61708],{"class":481},[151,61876,208],{"class":503},[151,61878,61879],{"class":481},"'space-separated words matching title, synopsis, website or tags'\n",[151,61881,61882],{"class":469,"line":7084},[151,61883,61718],{"class":503},[11,61885,61886,61887,61890,61891,61894,61895,46426],{},"Here we are using three fields to filter data, a ",[30,61888,61889],{},"published_before"," date, a ",[30,61892,61893],{},"published_after"," date and a ",[30,61896,61897],{},"keywords",[11,61899,61900,61901,61904,61905,61908,61909,61912,61913,61915],{},"Filtering this data can really clog up code in ",[30,61902,61903],{},"views.py",", and I know I was going to need this same code again for filtering data for CSV and XLS file downloads (more on this in a minute), so I decided to wrote a utility function that takes a ",[30,61906,61907],{},"Book"," queryset and parameter dictionary (",[30,61910,61911],{},"request.GET","), and returns a filtered ",[30,61914,61907],{}," queryset. Here's what that function looks like:",[459,61917,61919],{"className":13136,"code":61918,"language":12886,"meta":464,"style":464},"from django.db.models import Q\nfrom functools import reduce\nfrom ..models import Book\nimport datetime\n\ndef filter_books(books, paramDict):\n    # paramDict = request.GET\n    params = paramDict.keys()\n\n    # data filtering\n    if any(x!='' for x in paramDict.values()):\n        if paramDict['publish_date_after'] != '':\n            after_date = paramDict['publish_date_after']\n            _after_date = datetime.datetime.strptime(after_date, '%m/%d/%Y')\n\n            books = books.filter(publish_date__gte=_after_date)\n\n        if paramDict['publish_date_before'] != '':\n            before_date = paramDict['publish_date_before']\n            _before_date = datetime.datetime.strptime(before_date, '%m/%d/%Y')\n            books = books.filter(publish_date__lte=_before_date)\n\n        # filters records that contain any of the following keywords\n        if paramDict['keywords'] != '':\n            kws = paramDict['keywords'].split()\n            q_lookups = [Q(title__icontains=kw) for kw in kws] + \\\n                        [Q(synopsis__icontains=kw) for kw in kws] + \\\n                        [Q(website__icontains=kw) for kw in kws]\n            filters = Q()\n            filters |= reduce(lambda x, y: x | y, q_lookups)\n            books = books.filter(filters)\n\n    return books\n",[30,61920,61921,61933,61945,61956,61962,61966,61984,61989,61998,62002,62007,62030,62048,62061,62082,62086,62104,62108,62125,62138,62156,62172,62176,62181,62198,62212,62244,62268,62288,62298,62326,62335,62339],{"__ignoreMap":464},[151,61922,61923,61925,61928,61930],{"class":469,"line":470},[151,61924,16853],{"class":1869},[151,61926,61927],{"class":503}," django.db.models ",[151,61929,16859],{"class":1869},[151,61931,61932],{"class":503}," Q\n",[151,61934,61935,61937,61940,61942],{"class":469,"line":488},[151,61936,16853],{"class":1869},[151,61938,61939],{"class":503}," functools ",[151,61941,16859],{"class":1869},[151,61943,61944],{"class":12354}," reduce\n",[151,61946,61947,61949,61952,61954],{"class":469,"line":500},[151,61948,16853],{"class":1869},[151,61950,61951],{"class":503}," ..models ",[151,61953,16859],{"class":1869},[151,61955,61263],{"class":503},[151,61957,61958,61960],{"class":469,"line":509},[151,61959,16859],{"class":1869},[151,61961,45618],{"class":503},[151,61963,61964],{"class":469,"line":517},[151,61965,1090],{"emptyLinePlaceholder":609},[151,61967,61968,61970,61973,61975,61977,61979,61982],{"class":469,"line":534},[151,61969,16925],{"class":12347},[151,61971,61972],{"class":473}," filter_books",[151,61974,12386],{"class":503},[151,61976,59665],{"class":15232},[151,61978,106],{"class":503},[151,61980,61981],{"class":15232},"paramDict",[151,61983,15264],{"class":503},[151,61985,61986],{"class":469,"line":1413},[151,61987,61988],{"class":1527},"    # paramDict = request.GET\n",[151,61990,61991,61993,61995],{"class":469,"line":1418},[151,61992,57071],{"class":503},[151,61994,1876],{"class":1869},[151,61996,61997],{"class":503}," paramDict.keys()\n",[151,61999,62000],{"class":469,"line":2462},[151,62001,1090],{"emptyLinePlaceholder":609},[151,62003,62004],{"class":469,"line":2471},[151,62005,62006],{"class":1527},"    # data filtering\n",[151,62008,62009,62011,62014,62017,62019,62021,62023,62025,62027],{"class":469,"line":2480},[151,62010,23327],{"class":1869},[151,62012,62013],{"class":2226}," any",[151,62015,62016],{"class":503},"(x",[151,62018,58602],{"class":1869},[151,62020,2301],{"class":481},[151,62022,2235],{"class":1869},[151,62024,44552],{"class":503},[151,62026,16417],{"class":1869},[151,62028,62029],{"class":503}," paramDict.values()):\n",[151,62031,62032,62034,62037,62040,62042,62044,62046],{"class":469,"line":2489},[151,62033,23357],{"class":1869},[151,62035,62036],{"class":503}," paramDict[",[151,62038,62039],{"class":481},"'publish_date_after'",[151,62041,16654],{"class":503},[151,62043,58602],{"class":1869},[151,62045,58969],{"class":481},[151,62047,14372],{"class":503},[151,62049,62050,62053,62055,62057,62059],{"class":469,"line":2497},[151,62051,62052],{"class":503},"            after_date ",[151,62054,1876],{"class":1869},[151,62056,62036],{"class":503},[151,62058,62039],{"class":481},[151,62060,3691],{"class":503},[151,62062,62063,62066,62068,62071,62074,62077,62080],{"class":469,"line":3140},[151,62064,62065],{"class":503},"            _after_date ",[151,62067,1876],{"class":1869},[151,62069,62070],{"class":503}," datetime.datetime.strptime(after_date, ",[151,62072,62073],{"class":481},"'%m/",[151,62075,62076],{"class":477},"%d",[151,62078,62079],{"class":481},"/%Y'",[151,62081,3640],{"class":503},[151,62083,62084],{"class":469,"line":3149},[151,62085,1090],{"emptyLinePlaceholder":609},[151,62087,62088,62091,62093,62096,62099,62101],{"class":469,"line":3158},[151,62089,62090],{"class":503},"            books ",[151,62092,1876],{"class":1869},[151,62094,62095],{"class":503}," books.filter(",[151,62097,62098],{"class":15210},"publish_date__gte",[151,62100,1876],{"class":1869},[151,62102,62103],{"class":503},"_after_date)\n",[151,62105,62106],{"class":469,"line":3167},[151,62107,1090],{"emptyLinePlaceholder":609},[151,62109,62110,62112,62114,62117,62119,62121,62123],{"class":469,"line":3175},[151,62111,23357],{"class":1869},[151,62113,62036],{"class":503},[151,62115,62116],{"class":481},"'publish_date_before'",[151,62118,16654],{"class":503},[151,62120,58602],{"class":1869},[151,62122,58969],{"class":481},[151,62124,14372],{"class":503},[151,62126,62127,62130,62132,62134,62136],{"class":469,"line":3184},[151,62128,62129],{"class":503},"            before_date ",[151,62131,1876],{"class":1869},[151,62133,62036],{"class":503},[151,62135,62116],{"class":481},[151,62137,3691],{"class":503},[151,62139,62140,62143,62145,62148,62150,62152,62154],{"class":469,"line":3193},[151,62141,62142],{"class":503},"            _before_date ",[151,62144,1876],{"class":1869},[151,62146,62147],{"class":503}," datetime.datetime.strptime(before_date, ",[151,62149,62073],{"class":481},[151,62151,62076],{"class":477},[151,62153,62079],{"class":481},[151,62155,3640],{"class":503},[151,62157,62158,62160,62162,62164,62167,62169],{"class":469,"line":3720},[151,62159,62090],{"class":503},[151,62161,1876],{"class":1869},[151,62163,62095],{"class":503},[151,62165,62166],{"class":15210},"publish_date__lte",[151,62168,1876],{"class":1869},[151,62170,62171],{"class":503},"_before_date)\n",[151,62173,62174],{"class":469,"line":3729},[151,62175,1090],{"emptyLinePlaceholder":609},[151,62177,62178],{"class":469,"line":3735},[151,62179,62180],{"class":1527},"        # filters records that contain any of the following keywords\n",[151,62182,62183,62185,62187,62190,62192,62194,62196],{"class":469,"line":3745},[151,62184,23357],{"class":1869},[151,62186,62036],{"class":503},[151,62188,62189],{"class":481},"'keywords'",[151,62191,16654],{"class":503},[151,62193,58602],{"class":1869},[151,62195,58969],{"class":481},[151,62197,14372],{"class":503},[151,62199,62200,62203,62205,62207,62209],{"class":469,"line":3754},[151,62201,62202],{"class":503},"            kws ",[151,62204,1876],{"class":1869},[151,62206,62036],{"class":503},[151,62208,62189],{"class":481},[151,62210,62211],{"class":503},"].split()\n",[151,62213,62214,62217,62219,62222,62225,62227,62230,62232,62235,62237,62240,62242],{"class":469,"line":3760},[151,62215,62216],{"class":503},"            q_lookups ",[151,62218,1876],{"class":1869},[151,62220,62221],{"class":503}," [Q(",[151,62223,62224],{"class":15210},"title__icontains",[151,62226,1876],{"class":1869},[151,62228,62229],{"class":503},"kw) ",[151,62231,16732],{"class":1869},[151,62233,62234],{"class":503}," kw ",[151,62236,16417],{"class":1869},[151,62238,62239],{"class":503}," kws] ",[151,62241,22885],{"class":1869},[151,62243,485],{"class":503},[151,62245,62246,62249,62252,62254,62256,62258,62260,62262,62264,62266],{"class":469,"line":3773},[151,62247,62248],{"class":503},"                        [Q(",[151,62250,62251],{"class":15210},"synopsis__icontains",[151,62253,1876],{"class":1869},[151,62255,62229],{"class":503},[151,62257,16732],{"class":1869},[151,62259,62234],{"class":503},[151,62261,16417],{"class":1869},[151,62263,62239],{"class":503},[151,62265,22885],{"class":1869},[151,62267,485],{"class":503},[151,62269,62270,62272,62275,62277,62279,62281,62283,62285],{"class":469,"line":3782},[151,62271,62248],{"class":503},[151,62273,62274],{"class":15210},"website__icontains",[151,62276,1876],{"class":1869},[151,62278,62229],{"class":503},[151,62280,16732],{"class":1869},[151,62282,62234],{"class":503},[151,62284,16417],{"class":1869},[151,62286,62287],{"class":503}," kws]\n",[151,62289,62290,62293,62295],{"class":469,"line":3791},[151,62291,62292],{"class":503},"            filters ",[151,62294,1876],{"class":1869},[151,62296,62297],{"class":503}," Q()\n",[151,62299,62300,62302,62305,62308,62310,62312,62314,62316,62318,62321,62323],{"class":469,"line":3803},[151,62301,62292],{"class":503},[151,62303,62304],{"class":1869},"|=",[151,62306,62307],{"class":12354}," reduce",[151,62309,12386],{"class":503},[151,62311,43773],{"class":12347},[151,62313,27729],{"class":15232},[151,62315,106],{"class":503},[151,62317,25286],{"class":15232},[151,62319,62320],{"class":503},": x ",[151,62322,3947],{"class":1869},[151,62324,62325],{"class":503}," y, q_lookups)\n",[151,62327,62328,62330,62332],{"class":469,"line":3811},[151,62329,62090],{"class":503},[151,62331,1876],{"class":1869},[151,62333,62334],{"class":503}," books.filter(filters)\n",[151,62336,62337],{"class":469,"line":3820},[151,62338,1090],{"emptyLinePlaceholder":609},[151,62340,62341,62343],{"class":469,"line":7084},[151,62342,17496],{"class":1869},[151,62344,62345],{"class":503}," books\n",[11,62347,62348,62349,62351],{},"This makes things much more simple in our views. Here's the code in the main ",[30,62350,59665],{}," view that uses this filter function, truncated for simplicity:",[459,62353,62355],{"className":13136,"code":62354,"language":12886,"meta":464,"style":464},"def all_books(request):\n    books = Book.objects.all()\n    form = QueryForm(request.GET or None)\n    paramDict = request.GET\n    books = filter_books(books, paramDict)\n    [...]\n    context = {\n        'books':books,\n        'form':form,\n        [...],}\n    return render(request, 'books/books.html', context)\n",[30,62356,62357,62369,62377,62393,62403,62411,62419,62427,62433,62440,62449],{"__ignoreMap":464},[151,62358,62359,62361,62363,62365,62367],{"class":469,"line":470},[151,62360,16925],{"class":12347},[151,62362,59681],{"class":473},[151,62364,12386],{"class":503},[151,62366,59686],{"class":15232},[151,62368,15264],{"class":503},[151,62370,62371,62373,62375],{"class":469,"line":488},[151,62372,59706],{"class":503},[151,62374,1876],{"class":1869},[151,62376,59711],{"class":503},[151,62378,62379,62381,62383,62385,62387,62389,62391],{"class":469,"line":500},[151,62380,59716],{"class":503},[151,62382,1876],{"class":1869},[151,62384,59721],{"class":503},[151,62386,47765],{"class":477},[151,62388,2161],{"class":1869},[151,62390,40451],{"class":477},[151,62392,3640],{"class":503},[151,62394,62395,62397,62399,62401],{"class":469,"line":509},[151,62396,59734],{"class":503},[151,62398,1876],{"class":1869},[151,62400,40684],{"class":503},[151,62402,14433],{"class":477},[151,62404,62405,62407,62409],{"class":469,"line":517},[151,62406,59706],{"class":503},[151,62408,1876],{"class":1869},[151,62410,59753],{"class":503},[151,62412,62413,62415,62417],{"class":469,"line":534},[151,62414,33774],{"class":503},[151,62416,27455],{"class":477},[151,62418,3691],{"class":503},[151,62420,62421,62423,62425],{"class":469,"line":1413},[151,62422,59844],{"class":503},[151,62424,1876],{"class":1869},[151,62426,19833],{"class":503},[151,62428,62429,62431],{"class":469,"line":1418},[151,62430,59853],{"class":481},[151,62432,59856],{"class":503},[151,62434,62435,62437],{"class":469,"line":2462},[151,62436,59892],{"class":481},[151,62438,62439],{"class":503},":form,\n",[151,62441,62442,62444,62446],{"class":469,"line":2471},[151,62443,23249],{"class":503},[151,62445,27455],{"class":477},[151,62447,62448],{"class":503},"],}\n",[151,62450,62451,62453,62455,62457],{"class":469,"line":2480},[151,62452,17496],{"class":1869},[151,62454,59902],{"class":503},[151,62456,59905],{"class":481},[151,62458,59908],{"class":503},[56,62460,62462],{"id":62461},"exporting-data-as-csv-or-xls","Exporting data as CSV or XLS",[11,62464,62465],{},"One other requirement I gave myself for this project was giving users the option to export data in CSV or XLS file formats.",[11,62467,62468,62469,62474,62475,62479,62480,208],{},"With the filter data function, I was able to keep things DRY (Don't Repeart Yourself). This task taught me about a few aspects of HTML5 and forms that I wasn't aware of. First, let's take a look at the CSV export function that I learned about throught ",[20,62470,62473],{"href":62471,"rel":62472},"https://simpleisbetterthancomplex.com/tutorial/2016/07/29/how-to-export-to-excel.html",[24],"this blog post",", from a great Django blog called ",[20,62476,62478],{"href":62471,"rel":62477},[24],"Simple is Better than Complex"," by ",[20,62481,62484],{"href":62482,"rel":62483},"https://github.com/vitorfs",[24],"Vitor Freitas",[459,62486,62488],{"className":13136,"code":62487,"language":12886,"meta":464,"style":464},"def export_filtered_books_csv(request):\n    response = HttpResponse(content_type='text/csv')\n    response['Content-Disposition'] = 'attachment; filename=\"books.csv\"'\n\n    writer = csv.writer(response)\n    writer.writerow(['Title', 'Synopsis', 'Pages'])\n    books = Book.objects.all()\n    paramDict = request.GET\n    books = filter_books(books, paramDict)\n    books = books.values_list(\n        'title',\n        'synopsis',\n        'pages')\n\n    for book in books:\n        writer.writerow(book)\n\n    return response\n",[30,62489,62490,62503,62522,62537,62541,62551,62571,62579,62589,62597,62606,62613,62620,62627,62631,62642,62647,62651],{"__ignoreMap":464},[151,62491,62492,62494,62497,62499,62501],{"class":469,"line":470},[151,62493,16925],{"class":12347},[151,62495,62496],{"class":473}," export_filtered_books_csv",[151,62498,12386],{"class":503},[151,62500,59686],{"class":15232},[151,62502,15264],{"class":503},[151,62504,62505,62507,62509,62512,62515,62517,62520],{"class":469,"line":488},[151,62506,57136],{"class":503},[151,62508,1876],{"class":1869},[151,62510,62511],{"class":503}," HttpResponse(",[151,62513,62514],{"class":15210},"content_type",[151,62516,1876],{"class":1869},[151,62518,62519],{"class":481},"'text/csv'",[151,62521,3640],{"class":503},[151,62523,62524,62527,62530,62532,62534],{"class":469,"line":500},[151,62525,62526],{"class":503},"    response[",[151,62528,62529],{"class":481},"'Content-Disposition'",[151,62531,16654],{"class":503},[151,62533,1876],{"class":1869},[151,62535,62536],{"class":481}," 'attachment; filename=\"books.csv\"'\n",[151,62538,62539],{"class":469,"line":509},[151,62540,1090],{"emptyLinePlaceholder":609},[151,62542,62543,62546,62548],{"class":469,"line":517},[151,62544,62545],{"class":503},"    writer ",[151,62547,1876],{"class":1869},[151,62549,62550],{"class":503}," csv.writer(response)\n",[151,62552,62553,62556,62559,62561,62564,62566,62569],{"class":469,"line":534},[151,62554,62555],{"class":503},"    writer.writerow([",[151,62557,62558],{"class":481},"'Title'",[151,62560,106],{"class":503},[151,62562,62563],{"class":481},"'Synopsis'",[151,62565,106],{"class":503},[151,62567,62568],{"class":481},"'Pages'",[151,62570,38820],{"class":503},[151,62572,62573,62575,62577],{"class":469,"line":1413},[151,62574,59706],{"class":503},[151,62576,1876],{"class":1869},[151,62578,59711],{"class":503},[151,62580,62581,62583,62585,62587],{"class":469,"line":1418},[151,62582,59734],{"class":503},[151,62584,1876],{"class":1869},[151,62586,40684],{"class":503},[151,62588,14433],{"class":477},[151,62590,62591,62593,62595],{"class":469,"line":2462},[151,62592,59706],{"class":503},[151,62594,1876],{"class":1869},[151,62596,59753],{"class":503},[151,62598,62599,62601,62603],{"class":469,"line":2471},[151,62600,59706],{"class":503},[151,62602,1876],{"class":1869},[151,62604,62605],{"class":503}," books.values_list(\n",[151,62607,62608,62611],{"class":469,"line":2480},[151,62609,62610],{"class":481},"        'title'",[151,62612,9417],{"class":503},[151,62614,62615,62618],{"class":469,"line":2489},[151,62616,62617],{"class":481},"        'synopsis'",[151,62619,9417],{"class":503},[151,62621,62622,62625],{"class":469,"line":2497},[151,62623,62624],{"class":481},"        'pages'",[151,62626,3640],{"class":503},[151,62628,62629],{"class":469,"line":3140},[151,62630,1090],{"emptyLinePlaceholder":609},[151,62632,62633,62635,62637,62639],{"class":469,"line":3149},[151,62634,16411],{"class":1869},[151,62636,59834],{"class":503},[151,62638,16417],{"class":1869},[151,62640,62641],{"class":503}," books:\n",[151,62643,62644],{"class":469,"line":3158},[151,62645,62646],{"class":503},"        writer.writerow(book)\n",[151,62648,62649],{"class":469,"line":3167},[151,62650,1090],{"emptyLinePlaceholder":609},[151,62652,62653,62655],{"class":469,"line":3175},[151,62654,17496],{"class":1869},[151,62656,62657],{"class":503}," response\n",[11,62659,62660,62661,62663],{},"I wanted to put this button in the filter form on the main ",[30,62662,59665],{}," page, but I need to place the button outside of the form tag. To get around this, here is the HTML I used:",[11,62665,59441],{},[459,62667,62669],{"className":19811,"code":62668,"language":19813,"meta":464,"style":464},"\u003Cform action=\"{% url 'books:csv' %}\">\n  \u003Ca\n    class=\"btn btn-primary\"\n    data-toggle=\"collapse\"\n    href=\"#collapseExample\"\n    aria-expanded=\"false\"\n    aria-controls=\"collapseExample\"\n  >\n    Filter Books\n  \u003C/a>\n  \u003Cinput\n    type=\"submit\"\n    class=\"btn btn-info\"\n    form=\"id_query_form\"\n    formaction=\"{% url 'books:csv' %}\"\n    value=\"Export CSV\"\n  />\n  \u003Cinput\n    type=\"submit\"\n    class=\"btn btn-default\"\n    form=\"id_query_form\"\n    formaction=\"{% url 'books:xls' %}\"\n    value=\"Export XLS\"\n  />\n\u003C/form>\n\n\u003Cform method=\"get\" action=\".\" id=\"id_query_form\">\n  \u003Cp>{{ form.keywords }}\u003C/p>\n  \u003Cdiv class=\"row\">\n    \u003Cdiv class=\"col-md-6\">{{ form.publish_date_before }}\u003C/div>\n    \u003Cdiv class=\"col-md-6\">{{ form.publish_date_after }}\u003C/div>\n  \u003C/div>\n\n  \u003Cp>\u003C/p>\n  \u003Cp>\n    \u003Cinput class=\"btn btn-success\" type=\"submit\" />\n    \u003Cbutton type=\"reset\" class=\"btn btn-info\" value=\"Reset filters\">\n      Reset filters\n    \u003C/button>\n    \u003Cbutton\n      type=\"reset\"\n      class=\"btn btn-warning\"\n      id=\"id_clear_filters\"\n      onclick=\"return resetForm(this.form);\"\n    >\n      Clear Filters\n    \u003C/button>\n  \u003C/p>\n\u003C/form>\n",[30,62670,62671,62686,62692,62701,62711,62721,62731,62741,62745,62750,62758,62764,62773,62782,62792,62802,62812,62817,62823,62831,62840,62848,62857,62866,62870,62878,62882,62910,62923,62938,62958,62977,62985,62989,63002,63010,63031,63061,63066,63074,63081,63090,63100,63110,63136,63141,63146,63154,63162],{"__ignoreMap":464},[151,62672,62673,62675,62677,62679,62681,62684],{"class":469,"line":470},[151,62674,3613],{"class":503},[151,62676,48710],{"class":14368},[151,62678,60980],{"class":473},[151,62680,1876],{"class":503},[151,62682,62683],{"class":481},"\"{% url 'books:csv' %}\"",[151,62685,3742],{"class":503},[151,62687,62688,62690],{"class":469,"line":488},[151,62689,33991],{"class":503},[151,62691,61545],{"class":14368},[151,62693,62694,62696,62698],{"class":469,"line":500},[151,62695,15251],{"class":473},[151,62697,1876],{"class":503},[151,62699,62700],{"class":481},"\"btn btn-primary\"\n",[151,62702,62703,62706,62708],{"class":469,"line":509},[151,62704,62705],{"class":473},"    data-toggle",[151,62707,1876],{"class":503},[151,62709,62710],{"class":481},"\"collapse\"\n",[151,62712,62713,62716,62718],{"class":469,"line":517},[151,62714,62715],{"class":473},"    href",[151,62717,1876],{"class":503},[151,62719,62720],{"class":481},"\"#collapseExample\"\n",[151,62722,62723,62726,62728],{"class":469,"line":534},[151,62724,62725],{"class":473},"    aria-expanded",[151,62727,1876],{"class":503},[151,62729,62730],{"class":481},"\"false\"\n",[151,62732,62733,62736,62738],{"class":469,"line":1413},[151,62734,62735],{"class":473},"    aria-controls",[151,62737,1876],{"class":503},[151,62739,62740],{"class":481},"\"collapseExample\"\n",[151,62742,62743],{"class":469,"line":1418},[151,62744,61550],{"class":503},[151,62746,62747],{"class":469,"line":2462},[151,62748,62749],{"class":503},"    Filter Books\n",[151,62751,62752,62754,62756],{"class":469,"line":2471},[151,62753,34741],{"class":503},[151,62755,20],{"class":14368},[151,62757,3742],{"class":503},[151,62759,62760,62762],{"class":469,"line":2480},[151,62761,33991],{"class":503},[151,62763,48347],{"class":14368},[151,62765,62766,62769,62771],{"class":469,"line":2489},[151,62767,62768],{"class":473},"    type",[151,62770,1876],{"class":503},[151,62772,48646],{"class":481},[151,62774,62775,62777,62779],{"class":469,"line":2497},[151,62776,15251],{"class":473},[151,62778,1876],{"class":503},[151,62780,62781],{"class":481},"\"btn btn-info\"\n",[151,62783,62784,62787,62789],{"class":469,"line":3140},[151,62785,62786],{"class":473},"    form",[151,62788,1876],{"class":503},[151,62790,62791],{"class":481},"\"id_query_form\"\n",[151,62793,62794,62797,62799],{"class":469,"line":3149},[151,62795,62796],{"class":473},"    formaction",[151,62798,1876],{"class":503},[151,62800,62801],{"class":481},"\"{% url 'books:csv' %}\"\n",[151,62803,62804,62807,62809],{"class":469,"line":3158},[151,62805,62806],{"class":473},"    value",[151,62808,1876],{"class":503},[151,62810,62811],{"class":481},"\"Export CSV\"\n",[151,62813,62814],{"class":469,"line":3167},[151,62815,62816],{"class":503},"  />\n",[151,62818,62819,62821],{"class":469,"line":3175},[151,62820,33991],{"class":503},[151,62822,48347],{"class":14368},[151,62824,62825,62827,62829],{"class":469,"line":3184},[151,62826,62768],{"class":473},[151,62828,1876],{"class":503},[151,62830,48646],{"class":481},[151,62832,62833,62835,62837],{"class":469,"line":3193},[151,62834,15251],{"class":473},[151,62836,1876],{"class":503},[151,62838,62839],{"class":481},"\"btn btn-default\"\n",[151,62841,62842,62844,62846],{"class":469,"line":3720},[151,62843,62786],{"class":473},[151,62845,1876],{"class":503},[151,62847,62791],{"class":481},[151,62849,62850,62852,62854],{"class":469,"line":3729},[151,62851,62796],{"class":473},[151,62853,1876],{"class":503},[151,62855,62856],{"class":481},"\"{% url 'books:xls' %}\"\n",[151,62858,62859,62861,62863],{"class":469,"line":3735},[151,62860,62806],{"class":473},[151,62862,1876],{"class":503},[151,62864,62865],{"class":481},"\"Export XLS\"\n",[151,62867,62868],{"class":469,"line":3745},[151,62869,62816],{"class":503},[151,62871,62872,62874,62876],{"class":469,"line":3754},[151,62873,19966],{"class":503},[151,62875,48710],{"class":14368},[151,62877,3742],{"class":503},[151,62879,62880],{"class":469,"line":3760},[151,62881,1090],{"emptyLinePlaceholder":609},[151,62883,62884,62886,62888,62890,62892,62895,62897,62899,62901,62903,62905,62908],{"class":469,"line":3773},[151,62885,3613],{"class":503},[151,62887,48710],{"class":14368},[151,62889,60972],{"class":473},[151,62891,1876],{"class":503},[151,62893,62894],{"class":481},"\"get\"",[151,62896,60980],{"class":473},[151,62898,1876],{"class":503},[151,62900,44221],{"class":481},[151,62902,48210],{"class":473},[151,62904,1876],{"class":503},[151,62906,62907],{"class":481},"\"id_query_form\"",[151,62909,3742],{"class":503},[151,62911,62912,62914,62916,62919,62921],{"class":469,"line":3782},[151,62913,33991],{"class":503},[151,62915,11],{"class":14368},[151,62917,62918],{"class":503},">{{ form.keywords }}\u003C/",[151,62920,11],{"class":14368},[151,62922,3742],{"class":503},[151,62924,62925,62927,62929,62931,62933,62936],{"class":469,"line":3791},[151,62926,33991],{"class":503},[151,62928,23950],{"class":14368},[151,62930,48323],{"class":473},[151,62932,1876],{"class":503},[151,62934,62935],{"class":481},"\"row\"",[151,62937,3742],{"class":503},[151,62939,62940,62942,62944,62946,62948,62951,62954,62956],{"class":469,"line":3803},[151,62941,34669],{"class":503},[151,62943,23950],{"class":14368},[151,62945,48323],{"class":473},[151,62947,1876],{"class":503},[151,62949,62950],{"class":481},"\"col-md-6\"",[151,62952,62953],{"class":503},">{{ form.publish_date_before }}\u003C/",[151,62955,23950],{"class":14368},[151,62957,3742],{"class":503},[151,62959,62960,62962,62964,62966,62968,62970,62973,62975],{"class":469,"line":3811},[151,62961,34669],{"class":503},[151,62963,23950],{"class":14368},[151,62965,48323],{"class":473},[151,62967,1876],{"class":503},[151,62969,62950],{"class":481},[151,62971,62972],{"class":503},">{{ form.publish_date_after }}\u003C/",[151,62974,23950],{"class":14368},[151,62976,3742],{"class":503},[151,62978,62979,62981,62983],{"class":469,"line":3820},[151,62980,34741],{"class":503},[151,62982,23950],{"class":14368},[151,62984,3742],{"class":503},[151,62986,62987],{"class":469,"line":7084},[151,62988,1090],{"emptyLinePlaceholder":609},[151,62990,62991,62993,62995,62998,63000],{"class":469,"line":7148},[151,62992,33991],{"class":503},[151,62994,11],{"class":14368},[151,62996,62997],{"class":503},">\u003C/",[151,62999,11],{"class":14368},[151,63001,3742],{"class":503},[151,63003,63004,63006,63008],{"class":469,"line":7211},[151,63005,33991],{"class":503},[151,63007,11],{"class":14368},[151,63009,3742],{"class":503},[151,63011,63012,63014,63016,63018,63020,63023,63025,63027,63029],{"class":469,"line":7273},[151,63013,34669],{"class":503},[151,63015,29860],{"class":14368},[151,63017,48323],{"class":473},[151,63019,1876],{"class":503},[151,63021,63022],{"class":481},"\"btn btn-success\"",[151,63024,61000],{"class":473},[151,63026,1876],{"class":503},[151,63028,61057],{"class":481},[151,63030,34675],{"class":503},[151,63032,63033,63035,63038,63040,63042,63045,63047,63049,63052,63054,63056,63059],{"class":469,"line":7335},[151,63034,34669],{"class":503},[151,63036,63037],{"class":14368},"button",[151,63039,61000],{"class":473},[151,63041,1876],{"class":503},[151,63043,63044],{"class":481},"\"reset\"",[151,63046,48323],{"class":473},[151,63048,1876],{"class":503},[151,63050,63051],{"class":481},"\"btn btn-info\"",[151,63053,2186],{"class":473},[151,63055,1876],{"class":503},[151,63057,63058],{"class":481},"\"Reset filters\"",[151,63060,3742],{"class":503},[151,63062,63063],{"class":469,"line":7398},[151,63064,63065],{"class":503},"      Reset filters\n",[151,63067,63068,63070,63072],{"class":469,"line":7462},[151,63069,48717],{"class":503},[151,63071,63037],{"class":14368},[151,63073,3742],{"class":503},[151,63075,63076,63078],{"class":469,"line":7467},[151,63077,34669],{"class":503},[151,63079,63080],{"class":14368},"button\n",[151,63082,63083,63085,63087],{"class":469,"line":7532},[151,63084,40928],{"class":473},[151,63086,1876],{"class":503},[151,63088,63089],{"class":481},"\"reset\"\n",[151,63091,63092,63095,63097],{"class":469,"line":7537},[151,63093,63094],{"class":473},"      class",[151,63096,1876],{"class":503},[151,63098,63099],{"class":481},"\"btn btn-warning\"\n",[151,63101,63102,63105,63107],{"class":469,"line":7603},[151,63103,63104],{"class":473},"      id",[151,63106,1876],{"class":503},[151,63108,63109],{"class":481},"\"id_clear_filters\"\n",[151,63111,63112,63115,63117,63119,63122,63125,63127,63129,63131,63133],{"class":469,"line":7608},[151,63113,63114],{"class":473},"      onclick",[151,63116,1876],{"class":503},[151,63118,8592],{"class":481},[151,63120,63121],{"class":1869},"return",[151,63123,63124],{"class":473}," resetForm",[151,63126,12386],{"class":481},[151,63128,23252],{"class":15289},[151,63130,643],{"class":481},[151,63132,48710],{"class":503},[151,63134,63135],{"class":481},");\"\n",[151,63137,63138],{"class":469,"line":7673},[151,63139,63140],{"class":503},"    >\n",[151,63142,63143],{"class":469,"line":7678},[151,63144,63145],{"class":503},"      Clear Filters\n",[151,63147,63148,63150,63152],{"class":469,"line":7708},[151,63149,48717],{"class":503},[151,63151,63037],{"class":14368},[151,63153,3742],{"class":503},[151,63155,63156,63158,63160],{"class":469,"line":7713},[151,63157,34741],{"class":503},[151,63159,11],{"class":14368},[151,63161,3742],{"class":503},[151,63163,63164,63166,63168],{"class":469,"line":7746},[151,63165,19966],{"class":503},[151,63167,48710],{"class":14368},[151,63169,3742],{"class":503},[11,63171,59528],{},[11,63173,63174,63175,63177,63178,63181,63182,63185,63186,63189],{},"There are two forms here, but the first form has an ",[30,63176,29860],{}," of ",[30,63179,63180],{},"type=submit"," that references another form with ",[30,63183,63184],{},"form=\"id_query_form\"",", and this button's action is the url for the ",[30,63187,63188],{},"export_filtered_books_csv"," function shown above.",[11,63191,63192,63193,63196,63197,63199],{},"For the filter function, I make a ",[30,63194,63195],{},"utils"," folder in the app I am working with, so the directory structure looks like this (for the ",[30,63198,59665],{}," app):",[459,63201,63204],{"className":63202,"code":63203,"language":997},[995],".\n├── admin.py\n├── apps.py\n├── forms.py\n├── __init__.py\n├── migrations\n│   ├── 0001_initial.py\n│   ├── 0002_book_authors.py\n│   └── __init__.py\n├── models.py\n├── templates\n│   └── books\n├── tests.py\n├── urls.py\n├── utils\n│   ├── filter.py\n│   ├── __init__.py\n│   └── nearby.py\n└── views.py\n",[30,63205,63203],{"__ignoreMap":464},[11,63207,63208,63209,106,63211,106,63213,13576],{},"I initially had a litte bit of confusion on how to write a simple search query. I wanted a user to be able to enter terms that would return a query set where the items in the query set contained at least on of the terms in at least on of a few different fields (",[30,63210,19633],{},[30,63212,22052],{},[30,63214,63215],{},"website",[11,63217,63218],{},"To make this filter, I found a nice idiomatic pattern:",[459,63220,63222],{"className":13136,"code":63221,"language":12886,"meta":464,"style":464},"if paramDict['keywords'] != '':\n    kws = paramDict['keywords'].split()\n    q_lookups = [Q(title__icontains=kw) for kw in kws] + \\\n                [Q(synopsis__icontains=kw) for kw in kws] + \\\n                [Q(website__icontains=kw) for kw in kws]\n    filters = Q()\n    filters |= reduce(lambda x, y: x | y, q_lookups)\n    books = books.filter(filters)\n",[30,63223,63224,63240,63253,63280,63303,63321,63330,63354],{"__ignoreMap":464},[151,63225,63226,63228,63230,63232,63234,63236,63238],{"class":469,"line":470},[151,63227,17218],{"class":1869},[151,63229,62036],{"class":503},[151,63231,62189],{"class":481},[151,63233,16654],{"class":503},[151,63235,58602],{"class":1869},[151,63237,58969],{"class":481},[151,63239,14372],{"class":503},[151,63241,63242,63245,63247,63249,63251],{"class":469,"line":488},[151,63243,63244],{"class":503},"    kws ",[151,63246,1876],{"class":1869},[151,63248,62036],{"class":503},[151,63250,62189],{"class":481},[151,63252,62211],{"class":503},[151,63254,63255,63258,63260,63262,63264,63266,63268,63270,63272,63274,63276,63278],{"class":469,"line":500},[151,63256,63257],{"class":503},"    q_lookups ",[151,63259,1876],{"class":1869},[151,63261,62221],{"class":503},[151,63263,62224],{"class":15210},[151,63265,1876],{"class":1869},[151,63267,62229],{"class":503},[151,63269,16732],{"class":1869},[151,63271,62234],{"class":503},[151,63273,16417],{"class":1869},[151,63275,62239],{"class":503},[151,63277,22885],{"class":1869},[151,63279,485],{"class":503},[151,63281,63282,63285,63287,63289,63291,63293,63295,63297,63299,63301],{"class":469,"line":509},[151,63283,63284],{"class":503},"                [Q(",[151,63286,62251],{"class":15210},[151,63288,1876],{"class":1869},[151,63290,62229],{"class":503},[151,63292,16732],{"class":1869},[151,63294,62234],{"class":503},[151,63296,16417],{"class":1869},[151,63298,62239],{"class":503},[151,63300,22885],{"class":1869},[151,63302,485],{"class":503},[151,63304,63305,63307,63309,63311,63313,63315,63317,63319],{"class":469,"line":517},[151,63306,63284],{"class":503},[151,63308,62274],{"class":15210},[151,63310,1876],{"class":1869},[151,63312,62229],{"class":503},[151,63314,16732],{"class":1869},[151,63316,62234],{"class":503},[151,63318,16417],{"class":1869},[151,63320,62287],{"class":503},[151,63322,63323,63326,63328],{"class":469,"line":534},[151,63324,63325],{"class":503},"    filters ",[151,63327,1876],{"class":1869},[151,63329,62297],{"class":503},[151,63331,63332,63334,63336,63338,63340,63342,63344,63346,63348,63350,63352],{"class":469,"line":1413},[151,63333,63325],{"class":503},[151,63335,62304],{"class":1869},[151,63337,62307],{"class":12354},[151,63339,12386],{"class":503},[151,63341,43773],{"class":12347},[151,63343,27729],{"class":15232},[151,63345,106],{"class":503},[151,63347,25286],{"class":15232},[151,63349,62320],{"class":503},[151,63351,3947],{"class":1869},[151,63353,62325],{"class":503},[151,63355,63356,63358,63360],{"class":469,"line":1418},[151,63357,59706],{"class":503},[151,63359,1876],{"class":1869},[151,63361,62334],{"class":503},[11,63363,63364],{},"This is good for a small database, but could end up having to do lots of operations, especially if I decided that I want to add more fields to be searched, such as category or tags.",[11,63366,63367,63368,63371,63372,63375,63376,63378],{},"One option that I have considered to simplify the search query is to make one field called something like ",[30,63369,63370],{},"search_string",". This field would be modified on ",[30,63373,63374],{},"save",", and it would simply appends the text from all of the fields I'm interested in searching. Then, instead of doing Q lookups for each word over many different fields, I could search each word in ",[30,63377,61897],{}," over one field. I haven't implemented this here, but I would like to test this in the future.",[56,63380,63382],{"id":63381},"finding-books-nearby","Finding \"Books Nearby\"",[11,63384,63385],{},"On the pages that show details for each book I wanted to add additional information. I thought it would be interesting to show the 5 books closest to the book we are currently showing. Since we are dealing with coordinate points on a globe, the best way to calculate distance between two points is to use the Halversine formula:",[210,63387,63388],{},[11,63389,63390],{},"The haversine formula determines the great-circle distance between two points on a sphere given their longitudes and latitudes. Important in navigation, it is a special case of a more general formula in spherical trigonometry, the law of haversines, that relates the sides and angles of spherical triangles.",[11,63392,63393],{},[2718,63394],{"alt":20386,"src":63395},"/static/great_circle.png",[11,63397,63398],{},"Here's the code I used for calculating distance (using the Halversine formula):",[459,63400,63402],{"className":13136,"code":63401,"language":12886,"meta":464,"style":464},"from math import radians, cos, sin, asin, sqrt\n\ndef distance(origin, destination):\n    \"\"\"\n    Calculate the great circle distance between two points\n    on the earth (specified in decimal degrees)\n    \"\"\"\n\n    lon1, lat1 = origin\n    lon2, lat2 = destination\n    # convert decimal degrees to radians\n    lon1, lat1, lon2, lat2 = map(radians, [lon1, lat1, lon2, lat2])\n    # haversine formula\n    dlon = lon2 - lon1\n    dlat = lat2 - lat1\n    a = sin(dlat/2)**2 + cos(lat1) * cos(lat2) * sin(dlon/2)**2\n    c = 2 * asin(sqrt(a))\n    r = 3956 # miles\n    return c * r\n",[30,63403,63404,63416,63420,63439,63443,63448,63453,63457,63461,63471,63481,63486,63499,63504,63519,63534,63579,63593,63605],{"__ignoreMap":464},[151,63405,63406,63408,63411,63413],{"class":469,"line":470},[151,63407,16853],{"class":1869},[151,63409,63410],{"class":503}," math ",[151,63412,16859],{"class":1869},[151,63414,63415],{"class":503}," radians, cos, sin, asin, sqrt\n",[151,63417,63418],{"class":469,"line":488},[151,63419,1090],{"emptyLinePlaceholder":609},[151,63421,63422,63424,63427,63429,63432,63434,63437],{"class":469,"line":500},[151,63423,16925],{"class":12347},[151,63425,63426],{"class":473}," distance",[151,63428,12386],{"class":503},[151,63430,63431],{"class":15232},"origin",[151,63433,106],{"class":503},[151,63435,63436],{"class":15232},"destination",[151,63438,15264],{"class":503},[151,63440,63441],{"class":469,"line":509},[151,63442,17384],{"class":481},[151,63444,63445],{"class":469,"line":517},[151,63446,63447],{"class":481},"    Calculate the great circle distance between two points\n",[151,63449,63450],{"class":469,"line":534},[151,63451,63452],{"class":481},"    on the earth (specified in decimal degrees)\n",[151,63454,63455],{"class":469,"line":1413},[151,63456,17384],{"class":481},[151,63458,63459],{"class":469,"line":1418},[151,63460,1090],{"emptyLinePlaceholder":609},[151,63462,63463,63466,63468],{"class":469,"line":2462},[151,63464,63465],{"class":503},"    lon1, lat1 ",[151,63467,1876],{"class":1869},[151,63469,63470],{"class":503}," origin\n",[151,63472,63473,63476,63478],{"class":469,"line":2471},[151,63474,63475],{"class":503},"    lon2, lat2 ",[151,63477,1876],{"class":1869},[151,63479,63480],{"class":503}," destination\n",[151,63482,63483],{"class":469,"line":2480},[151,63484,63485],{"class":1527},"    # convert decimal degrees to radians\n",[151,63487,63488,63491,63493,63496],{"class":469,"line":2489},[151,63489,63490],{"class":503},"    lon1, lat1, lon2, lat2 ",[151,63492,1876],{"class":1869},[151,63494,63495],{"class":2226}," map",[151,63497,63498],{"class":503},"(radians, [lon1, lat1, lon2, lat2])\n",[151,63500,63501],{"class":469,"line":2497},[151,63502,63503],{"class":1527},"    # haversine formula\n",[151,63505,63506,63509,63511,63514,63516],{"class":469,"line":3140},[151,63507,63508],{"class":503},"    dlon ",[151,63510,1876],{"class":1869},[151,63512,63513],{"class":503}," lon2 ",[151,63515,12445],{"class":1869},[151,63517,63518],{"class":503}," lon1\n",[151,63520,63521,63524,63526,63529,63531],{"class":469,"line":3149},[151,63522,63523],{"class":503},"    dlat ",[151,63525,1876],{"class":1869},[151,63527,63528],{"class":503}," lat2 ",[151,63530,12445],{"class":1869},[151,63532,63533],{"class":503}," lat1\n",[151,63535,63536,63539,63541,63544,63546,63548,63550,63552,63554,63556,63559,63561,63564,63566,63569,63571,63573,63575,63577],{"class":469,"line":3158},[151,63537,63538],{"class":503},"    a ",[151,63540,1876],{"class":1869},[151,63542,63543],{"class":503}," sin(dlat",[151,63545,19883],{"class":1869},[151,63547,6619],{"class":477},[151,63549,748],{"class":503},[151,63551,24677],{"class":1869},[151,63553,6619],{"class":477},[151,63555,23378],{"class":1869},[151,63557,63558],{"class":503}," cos(lat1) ",[151,63560,23268],{"class":1869},[151,63562,63563],{"class":503}," cos(lat2) ",[151,63565,23268],{"class":1869},[151,63567,63568],{"class":503}," sin(dlon",[151,63570,19883],{"class":1869},[151,63572,6619],{"class":477},[151,63574,748],{"class":503},[151,63576,24677],{"class":1869},[151,63578,56807],{"class":477},[151,63580,63581,63584,63586,63588,63590],{"class":469,"line":3167},[151,63582,63583],{"class":503},"    c ",[151,63585,1876],{"class":1869},[151,63587,59070],{"class":477},[151,63589,12439],{"class":1869},[151,63591,63592],{"class":503}," asin(sqrt(a))\n",[151,63594,63595,63597,63599,63602],{"class":469,"line":3175},[151,63596,37133],{"class":503},[151,63598,1876],{"class":1869},[151,63600,63601],{"class":477}," 3956",[151,63603,63604],{"class":1527}," # miles\n",[151,63606,63607,63609,63612,63614],{"class":469,"line":3184},[151,63608,17496],{"class":1869},[151,63610,63611],{"class":503}," c ",[151,63613,23268],{"class":1869},[151,63615,63616],{"class":503}," r\n",[11,63618,63619],{},"This formula assumes that the earth is a perfect sphere. Another approach would be to use the the Vincety Formula:",[210,63621,63622],{},[11,63623,63624],{},"Vincenty's formulae are two related iterative methods used in geodesy to calculate the distance between two points on the surface of a spheroid, developed by Thaddeus Vincenty (1975a). They are based on the assumption that the figure of the Earth is an oblate spheroid, and hence are more accurate than methods that assume a spherical Earth, such as great-circle distance.",[11,63626,63627,63628,63631],{},"This distance is packaged in ",[30,63629,63630],{},"geopy"," and can be calculated easily:",[459,63633,63635],{"className":13136,"code":63634,"language":12886,"meta":464,"style":464},"import geopy.distance\n\ncoords_1 = (52.2296756, 21.0122287)\ncoords_2 = (52.406374, 16.9251681)\n\nprint geopy.distance.vincenty(coords_1, coords_2).km\n",[30,63636,63637,63644,63648,63667,63686,63690],{"__ignoreMap":464},[151,63638,63639,63641],{"class":469,"line":470},[151,63640,16859],{"class":1869},[151,63642,63643],{"class":503}," geopy.distance\n",[151,63645,63646],{"class":469,"line":488},[151,63647,1090],{"emptyLinePlaceholder":609},[151,63649,63650,63653,63655,63657,63660,63662,63665],{"class":469,"line":500},[151,63651,63652],{"class":503},"coords_1 ",[151,63654,1876],{"class":1869},[151,63656,129],{"class":503},[151,63658,63659],{"class":477},"52.2296756",[151,63661,106],{"class":503},[151,63663,63664],{"class":477},"21.0122287",[151,63666,3640],{"class":503},[151,63668,63669,63672,63674,63676,63679,63681,63684],{"class":469,"line":509},[151,63670,63671],{"class":503},"coords_2 ",[151,63673,1876],{"class":1869},[151,63675,129],{"class":503},[151,63677,63678],{"class":477},"52.406374",[151,63680,106],{"class":503},[151,63682,63683],{"class":477},"16.9251681",[151,63685,3640],{"class":503},[151,63687,63688],{"class":469,"line":517},[151,63689,1090],{"emptyLinePlaceholder":609},[151,63691,63692,63694],{"class":469,"line":534},[151,63693,18513],{"class":2226},[151,63695,63696],{"class":503}," geopy.distance.vincenty(coords_1, coords_2).km\n",[11,63698,63699,63700,63702],{},"Here's a look at the function I wrote for the ",[30,63701,61503],{}," view. I have commented out parts that aren't related to finding nearby books:",[459,63704,63706],{"className":13136,"code":63705,"language":12886,"meta":464,"style":464},"def book_detail(request, id, slug):\n\n    book = Book.objects.get(id=id, slug=slug)\n    b_coords = (book.lat, book.lon)\n    all_books = Book.objects.all()\n    coords = [((b.lat, b.lon),b) for b in all_books]\n\n    distance_dict = {}\n    for c in coords:\n        if c[0] != b_coords:\n            distance_dict[c[0]]=(distance(c[0],b_coords),c)\n\n    sorted_nearby = sorted(distance_dict.items(), key=lambda x: x[1][0])[:5]\n\n    # map_book = [{'loc':[float(book.lon), float(book.lat)],\n    #              'title':book.title,\n    #              'url':book.get_absolute_url()}]\n    context = {\n        'book':book,\n        # 'map_book':mark_safe(escapejs(json.dumps(map_book))),\n        'sorted_nearby':sorted_nearby,\n    }\n    return render(request, 'books/book_detail.html', context)\n",[30,63707,63708,63729,63733,63758,63768,63777,63796,63800,63809,63820,63836,63856,63860,63895,63899,63904,63909,63914,63922,63930,63935,63943,63947],{"__ignoreMap":464},[151,63709,63710,63712,63715,63717,63719,63721,63723,63725,63727],{"class":469,"line":470},[151,63711,16925],{"class":12347},[151,63713,63714],{"class":473}," book_detail",[151,63716,12386],{"class":503},[151,63718,59686],{"class":15232},[151,63720,106],{"class":503},[151,63722,47409],{"class":15232},[151,63724,106],{"class":503},[151,63726,19918],{"class":15232},[151,63728,15264],{"class":503},[151,63730,63731],{"class":469,"line":488},[151,63732,1090],{"emptyLinePlaceholder":609},[151,63734,63735,63738,63740,63743,63745,63747,63749,63751,63753,63755],{"class":469,"line":500},[151,63736,63737],{"class":503},"    book ",[151,63739,1876],{"class":1869},[151,63741,63742],{"class":503}," Book.objects.get(",[151,63744,47409],{"class":15210},[151,63746,1876],{"class":1869},[151,63748,47409],{"class":2226},[151,63750,106],{"class":503},[151,63752,19918],{"class":15210},[151,63754,1876],{"class":1869},[151,63756,63757],{"class":503},"slug)\n",[151,63759,63760,63763,63765],{"class":469,"line":509},[151,63761,63762],{"class":503},"    b_coords ",[151,63764,1876],{"class":1869},[151,63766,63767],{"class":503}," (book.lat, book.lon)\n",[151,63769,63770,63773,63775],{"class":469,"line":517},[151,63771,63772],{"class":503},"    all_books ",[151,63774,1876],{"class":1869},[151,63776,59711],{"class":503},[151,63778,63779,63782,63784,63787,63789,63791,63793],{"class":469,"line":534},[151,63780,63781],{"class":503},"    coords ",[151,63783,1876],{"class":1869},[151,63785,63786],{"class":503}," [((b.lat, b.lon),b) ",[151,63788,16732],{"class":1869},[151,63790,58916],{"class":503},[151,63792,16417],{"class":1869},[151,63794,63795],{"class":503}," all_books]\n",[151,63797,63798],{"class":469,"line":1413},[151,63799,1090],{"emptyLinePlaceholder":609},[151,63801,63802,63805,63807],{"class":469,"line":1418},[151,63803,63804],{"class":503},"    distance_dict ",[151,63806,1876],{"class":1869},[151,63808,16634],{"class":503},[151,63810,63811,63813,63815,63817],{"class":469,"line":2462},[151,63812,16411],{"class":1869},[151,63814,63611],{"class":503},[151,63816,16417],{"class":1869},[151,63818,63819],{"class":503}," coords:\n",[151,63821,63822,63824,63827,63829,63831,63833],{"class":469,"line":2471},[151,63823,23357],{"class":1869},[151,63825,63826],{"class":503}," c[",[151,63828,9181],{"class":477},[151,63830,16654],{"class":503},[151,63832,58602],{"class":1869},[151,63834,63835],{"class":503}," b_coords:\n",[151,63837,63838,63841,63843,63846,63848,63851,63853],{"class":469,"line":2480},[151,63839,63840],{"class":503},"            distance_dict[c[",[151,63842,9181],{"class":477},[151,63844,63845],{"class":503},"]]",[151,63847,1876],{"class":1869},[151,63849,63850],{"class":503},"(distance(c[",[151,63852,9181],{"class":477},[151,63854,63855],{"class":503},"],b_coords),c)\n",[151,63857,63858],{"class":469,"line":2489},[151,63859,1090],{"emptyLinePlaceholder":609},[151,63861,63862,63865,63867,63869,63872,63874,63876,63878,63880,63882,63884,63886,63888,63891,63893],{"class":469,"line":2497},[151,63863,63864],{"class":503},"    sorted_nearby ",[151,63866,1876],{"class":1869},[151,63868,44767],{"class":2226},[151,63870,63871],{"class":503},"(distance_dict.items(), ",[151,63873,18175],{"class":15210},[151,63875,1876],{"class":1869},[151,63877,43773],{"class":12347},[151,63879,27729],{"class":15232},[151,63881,44811],{"class":503},[151,63883,6760],{"class":477},[151,63885,6704],{"class":503},[151,63887,9181],{"class":477},[151,63889,63890],{"class":503},"])[:",[151,63892,24380],{"class":477},[151,63894,3691],{"class":503},[151,63896,63897],{"class":469,"line":3140},[151,63898,1090],{"emptyLinePlaceholder":609},[151,63900,63901],{"class":469,"line":3149},[151,63902,63903],{"class":1527},"    # map_book = [{'loc':[float(book.lon), float(book.lat)],\n",[151,63905,63906],{"class":469,"line":3158},[151,63907,63908],{"class":1527},"    #              'title':book.title,\n",[151,63910,63911],{"class":469,"line":3167},[151,63912,63913],{"class":1527},"    #              'url':book.get_absolute_url()}]\n",[151,63915,63916,63918,63920],{"class":469,"line":3175},[151,63917,59844],{"class":503},[151,63919,1876],{"class":1869},[151,63921,19833],{"class":503},[151,63923,63924,63927],{"class":469,"line":3184},[151,63925,63926],{"class":481},"        'book'",[151,63928,63929],{"class":503},":book,\n",[151,63931,63932],{"class":469,"line":3193},[151,63933,63934],{"class":1527},"        # 'map_book':mark_safe(escapejs(json.dumps(map_book))),\n",[151,63936,63937,63940],{"class":469,"line":3720},[151,63938,63939],{"class":481},"        'sorted_nearby'",[151,63941,63942],{"class":503},":sorted_nearby,\n",[151,63944,63945],{"class":469,"line":3729},[151,63946,9461],{"class":503},[151,63948,63949,63951,63953,63956],{"class":469,"line":3735},[151,63950,17496],{"class":1869},[151,63952,59902],{"class":503},[151,63954,63955],{"class":481},"'books/book_detail.html'",[151,63957,59908],{"class":503},[11,63959,63960,63961,63964,63965,63968,63969,63972,63973,33,63975,63978,63979,26802,63982,63985,63986,22696,63988,63991,63992,63995,63996,63999,64000,64003,64004,64007],{},"To find the 5 closest books I arrived at a solution that seems fairly convoluted and should be refactored, but works! I start with a queryset of all books, then use list comprehension to make a list of tuples containing ",[30,63962,63963],{},"((\u003C longitutde >, \u003C latitude >), \u003C Book Object >)",". Then I loop over this list and create a dictionary where the keys are ",[30,63966,63967],{},"(\u003C longitutde >, \u003C latitude >)"," and the values are tuples of the form: ",[30,63970,63971],{},"(\u003C distance in miles >, \u003C Book Object >)",". Finally, I use ",[30,63974,43956],{},[30,63976,63977],{},"dictionary_dict.items()"," where the key is ",[30,63980,63981],{},"lambda x: x[1][0]",[30,63983,63984],{},"[1]"," accesses the ",[30,63987,18184],{},[30,63989,63990],{},"(\u003C key >, \u003C value>)"," tuple returned by ",[30,63993,63994],{},"items()",", and the ",[30,63997,63998],{},"[0]"," access the first item, which is the ",[30,64001,64002],{},"distance"," value we calculated. Finally, I take the first ",[30,64005,64006],{},"[:5]"," items of this sorted list.",[11,64009,64010],{},"With this approach I think I avoided the possible issue of ambiguity if we have two books with the same coordinates. I'll need to add this scenario to my test suite later.",[56,64012,64014],{"id":64013},"testing-travis-ci","Testing & Travis CI",[11,64016,64017],{},"Speaking of testing, this applcation includes a simple testing suite (that I am currently working on expanding). I have also managed to setup Travis CI for this project. When I push code from my local repo to GitHub, Travis CI automatically runs test, and the commit message contains additional information about whether or not all of the tests were successful.",[11,64019,64020,64021,64024,64025,64027],{},"Setting up Travis CI is very simple. We need to grant Travis CI (.org) access to our GitHub account, enable Travis CI on the repository we are working with, and then add a ",[30,64022,64023],{},".travis.yml"," file to the top level of the project directory. Here's the ",[30,64026,64023],{}," file I have for this project:",[459,64029,64031],{"className":21928,"code":64030,"language":21930,"meta":464,"style":464},"language: python\n\npython:\n  - '3.5'\n\nservices:\n  - postgresql\n\nenv: -DJANGO=2.0 DB=postgresql\n\ninstall:\n  - pip install -r requirements.txt\n\nbefore_script:\n  - psql -c \"CREATE USER u_brian WITH PASSWORD 'password'; ALTER USER u_brian CREATEDB;\" -U postgres\n\nscript:\n  - python manage.py test books/\n",[30,64032,64033,64042,64046,64052,64059,64063,64069,64076,64080,64090,64094,64101,64108,64112,64118,64125,64129,64135],{"__ignoreMap":464},[151,64034,64035,64037,64039],{"class":469,"line":470},[151,64036,26122],{"class":14368},[151,64038,6208],{"class":503},[151,64040,64041],{"class":481},"python\n",[151,64043,64044],{"class":469,"line":488},[151,64045,1090],{"emptyLinePlaceholder":609},[151,64047,64048,64050],{"class":469,"line":500},[151,64049,12886],{"class":14368},[151,64051,14372],{"class":503},[151,64053,64054,64056],{"class":469,"line":509},[151,64055,19688],{"class":503},[151,64057,64058],{"class":481},"'3.5'\n",[151,64060,64061],{"class":469,"line":517},[151,64062,1090],{"emptyLinePlaceholder":609},[151,64064,64065,64067],{"class":469,"line":534},[151,64066,15643],{"class":14368},[151,64068,14372],{"class":503},[151,64070,64071,64073],{"class":469,"line":1413},[151,64072,19688],{"class":503},[151,64074,64075],{"class":481},"postgresql\n",[151,64077,64078],{"class":469,"line":1418},[151,64079,1090],{"emptyLinePlaceholder":609},[151,64081,64082,64085,64087],{"class":469,"line":2462},[151,64083,64084],{"class":14368},"env",[151,64086,6208],{"class":503},[151,64088,64089],{"class":481},"-DJANGO=2.0 DB=postgresql\n",[151,64091,64092],{"class":469,"line":2471},[151,64093,1090],{"emptyLinePlaceholder":609},[151,64095,64096,64099],{"class":469,"line":2480},[151,64097,64098],{"class":14368},"install",[151,64100,14372],{"class":503},[151,64102,64103,64105],{"class":469,"line":2489},[151,64104,19688],{"class":503},[151,64106,64107],{"class":481},"pip install -r requirements.txt\n",[151,64109,64110],{"class":469,"line":2497},[151,64111,1090],{"emptyLinePlaceholder":609},[151,64113,64114,64116],{"class":469,"line":3140},[151,64115,49813],{"class":14368},[151,64117,14372],{"class":503},[151,64119,64120,64122],{"class":469,"line":3149},[151,64121,19688],{"class":503},[151,64123,64124],{"class":481},"psql -c \"CREATE USER u_brian WITH PASSWORD 'password'; ALTER USER u_brian CREATEDB;\" -U postgres\n",[151,64126,64127],{"class":469,"line":3158},[151,64128,1090],{"emptyLinePlaceholder":609},[151,64130,64131,64133],{"class":469,"line":3167},[151,64132,19822],{"class":14368},[151,64134,14372],{"class":503},[151,64136,64137,64139],{"class":469,"line":3175},[151,64138,19688],{"class":503},[151,64140,64141],{"class":481},"python manage.py test books/\n",[11,64143,64144],{},"We can generate a Travis CI badge that shows the status of the current Github deploy with the following code:",[459,64146,64148],{"className":19811,"code":64147,"language":19813,"meta":464,"style":464},"[![Build\nStatus](https://travis-ci.org/briancaffey/django-leaflet-demo.svg?branch=master)](https://travis-ci.org/briancaffey/django-leaflet-demo)\n",[30,64149,64150,64155],{"__ignoreMap":464},[151,64151,64152],{"class":469,"line":470},[151,64153,64154],{"class":503},"[![Build\n",[151,64156,64157],{"class":469,"line":488},[151,64158,64159],{"class":503},"Status](https://travis-ci.org/briancaffey/django-leaflet-demo.svg?branch=master)](https://travis-ci.org/briancaffey/django-leaflet-demo)\n",[11,64161,64162],{},[20,64163,64166],{"href":64164,"rel":64165},"https://travis-ci.org/briancaffey/django-leaflet-demo",[24],[2718,64167],{"alt":64168,"src":64169},"Build Status","https://travis-ci.org/briancaffey/django-leaflet-demo.svg?branch=master",[56,64171,64173],{"id":64172},"deploying-to-digitalocean","Deploying to DigitalOcean",[11,64175,64176],{},"Finally, I used DigitalOcean to deploy this app. I have used Heroku for most of my previous projects, but I decided to use DigitalOcean for this one to learn something new and get more experience with using Ubuntu and related tools for running a website: nginx and gunicorn.",[11,64178,64179,64180,64184,64185,187,64187,64189],{},"Again, for I turned to Vitor's blog for a very straightforward introduction to deploying on DigitalOcean with a simple Droplet. You can read more about the instructions for deployment ",[20,64181,13074],{"href":64182,"rel":64183},"https://simpleisbetterthancomplex.com/tutorial/2016/10/14/how-to-deploy-to-digital-ocean.html",[24],", and I can say that I had no problems following the instructions step by step. Here are the ",[30,64186,56146],{},[30,64188,49765],{}," scripts I have used to successfully deploy my project:",[11,64191,64192],{},[51,64193,64194],{},"/home/brian/bin/gunicorn_start",[459,64196,64198],{"className":461,"code":64197,"language":463,"meta":464,"style":464},"#!/bin/bash\n\nNAME=\"django-leaflet-demo\"\nDIR=/home/brian/django-leaflet-demo\nUSER=brian\nGROUP=brian\nWORKERS=3\nBIND=unix:/home/brian/run/gunicorn.sock\nDJANGO_SETTINGS_MODULE=djangoapp.settings\nDJANGO_WSGI_MODULE=djangoapp.wsgi\nLOG_LEVEL=error\ncd $DIR\nsource ../bin/activate\n\nexport DJANGO_SETTINGS_MODULE=$DJANGO_SETTINGS_MODULE\nexport PYTHONPATH=$DIR:$PYTHONPATH\n\nexec ../bin/gunicorn ${DJANGO_WSGI_MODULE}:application \\\n  --name=$NAME \\\n  --workers=$WORKERS \\\n  --user=$USER \\\n  --group=$GROUP \\\n  --bind=$BIND \\\n  --log-level=$LOG_LEVEL \\\n  --log-file=-\n",[30,64199,64200,64204,64208,64217,64227,64237,64246,64256,64266,64276,64286,64296,64304,64311,64315,64327,64339,64343,64359,64369,64379,64389,64399,64409,64419],{"__ignoreMap":464},[151,64201,64202],{"class":469,"line":470},[151,64203,31244],{"class":1527},[151,64205,64206],{"class":469,"line":488},[151,64207,1090],{"emptyLinePlaceholder":609},[151,64209,64210,64212,64214],{"class":469,"line":500},[151,64211,5422],{"class":503},[151,64213,1876],{"class":1869},[151,64215,64216],{"class":481},"\"django-leaflet-demo\"\n",[151,64218,64219,64222,64224],{"class":469,"line":509},[151,64220,64221],{"class":503},"DIR",[151,64223,1876],{"class":1869},[151,64225,64226],{"class":481},"/home/brian/django-leaflet-demo\n",[151,64228,64229,64232,64234],{"class":469,"line":517},[151,64230,64231],{"class":503},"USER",[151,64233,1876],{"class":1869},[151,64235,64236],{"class":481},"brian\n",[151,64238,64239,64242,64244],{"class":469,"line":534},[151,64240,64241],{"class":503},"GROUP",[151,64243,1876],{"class":1869},[151,64245,64236],{"class":481},[151,64247,64248,64251,64253],{"class":469,"line":1413},[151,64249,64250],{"class":503},"WORKERS",[151,64252,1876],{"class":1869},[151,64254,64255],{"class":481},"3\n",[151,64257,64258,64261,64263],{"class":469,"line":1418},[151,64259,64260],{"class":503},"BIND",[151,64262,1876],{"class":1869},[151,64264,64265],{"class":481},"unix:/home/brian/run/gunicorn.sock\n",[151,64267,64268,64271,64273],{"class":469,"line":2462},[151,64269,64270],{"class":503},"DJANGO_SETTINGS_MODULE",[151,64272,1876],{"class":1869},[151,64274,64275],{"class":481},"djangoapp.settings\n",[151,64277,64278,64281,64283],{"class":469,"line":2471},[151,64279,64280],{"class":503},"DJANGO_WSGI_MODULE",[151,64282,1876],{"class":1869},[151,64284,64285],{"class":481},"djangoapp.wsgi\n",[151,64287,64288,64291,64293],{"class":469,"line":2480},[151,64289,64290],{"class":503},"LOG_LEVEL",[151,64292,1876],{"class":1869},[151,64294,64295],{"class":481},"error\n",[151,64297,64298,64301],{"class":469,"line":2489},[151,64299,64300],{"class":2226},"cd",[151,64302,64303],{"class":503}," $DIR\n",[151,64305,64306,64308],{"class":469,"line":2497},[151,64307,23905],{"class":2226},[151,64309,64310],{"class":481}," ../bin/activate\n",[151,64312,64313],{"class":469,"line":3140},[151,64314,1090],{"emptyLinePlaceholder":609},[151,64316,64317,64319,64322,64324],{"class":469,"line":3149},[151,64318,1870],{"class":1869},[151,64320,64321],{"class":503}," DJANGO_SETTINGS_MODULE",[151,64323,1876],{"class":1869},[151,64325,64326],{"class":503},"$DJANGO_SETTINGS_MODULE\n",[151,64328,64329,64331,64334,64336],{"class":469,"line":3158},[151,64330,1870],{"class":1869},[151,64332,64333],{"class":503}," PYTHONPATH",[151,64335,1876],{"class":1869},[151,64337,64338],{"class":503},"$DIR:$PYTHONPATH\n",[151,64340,64341],{"class":469,"line":3167},[151,64342,1090],{"emptyLinePlaceholder":609},[151,64344,64345,64348,64351,64354,64357],{"class":469,"line":3175},[151,64346,64347],{"class":2226},"exec",[151,64349,64350],{"class":481}," ../bin/gunicorn",[151,64352,64353],{"class":503}," ${DJANGO_WSGI_MODULE}",[151,64355,64356],{"class":481},":application",[151,64358,485],{"class":477},[151,64360,64361,64364,64367],{"class":469,"line":3184},[151,64362,64363],{"class":477},"  --name=",[151,64365,64366],{"class":503},"$NAME",[151,64368,485],{"class":477},[151,64370,64371,64374,64377],{"class":469,"line":3193},[151,64372,64373],{"class":477},"  --workers=",[151,64375,64376],{"class":503},"$WORKERS",[151,64378,485],{"class":477},[151,64380,64381,64384,64387],{"class":469,"line":3720},[151,64382,64383],{"class":477},"  --user=",[151,64385,64386],{"class":503},"$USER",[151,64388,485],{"class":477},[151,64390,64391,64394,64397],{"class":469,"line":3729},[151,64392,64393],{"class":477},"  --group=",[151,64395,64396],{"class":503},"$GROUP",[151,64398,485],{"class":477},[151,64400,64401,64404,64407],{"class":469,"line":3735},[151,64402,64403],{"class":477},"  --bind=",[151,64405,64406],{"class":503},"$BIND",[151,64408,485],{"class":477},[151,64410,64411,64414,64417],{"class":469,"line":3745},[151,64412,64413],{"class":477},"  --log-level=",[151,64415,64416],{"class":503},"$LOG_LEVEL",[151,64418,485],{"class":477},[151,64420,64421],{"class":469,"line":3754},[151,64422,64423],{"class":477},"  --log-file=-\n",[11,64425,64426],{},[51,64427,64428],{},"/etc/nginx/sites-available",[459,64430,64432],{"className":21928,"code":64431,"language":21930,"meta":464,"style":464},"upstream app_server {\nserver unix:/home/brian/run/gunicorn.sock fail_timeout=0;\n}\n\nserver {\nlisten 80;\nserver_name 159.89.235.193\nkeepalive_timeout 5;\nclient_max_body_size 4G;\naccess_log /home/brian/logs/nginx-access.log;\nerror_log /home/brian/logs/nginx-error.log;\n\nlocation /static/ {\nalias /home/brian/static/;\n}\n\nlocation / {\ntry_files $uri @proxy_to_app;\n}\n\nlocation @proxy_to_app {\nproxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;\nproxy_set_header Host $http_host;\nproxy_redirect off;\nproxy_pass http://app_server;\n}\n}\n",[30,64433,64434,64439,64444,64448,64452,64457,64462,64467,64472,64477,64482,64487,64491,64496,64501,64505,64509,64514,64519,64523,64527,64532,64537,64542,64547,64552,64556],{"__ignoreMap":464},[151,64435,64436],{"class":469,"line":470},[151,64437,64438],{"class":481},"upstream app_server {\n",[151,64440,64441],{"class":469,"line":488},[151,64442,64443],{"class":481},"server unix:/home/brian/run/gunicorn.sock fail_timeout=0;\n",[151,64445,64446],{"class":469,"line":500},[151,64447,6274],{"class":503},[151,64449,64450],{"class":469,"line":509},[151,64451,1090],{"emptyLinePlaceholder":609},[151,64453,64454],{"class":469,"line":517},[151,64455,64456],{"class":481},"server {\n",[151,64458,64459],{"class":469,"line":534},[151,64460,64461],{"class":481},"listen 80;\n",[151,64463,64464],{"class":469,"line":1413},[151,64465,64466],{"class":481},"server_name 159.89.235.193\n",[151,64468,64469],{"class":469,"line":1418},[151,64470,64471],{"class":481},"keepalive_timeout 5;\n",[151,64473,64474],{"class":469,"line":2462},[151,64475,64476],{"class":481},"client_max_body_size 4G;\n",[151,64478,64479],{"class":469,"line":2471},[151,64480,64481],{"class":481},"access_log /home/brian/logs/nginx-access.log;\n",[151,64483,64484],{"class":469,"line":2480},[151,64485,64486],{"class":481},"error_log /home/brian/logs/nginx-error.log;\n",[151,64488,64489],{"class":469,"line":2489},[151,64490,1090],{"emptyLinePlaceholder":609},[151,64492,64493],{"class":469,"line":2497},[151,64494,64495],{"class":481},"location /static/ {\n",[151,64497,64498],{"class":469,"line":3140},[151,64499,64500],{"class":481},"alias /home/brian/static/;\n",[151,64502,64503],{"class":469,"line":3149},[151,64504,6274],{"class":503},[151,64506,64507],{"class":469,"line":3158},[151,64508,1090],{"emptyLinePlaceholder":609},[151,64510,64511],{"class":469,"line":3167},[151,64512,64513],{"class":481},"location / {\n",[151,64515,64516],{"class":469,"line":3175},[151,64517,64518],{"class":481},"try_files $uri @proxy_to_app;\n",[151,64520,64521],{"class":469,"line":3184},[151,64522,6274],{"class":503},[151,64524,64525],{"class":469,"line":3193},[151,64526,1090],{"emptyLinePlaceholder":609},[151,64528,64529],{"class":469,"line":3720},[151,64530,64531],{"class":481},"location @proxy_to_app {\n",[151,64533,64534],{"class":469,"line":3729},[151,64535,64536],{"class":481},"proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;\n",[151,64538,64539],{"class":469,"line":3735},[151,64540,64541],{"class":481},"proxy_set_header Host $http_host;\n",[151,64543,64544],{"class":469,"line":3745},[151,64545,64546],{"class":481},"proxy_redirect off;\n",[151,64548,64549],{"class":469,"line":3754},[151,64550,64551],{"class":481},"proxy_pass http://app_server;\n",[151,64553,64554],{"class":469,"line":3760},[151,64555,6274],{"class":503},[151,64557,64558],{"class":469,"line":3773},[151,64559,6274],{"class":503},[11,64561,64562],{},"Finally, here is the Supervisor configuration file that runs the gunicorn server:",[459,64564,64567],{"className":64565,"code":64566,"language":997},[995],"[program:django-leaflet-demo]\ncommand=/home/brian/bin/gunicorn_start\nuser=brian\nautostart=true\nautorestart=true\nredirect_stderr=true\nstdout_logfile=/home/brian/logs/gunicorn-error.log\n",[30,64568,64566],{"__ignoreMap":464},[11,64570,64571],{},"I would definitely like to dive into DigitalOcean deployment in more depth in my next post.",[11,64573,64574,64575,64578],{},"For now, you can view the live project on DigitalOcean here: ",[20,64576,59403],{"href":59403,"rel":64577},[24],". In the future I would like to use this Droplet to do more demo apps like this one as I continue to learn the ins and outs of using Django and more frontend tools. Thanks for reading to the end and let me know if you have any comments or critiques on how I went about this project, I would be happy to hear from you!",[589,64580,64581],{},"html pre.shiki code .sXSQT, html code.shiki .sXSQT{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#F8F8F2}html pre.shiki code .sMOD_, html code.shiki .sMOD_{--shiki-default:#24292E;--shiki-dark:#E1E4E8;--shiki-sepia:#F8F8F2}html pre.shiki code .srTi1, html code.shiki .srTi1{--shiki-default:#6F42C1;--shiki-dark:#B392F0;--shiki-sepia:#A6E22E}html pre.shiki code .sstjo, html code.shiki .sstjo{--shiki-default:#032F62;--shiki-dark:#9ECBFF;--shiki-sepia:#E6DB74}html pre.shiki code .s7F3e, html code.shiki .s7F3e{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#AE81FF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html .sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html.sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html pre.shiki code .sq6CD, html code.shiki .sq6CD{--shiki-default:#D73A49;--shiki-default-font-style:inherit;--shiki-dark:#F97583;--shiki-dark-font-style:inherit;--shiki-sepia:#66D9EF;--shiki-sepia-font-style:italic}html pre.shiki code .sz2Vg, html code.shiki .sz2Vg{--shiki-default:#6F42C1;--shiki-default-text-decoration:inherit;--shiki-dark:#B392F0;--shiki-dark-text-decoration:inherit;--shiki-sepia:#A6E22E;--shiki-sepia-text-decoration:underline}html pre.shiki code .s30JN, html code.shiki .s30JN{--shiki-default:#6F42C1;--shiki-default-font-style:inherit;--shiki-default-text-decoration:inherit;--shiki-dark:#B392F0;--shiki-dark-font-style:inherit;--shiki-dark-text-decoration:inherit;--shiki-sepia:#A6E22E;--shiki-sepia-font-style:italic;--shiki-sepia-text-decoration:underline}html pre.shiki code .sC2Qs, html code.shiki .sC2Qs{--shiki-default:#D73A49;--shiki-dark:#F97583;--shiki-sepia:#F92672}html pre.shiki code .sTHNf, html code.shiki .sTHNf{--shiki-default:#E36209;--shiki-default-font-style:inherit;--shiki-dark:#FFAB70;--shiki-dark-font-style:inherit;--shiki-sepia:#FD971F;--shiki-sepia-font-style:italic}html pre.shiki code .so59x, html code.shiki .so59x{--shiki-default:#24292E;--shiki-default-font-style:inherit;--shiki-dark:#E1E4E8;--shiki-dark-font-style:inherit;--shiki-sepia:#FD971F;--shiki-sepia-font-style:italic}html pre.shiki code .s8-w5, html code.shiki .s8-w5{--shiki-default:#6A737D;--shiki-dark:#6A737D;--shiki-sepia:#88846F}html pre.shiki code .s-m8C, html code.shiki .s-m8C{--shiki-default:#005CC5;--shiki-default-font-style:inherit;--shiki-dark:#79B8FF;--shiki-dark-font-style:inherit;--shiki-sepia:#66D9EF;--shiki-sepia-font-style:italic}html pre.shiki code .s5clZ, html code.shiki .s5clZ{--shiki-default:#22863A;--shiki-dark:#85E89D;--shiki-sepia:#F92672}html pre.shiki code .sTrkL, html code.shiki .sTrkL{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#66D9EF}html pre.shiki code .sOrwc, html code.shiki .sOrwc{--shiki-default:#E36209;--shiki-dark:#FFAB70;--shiki-sepia:#F8F8F2}html pre.shiki code .sP7S_, html code.shiki .sP7S_{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#FD971F}",{"title":464,"searchDepth":488,"depth":488,"links":64583},[64584,64585,64586,64587,64588,64589,64590,64591,64592,64593],{"id":59400,"depth":500,"text":59405},{"id":59534,"depth":488,"text":59535},{"id":59653,"depth":488,"text":59654},{"id":60733,"depth":488,"text":60734},{"id":61158,"depth":488,"text":61159},{"id":61575,"depth":488,"text":61576},{"id":62461,"depth":488,"text":62462},{"id":63381,"depth":488,"text":63382},{"id":64013,"depth":488,"text":64014},{"id":64172,"depth":488,"text":64173},"2018-02-19","/static/map_homepage.png",{"layout":48045,"disqus_id":64597},"/2018/02/19/leaflet-maps-with-django.html","/2018/02/19/leaflet-maps-with-django",{"title":59390,"description":464},"2018/02/19/leaflet-maps-with-django",[64602,30122,64603,64604,64605,64606],"leaflet","mapbox","data-tables","bootstrap","travis-ci","Xt_ccnWuMfb6U5MH_8efVqQkdMDyMBRNzt_X6mbe8mI",{"id":64609,"title":64610,"body":64611,"comments":609,"date":65986,"description":65987,"draft":602,"extension":605,"external":606,"image":65988,"meta":65989,"navigation":609,"path":65990,"seo":65991,"stem":65992,"tags":65993,"__hash__":65995},"blog/2018/01/30/reading-13f-sec-filings-with-python.md","Reading 13F SEC filings with python",{"type":8,"value":64612,"toc":65983},[64613,64621,64638,64648,64651,64660,64670,64676,64687,64690,64735,64741,64744,65032,65035,65174,65179,65183,65186,65413,65424,65741,65744,65750,65756,65759,65975,65980],[11,64614,64615,64617,64618,64620],{},[15,64616,21589],{},": This project has been updated, please see ",[20,64619,49337],{"href":47667}," to read about the most recent updates.",[76,64622,64623,64628,64633],{},[79,64624,64625,47549],{},[20,64626,47547],{"href":47547,"rel":64627},[24],[79,64629,64630,47555],{},[20,64631,46158],{"href":46158,"rel":64632},[24],[79,64634,64635,47562],{},[20,64636,47560],{"href":47560,"rel":64637},[24],[210,64639,64640],{},[11,64641,64642,64643],{},"The SEC Form 13F is a filing with the Securities and Exchange Commission (SEC) also known as the Information Required of Institutional Investment Managers Form. It is a quarterly filing required of institutional investment managers with over $100 million in qualifying assets. -",[20,64644,64647],{"href":64645,"rel":64646},"https://www.investopedia.com/terms/f/form-13f.asp",[24],"Investopedia",[11,64649,64650],{},"In this article I will show how to collect and parse 13F filing data from the SEC.",[11,64652,64653,64654,64659],{},"First, use ",[20,64655,64658],{"href":64656,"rel":64657},"https://www.sec.gov/edgar/searchedgar/companysearch.html",[24],"EDGAR"," to search the company of interest.",[210,64661,64662],{},[11,64663,64664,64665],{},"EDGAR, the Electronic Data Gathering, Analysis, and Retrieval system, performs automated collection, validation, indexing, acceptance, and forwarding of submissions by companies and others who are required by law to file forms with the U.S. Securities and Exchange Commission (the \"SEC\"). -",[20,64666,64669],{"href":64667,"rel":64668},"https://en.wikipedia.org/wiki/EDGAR",[24],"Wikipedia",[11,64671,64672,64673,643],{},"Click on the Central Index Key (CIK) of the company you are search for, and then click on ",[30,64674,64675],{},"Documents",[11,64677,64678,64679,64682,64683,64686],{},"You'll want to grab the HTML version of the ",[30,64680,64681],{},"Information Table",". I have saved them in a folder with their file names cooresponding to their dates (",[30,64684,64685],{},"YYYY-MM-DD"," format).",[11,64688,64689],{},"For this example, I have manually collected the files for a few years of data filed by a hedge fund. Here are the files I'll be working with:",[459,64691,64693],{"className":13136,"code":64692,"language":12886,"meta":464,"style":464},"files = os.listdir(\"13f/\")\nprint(*sorted(files), sep=\"\\n\")\n",[30,64694,64695,64709],{"__ignoreMap":464},[151,64696,64697,64700,64702,64704,64707],{"class":469,"line":470},[151,64698,64699],{"class":503},"files ",[151,64701,1876],{"class":1869},[151,64703,44561],{"class":503},[151,64705,64706],{"class":481},"\"13f/\"",[151,64708,3640],{"class":503},[151,64710,64711,64713,64715,64717,64719,64722,64725,64727,64729,64731,64733],{"class":469,"line":488},[151,64712,18513],{"class":2226},[151,64714,12386],{"class":503},[151,64716,23268],{"class":1869},[151,64718,43956],{"class":2226},[151,64720,64721],{"class":503},"(files), ",[151,64723,64724],{"class":15210},"sep",[151,64726,1876],{"class":1869},[151,64728,8592],{"class":481},[151,64730,8043],{"class":477},[151,64732,8592],{"class":481},[151,64734,3640],{"class":503},[459,64736,64739],{"className":64737,"code":64738,"language":997},[995],"2014-02-14.html\n2014-05-15.html\n2014-08-14.html\n2014-11-14.html\n2015-02-17.html\n2015-05-14.html\n2015-08-14.html\n2015-11-12.html\n2016-02-16.html\n2016-05-16.html\n2016-08-12.html\n2016-11-14.html\n2017-02-14.html\n2017-05-15.html\n2017-08-10.html\n2017-10-30.html\n",[30,64740,64738],{"__ignoreMap":464},[11,64742,64743],{},"Here's a quick script we can use to parse information from each filing document:",[459,64745,64747],{"className":13136,"code":64746,"language":12886,"meta":464,"style":464},"def scrape_13f(file):\n    date = file\n    html = open(\"13f/\"+file).read()\n    soup = BeautifulSoup(html, 'lxml')\n    rows = soup.find_all('tr')[11:]\n    positions = []\n    for row in rows:\n        dic = {}\n        position = row.find_all('td')\n        dic[\"NAME_OF_ISSUER\"] = position[0].text\n        dic[\"TITLE_OF_CLASS\"] = position[1].text\n        dic[\"CUSIP\"] = position[2].text\n        dic[\"VALUE\"] = int(position[3].text.replace(',', ''))*1000\n        dic[\"SHARES\"] = int(position[4].text.replace(',', ''))\n        dic[\"DATE\"] = date.strip(\".html\")\n        positions.append(dic)\n\n    df = pd.DataFrame(positions)\n    return df\n",[30,64748,64749,64762,64772,64792,64807,64826,64835,64847,64856,64871,64891,64908,64925,64960,64987,65006,65011,65015,65025],{"__ignoreMap":464},[151,64750,64751,64753,64756,64758,64760],{"class":469,"line":470},[151,64752,16925],{"class":12347},[151,64754,64755],{"class":473}," scrape_13f",[151,64757,12386],{"class":503},[151,64759,56116],{"class":15232},[151,64761,15264],{"class":503},[151,64763,64764,64767,64769],{"class":469,"line":488},[151,64765,64766],{"class":503},"    date ",[151,64768,1876],{"class":1869},[151,64770,64771],{"class":12354}," file\n",[151,64773,64774,64777,64779,64781,64783,64785,64787,64789],{"class":469,"line":500},[151,64775,64776],{"class":503},"    html ",[151,64778,1876],{"class":1869},[151,64780,16970],{"class":2226},[151,64782,12386],{"class":503},[151,64784,64706],{"class":481},[151,64786,22885],{"class":1869},[151,64788,56116],{"class":12354},[151,64790,64791],{"class":503},").read()\n",[151,64793,64794,64797,64799,64802,64805],{"class":469,"line":509},[151,64795,64796],{"class":503},"    soup ",[151,64798,1876],{"class":1869},[151,64800,64801],{"class":503}," BeautifulSoup(html, ",[151,64803,64804],{"class":481},"'lxml'",[151,64806,3640],{"class":503},[151,64808,64809,64812,64814,64817,64820,64822,64824],{"class":469,"line":517},[151,64810,64811],{"class":503},"    rows ",[151,64813,1876],{"class":1869},[151,64815,64816],{"class":503}," soup.find_all(",[151,64818,64819],{"class":481},"'tr'",[151,64821,40832],{"class":503},[151,64823,42377],{"class":477},[151,64825,59118],{"class":503},[151,64827,64828,64831,64833],{"class":469,"line":534},[151,64829,64830],{"class":503},"    positions ",[151,64832,1876],{"class":1869},[151,64834,16606],{"class":503},[151,64836,64837,64839,64842,64844],{"class":469,"line":1413},[151,64838,16411],{"class":1869},[151,64840,64841],{"class":503}," row ",[151,64843,16417],{"class":1869},[151,64845,64846],{"class":503}," rows:\n",[151,64848,64849,64852,64854],{"class":469,"line":1418},[151,64850,64851],{"class":503},"        dic ",[151,64853,1876],{"class":1869},[151,64855,16634],{"class":503},[151,64857,64858,64861,64863,64866,64869],{"class":469,"line":2462},[151,64859,64860],{"class":503},"        position ",[151,64862,1876],{"class":1869},[151,64864,64865],{"class":503}," row.find_all(",[151,64867,64868],{"class":481},"'td'",[151,64870,3640],{"class":503},[151,64872,64873,64876,64879,64881,64883,64886,64888],{"class":469,"line":2471},[151,64874,64875],{"class":503},"        dic[",[151,64877,64878],{"class":481},"\"NAME_OF_ISSUER\"",[151,64880,16654],{"class":503},[151,64882,1876],{"class":1869},[151,64884,64885],{"class":503}," position[",[151,64887,9181],{"class":477},[151,64889,64890],{"class":503},"].text\n",[151,64892,64893,64895,64898,64900,64902,64904,64906],{"class":469,"line":2480},[151,64894,64875],{"class":503},[151,64896,64897],{"class":481},"\"TITLE_OF_CLASS\"",[151,64899,16654],{"class":503},[151,64901,1876],{"class":1869},[151,64903,64885],{"class":503},[151,64905,6760],{"class":477},[151,64907,64890],{"class":503},[151,64909,64910,64912,64915,64917,64919,64921,64923],{"class":469,"line":2489},[151,64911,64875],{"class":503},[151,64913,64914],{"class":481},"\"CUSIP\"",[151,64916,16654],{"class":503},[151,64918,1876],{"class":1869},[151,64920,64885],{"class":503},[151,64922,6619],{"class":477},[151,64924,64890],{"class":503},[151,64926,64927,64929,64932,64934,64936,64938,64941,64943,64946,64949,64951,64953,64955,64957],{"class":469,"line":2497},[151,64928,64875],{"class":503},[151,64930,64931],{"class":481},"\"VALUE\"",[151,64933,16654],{"class":503},[151,64935,1876],{"class":1869},[151,64937,16673],{"class":6205},[151,64939,64940],{"class":503},"(position[",[151,64942,6557],{"class":477},[151,64944,64945],{"class":503},"].text.replace(",[151,64947,64948],{"class":481},"','",[151,64950,106],{"class":503},[151,64952,2301],{"class":481},[151,64954,27742],{"class":503},[151,64956,23268],{"class":1869},[151,64958,64959],{"class":477},"1000\n",[151,64961,64962,64964,64967,64969,64971,64973,64975,64977,64979,64981,64983,64985],{"class":469,"line":3140},[151,64963,64875],{"class":503},[151,64965,64966],{"class":481},"\"SHARES\"",[151,64968,16654],{"class":503},[151,64970,1876],{"class":1869},[151,64972,16673],{"class":6205},[151,64974,64940],{"class":503},[151,64976,9187],{"class":477},[151,64978,64945],{"class":503},[151,64980,64948],{"class":481},[151,64982,106],{"class":503},[151,64984,2301],{"class":481},[151,64986,12451],{"class":503},[151,64988,64989,64991,64994,64996,64998,65001,65004],{"class":469,"line":3149},[151,64990,64875],{"class":503},[151,64992,64993],{"class":481},"\"DATE\"",[151,64995,16654],{"class":503},[151,64997,1876],{"class":1869},[151,64999,65000],{"class":503}," date.strip(",[151,65002,65003],{"class":481},"\".html\"",[151,65005,3640],{"class":503},[151,65007,65008],{"class":469,"line":3158},[151,65009,65010],{"class":503},"        positions.append(dic)\n",[151,65012,65013],{"class":469,"line":3167},[151,65014,1090],{"emptyLinePlaceholder":609},[151,65016,65017,65020,65022],{"class":469,"line":3175},[151,65018,65019],{"class":503},"    df ",[151,65021,1876],{"class":1869},[151,65023,65024],{"class":503}," pd.DataFrame(positions)\n",[151,65026,65027,65029],{"class":469,"line":3184},[151,65028,17496],{"class":1869},[151,65030,65031],{"class":503}," df\n",[11,65033,65034],{},"Using this function we can get a quick snapshot of this hedge fund by filing total over the last 4 years:",[459,65036,65038],{"className":13136,"code":65037,"language":12886,"meta":464,"style":464},"fund_growth = [sum(scrape_13f(file).VALUE) for file in sorted(files)]\ndates = [f.strip('.html') for f in sorted(files)]\nplt.figure(figsize=(10,5))\nplt.title('Total Fund Size')\nplt.xlabel('Filing Date')\nplt.ylabel('USD')\nplt.bar(dates, fund_growth)\nplt.yticks()\nplt.xticks(rotation='vertical')\n",[30,65039,65040,65075,65101,65119,65129,65139,65149,65154,65159],{"__ignoreMap":464},[151,65041,65042,65045,65047,65049,65052,65055,65057,65059,65062,65064,65066,65068,65070,65072],{"class":469,"line":470},[151,65043,65044],{"class":503},"fund_growth ",[151,65046,1876],{"class":1869},[151,65048,6604],{"class":503},[151,65050,65051],{"class":2226},"sum",[151,65053,65054],{"class":503},"(scrape_13f(",[151,65056,56116],{"class":12354},[151,65058,13576],{"class":503},[151,65060,65061],{"class":477},"VALUE",[151,65063,16995],{"class":503},[151,65065,16732],{"class":1869},[151,65067,4231],{"class":12354},[151,65069,2820],{"class":1869},[151,65071,44767],{"class":2226},[151,65073,65074],{"class":503},"(files)]\n",[151,65076,65077,65080,65082,65085,65088,65090,65092,65095,65097,65099],{"class":469,"line":488},[151,65078,65079],{"class":503},"dates ",[151,65081,1876],{"class":1869},[151,65083,65084],{"class":503}," [f.strip(",[151,65086,65087],{"class":481},"'.html'",[151,65089,16995],{"class":503},[151,65091,16732],{"class":1869},[151,65093,65094],{"class":503}," f ",[151,65096,16417],{"class":1869},[151,65098,44767],{"class":2226},[151,65100,65074],{"class":503},[151,65102,65103,65105,65107,65109,65111,65113,65115,65117],{"class":469,"line":500},[151,65104,44355],{"class":503},[151,65106,44358],{"class":15210},[151,65108,1876],{"class":1869},[151,65110,12386],{"class":503},[151,65112,12423],{"class":477},[151,65114,3634],{"class":503},[151,65116,24380],{"class":477},[151,65118,12451],{"class":503},[151,65120,65121,65124,65127],{"class":469,"line":509},[151,65122,65123],{"class":503},"plt.title(",[151,65125,65126],{"class":481},"'Total Fund Size'",[151,65128,3640],{"class":503},[151,65130,65131,65134,65137],{"class":469,"line":517},[151,65132,65133],{"class":503},"plt.xlabel(",[151,65135,65136],{"class":481},"'Filing Date'",[151,65138,3640],{"class":503},[151,65140,65141,65144,65147],{"class":469,"line":534},[151,65142,65143],{"class":503},"plt.ylabel(",[151,65145,65146],{"class":481},"'USD'",[151,65148,3640],{"class":503},[151,65150,65151],{"class":469,"line":1413},[151,65152,65153],{"class":503},"plt.bar(dates, fund_growth)\n",[151,65155,65156],{"class":469,"line":1418},[151,65157,65158],{"class":503},"plt.yticks()\n",[151,65160,65161,65164,65167,65169,65172],{"class":469,"line":2462},[151,65162,65163],{"class":503},"plt.xticks(",[151,65165,65166],{"class":15210},"rotation",[151,65168,1876],{"class":1869},[151,65170,65171],{"class":481},"'vertical'",[151,65173,3640],{"class":503},[11,65175,65176],{},[2718,65177],{"alt":20386,"src":65178},"/static/sec/fund_size.png",[56,65180,65182],{"id":65181},"fund-positions-with-bubble-chart","Fund Positions with Bubble Chart",[11,65184,65185],{},"Next, it would be great to get a snapshot of the stocks owned by this fund in a given year. Let's use a D3 bubble chart. The names for each stock are quite long, so first let's convert them to stock ticker values. Here's a quick script I hacked together using a Fidelity lookup service:",[459,65187,65189],{"className":13136,"code":65188,"language":12886,"meta":464,"style":464},"cusip_nums = set()\nfor file in files:\n    cusip_nums = cusip_nums | set(scrape_13f(file).CUSIP)\n\nticker_dic = {c:\"\" for c in cusip_nums}\nfor c in list(ticker_dic.keys()):\n    url = \"http://quotes.fidelity.com/mmnet/SymLookup.phtml?reqforlookup=REQUESTFORLOOKUP&productid=mmnet&isLoggedIn=mmnet&rows=50&for=stock&by=cusip&criteria=\"+c+\"&submit=Search\"\n    html = requests.get(url).text\n    soup = BeautifulSoup(html, 'lxml')\n    ticker_elem = soup.find('tr', attrs={\"bgcolor\":\"#666666\"})\n    ticker = \"\"\n    try:\n        ticker = ticker_elem.next_sibling.next_sibling.find('a').text\n        ticker_dic[c] = ticker\n    except:\n        pass\n\n    time.sleep(1)\n",[30,65190,65191,65202,65213,65238,65242,65263,65276,65296,65305,65317,65348,65357,65363,65379,65389,65395,65400,65404],{"__ignoreMap":464},[151,65192,65193,65196,65198,65200],{"class":469,"line":470},[151,65194,65195],{"class":503},"cusip_nums ",[151,65197,1876],{"class":1869},[151,65199,2309],{"class":6205},[151,65201,12461],{"class":503},[151,65203,65204,65206,65208,65210],{"class":469,"line":488},[151,65205,16732],{"class":1869},[151,65207,4231],{"class":12354},[151,65209,2820],{"class":1869},[151,65211,65212],{"class":503}," files:\n",[151,65214,65215,65218,65220,65223,65225,65227,65229,65231,65233,65236],{"class":469,"line":500},[151,65216,65217],{"class":503},"    cusip_nums ",[151,65219,1876],{"class":1869},[151,65221,65222],{"class":503}," cusip_nums ",[151,65224,3947],{"class":1869},[151,65226,2309],{"class":6205},[151,65228,65054],{"class":503},[151,65230,56116],{"class":12354},[151,65232,13576],{"class":503},[151,65234,65235],{"class":477},"CUSIP",[151,65237,3640],{"class":503},[151,65239,65240],{"class":469,"line":509},[151,65241,1090],{"emptyLinePlaceholder":609},[151,65243,65244,65247,65249,65252,65254,65256,65258,65260],{"class":469,"line":517},[151,65245,65246],{"class":503},"ticker_dic ",[151,65248,1876],{"class":1869},[151,65250,65251],{"class":503}," {c:",[151,65253,38471],{"class":481},[151,65255,2235],{"class":1869},[151,65257,63611],{"class":503},[151,65259,16417],{"class":1869},[151,65261,65262],{"class":503}," cusip_nums}\n",[151,65264,65265,65267,65269,65271,65273],{"class":469,"line":534},[151,65266,16732],{"class":1869},[151,65268,63611],{"class":503},[151,65270,16417],{"class":1869},[151,65272,59145],{"class":6205},[151,65274,65275],{"class":503},"(ticker_dic.keys()):\n",[151,65277,65278,65281,65283,65286,65288,65291,65293],{"class":469,"line":1413},[151,65279,65280],{"class":503},"    url ",[151,65282,1876],{"class":1869},[151,65284,65285],{"class":481}," \"http://quotes.fidelity.com/mmnet/SymLookup.phtml?reqforlookup=REQUESTFORLOOKUP&productid=mmnet&isLoggedIn=mmnet&rows=50&for=stock&by=cusip&criteria=\"",[151,65287,22885],{"class":1869},[151,65289,65290],{"class":503},"c",[151,65292,22885],{"class":1869},[151,65294,65295],{"class":481},"\"&submit=Search\"\n",[151,65297,65298,65300,65302],{"class":469,"line":1418},[151,65299,64776],{"class":503},[151,65301,1876],{"class":1869},[151,65303,65304],{"class":503}," requests.get(url).text\n",[151,65306,65307,65309,65311,65313,65315],{"class":469,"line":2462},[151,65308,64796],{"class":503},[151,65310,1876],{"class":1869},[151,65312,64801],{"class":503},[151,65314,64804],{"class":481},[151,65316,3640],{"class":503},[151,65318,65319,65322,65324,65327,65329,65331,65334,65336,65338,65341,65343,65346],{"class":469,"line":2471},[151,65320,65321],{"class":503},"    ticker_elem ",[151,65323,1876],{"class":1869},[151,65325,65326],{"class":503}," soup.find(",[151,65328,64819],{"class":481},[151,65330,106],{"class":503},[151,65332,65333],{"class":15210},"attrs",[151,65335,1876],{"class":1869},[151,65337,5729],{"class":503},[151,65339,65340],{"class":481},"\"bgcolor\"",[151,65342,208],{"class":503},[151,65344,65345],{"class":481},"\"#666666\"",[151,65347,19610],{"class":503},[151,65349,65350,65353,65355],{"class":469,"line":2480},[151,65351,65352],{"class":503},"    ticker ",[151,65354,1876],{"class":1869},[151,65356,38981],{"class":481},[151,65358,65359,65361],{"class":469,"line":2489},[151,65360,18280],{"class":1869},[151,65362,14372],{"class":503},[151,65364,65365,65368,65370,65373,65376],{"class":469,"line":2497},[151,65366,65367],{"class":503},"        ticker ",[151,65369,1876],{"class":1869},[151,65371,65372],{"class":503}," ticker_elem.next_sibling.next_sibling.find(",[151,65374,65375],{"class":481},"'a'",[151,65377,65378],{"class":503},").text\n",[151,65380,65381,65384,65386],{"class":469,"line":3140},[151,65382,65383],{"class":503},"        ticker_dic[c] ",[151,65385,1876],{"class":1869},[151,65387,65388],{"class":503}," ticker\n",[151,65390,65391,65393],{"class":469,"line":3149},[151,65392,18341],{"class":1869},[151,65394,14372],{"class":503},[151,65396,65397],{"class":469,"line":3158},[151,65398,65399],{"class":1869},"        pass\n",[151,65401,65402],{"class":469,"line":3167},[151,65403,1090],{"emptyLinePlaceholder":609},[151,65405,65406,65409,65411],{"class":469,"line":3175},[151,65407,65408],{"class":503},"    time.sleep(",[151,65410,6760],{"class":477},[151,65412,3640],{"class":503},[11,65414,65415,65416,65419,65420,65423],{},"I couldn't get all the CUSIP numbers, but I was able to get most of them. Some of the CUSIP numbers have changed for certain stocks and couldn't be looked up with this service. For now I won't fill these in. With the ",[30,65417,65418],{},"ticker_dic"," dictionary, we can make a quick edit to our ",[30,65421,65422],{},"scrape_13f"," function to populate ticker data for each holding:",[459,65425,65427],{"className":13136,"code":65426,"language":12886,"meta":464,"style":464},"ticker_dict = {'00206R102': 'T', '00507V109': 'ATVI', '00724F101': 'ADBE', ... }\n\ndef scrape_13f(file):\n    date = file\n    html = open(\"13f/\"+file).read()\n    soup = BeautifulSoup(html, 'lxml')\n    rows = soup.find_all('tr')[11:]\n    positions = []\n    for row in rows:\n        dic = {}\n        position = row.find_all('td')\n        dic[\"NAME_OF_ISSUER\"] = position[0].text\n        dic[\"TITLE_OF_CLASS\"] = position[1].text\n        dic[\"CUSIP\"] = position[2].text\n        dic[\"VALUE\"] = int(position[3].text.replace(',', ''))*1000\n        dic[\"SHARES\"] = int(position[4].text.replace(',', ''))\n        dic[\"DATE\"] = date.strip(\".html\")\n        dic[\"TICKER\"] = ticker_dict[position[2].text]\n        positions.append(dic)\n\n    df = pd.DataFrame(positions)\n    return df\n",[30,65428,65429,65472,65476,65488,65496,65514,65526,65542,65550,65560,65568,65580,65596,65612,65628,65658,65684,65700,65719,65723,65727,65735],{"__ignoreMap":464},[151,65430,65431,65434,65436,65438,65441,65443,65446,65448,65451,65453,65456,65458,65461,65463,65466,65468,65470],{"class":469,"line":470},[151,65432,65433],{"class":503},"ticker_dict ",[151,65435,1876],{"class":1869},[151,65437,52023],{"class":503},[151,65439,65440],{"class":481},"'00206R102'",[151,65442,6208],{"class":503},[151,65444,65445],{"class":481},"'T'",[151,65447,106],{"class":503},[151,65449,65450],{"class":481},"'00507V109'",[151,65452,6208],{"class":503},[151,65454,65455],{"class":481},"'ATVI'",[151,65457,106],{"class":503},[151,65459,65460],{"class":481},"'00724F101'",[151,65462,6208],{"class":503},[151,65464,65465],{"class":481},"'ADBE'",[151,65467,106],{"class":503},[151,65469,27455],{"class":477},[151,65471,19600],{"class":503},[151,65473,65474],{"class":469,"line":488},[151,65475,1090],{"emptyLinePlaceholder":609},[151,65477,65478,65480,65482,65484,65486],{"class":469,"line":500},[151,65479,16925],{"class":12347},[151,65481,64755],{"class":473},[151,65483,12386],{"class":503},[151,65485,56116],{"class":15232},[151,65487,15264],{"class":503},[151,65489,65490,65492,65494],{"class":469,"line":509},[151,65491,64766],{"class":503},[151,65493,1876],{"class":1869},[151,65495,64771],{"class":12354},[151,65497,65498,65500,65502,65504,65506,65508,65510,65512],{"class":469,"line":517},[151,65499,64776],{"class":503},[151,65501,1876],{"class":1869},[151,65503,16970],{"class":2226},[151,65505,12386],{"class":503},[151,65507,64706],{"class":481},[151,65509,22885],{"class":1869},[151,65511,56116],{"class":12354},[151,65513,64791],{"class":503},[151,65515,65516,65518,65520,65522,65524],{"class":469,"line":534},[151,65517,64796],{"class":503},[151,65519,1876],{"class":1869},[151,65521,64801],{"class":503},[151,65523,64804],{"class":481},[151,65525,3640],{"class":503},[151,65527,65528,65530,65532,65534,65536,65538,65540],{"class":469,"line":1413},[151,65529,64811],{"class":503},[151,65531,1876],{"class":1869},[151,65533,64816],{"class":503},[151,65535,64819],{"class":481},[151,65537,40832],{"class":503},[151,65539,42377],{"class":477},[151,65541,59118],{"class":503},[151,65543,65544,65546,65548],{"class":469,"line":1418},[151,65545,64830],{"class":503},[151,65547,1876],{"class":1869},[151,65549,16606],{"class":503},[151,65551,65552,65554,65556,65558],{"class":469,"line":2462},[151,65553,16411],{"class":1869},[151,65555,64841],{"class":503},[151,65557,16417],{"class":1869},[151,65559,64846],{"class":503},[151,65561,65562,65564,65566],{"class":469,"line":2471},[151,65563,64851],{"class":503},[151,65565,1876],{"class":1869},[151,65567,16634],{"class":503},[151,65569,65570,65572,65574,65576,65578],{"class":469,"line":2480},[151,65571,64860],{"class":503},[151,65573,1876],{"class":1869},[151,65575,64865],{"class":503},[151,65577,64868],{"class":481},[151,65579,3640],{"class":503},[151,65581,65582,65584,65586,65588,65590,65592,65594],{"class":469,"line":2489},[151,65583,64875],{"class":503},[151,65585,64878],{"class":481},[151,65587,16654],{"class":503},[151,65589,1876],{"class":1869},[151,65591,64885],{"class":503},[151,65593,9181],{"class":477},[151,65595,64890],{"class":503},[151,65597,65598,65600,65602,65604,65606,65608,65610],{"class":469,"line":2497},[151,65599,64875],{"class":503},[151,65601,64897],{"class":481},[151,65603,16654],{"class":503},[151,65605,1876],{"class":1869},[151,65607,64885],{"class":503},[151,65609,6760],{"class":477},[151,65611,64890],{"class":503},[151,65613,65614,65616,65618,65620,65622,65624,65626],{"class":469,"line":3140},[151,65615,64875],{"class":503},[151,65617,64914],{"class":481},[151,65619,16654],{"class":503},[151,65621,1876],{"class":1869},[151,65623,64885],{"class":503},[151,65625,6619],{"class":477},[151,65627,64890],{"class":503},[151,65629,65630,65632,65634,65636,65638,65640,65642,65644,65646,65648,65650,65652,65654,65656],{"class":469,"line":3149},[151,65631,64875],{"class":503},[151,65633,64931],{"class":481},[151,65635,16654],{"class":503},[151,65637,1876],{"class":1869},[151,65639,16673],{"class":6205},[151,65641,64940],{"class":503},[151,65643,6557],{"class":477},[151,65645,64945],{"class":503},[151,65647,64948],{"class":481},[151,65649,106],{"class":503},[151,65651,2301],{"class":481},[151,65653,27742],{"class":503},[151,65655,23268],{"class":1869},[151,65657,64959],{"class":477},[151,65659,65660,65662,65664,65666,65668,65670,65672,65674,65676,65678,65680,65682],{"class":469,"line":3158},[151,65661,64875],{"class":503},[151,65663,64966],{"class":481},[151,65665,16654],{"class":503},[151,65667,1876],{"class":1869},[151,65669,16673],{"class":6205},[151,65671,64940],{"class":503},[151,65673,9187],{"class":477},[151,65675,64945],{"class":503},[151,65677,64948],{"class":481},[151,65679,106],{"class":503},[151,65681,2301],{"class":481},[151,65683,12451],{"class":503},[151,65685,65686,65688,65690,65692,65694,65696,65698],{"class":469,"line":3167},[151,65687,64875],{"class":503},[151,65689,64993],{"class":481},[151,65691,16654],{"class":503},[151,65693,1876],{"class":1869},[151,65695,65000],{"class":503},[151,65697,65003],{"class":481},[151,65699,3640],{"class":503},[151,65701,65702,65704,65707,65709,65711,65714,65716],{"class":469,"line":3175},[151,65703,64875],{"class":503},[151,65705,65706],{"class":481},"\"TICKER\"",[151,65708,16654],{"class":503},[151,65710,1876],{"class":1869},[151,65712,65713],{"class":503}," ticker_dict[position[",[151,65715,6619],{"class":477},[151,65717,65718],{"class":503},"].text]\n",[151,65720,65721],{"class":469,"line":3184},[151,65722,65010],{"class":503},[151,65724,65725],{"class":469,"line":3193},[151,65726,1090],{"emptyLinePlaceholder":609},[151,65728,65729,65731,65733],{"class":469,"line":3720},[151,65730,65019],{"class":503},[151,65732,1876],{"class":1869},[151,65734,65024],{"class":503},[151,65736,65737,65739],{"class":469,"line":3729},[151,65738,17496],{"class":1869},[151,65740,65031],{"class":503},[11,65742,65743],{},"Let's check this:",[459,65745,65748],{"className":65746,"code":65747,"language":997},[995],"df = scrape_13f(files[2])\nprint(df[[\"CUSIP\", \"NAME_OF_ISSUER\", \"TICKER\"]].head())\n",[30,65749,65747],{"__ignoreMap":464},[459,65751,65754],{"className":65752,"code":65753,"language":997},[995],"       CUSIP         NAME_OF_ISSUER TICKER\n0  88579Y101                  3M CO    MMM\n1  G1151C101  ACCENTURE PLC IRELAND    ACN\n2  02209S103       ALTRIA GROUP INC     MO\n3  03076C106    AMERIPRISE FINL INC    AMP\n4  035710409    ANNALY CAP MGMT INC    NLY\n",[30,65755,65753],{"__ignoreMap":464},[11,65757,65758],{},"Let's take a look at the last filing, Q4 2017.",[459,65760,65762],{"className":13136,"code":65761,"language":12886,"meta":464,"style":464},"q4_2017 = sorted(files)[-1]\ndf_q4_2017 = scrape_13f(q4_2017)\n\ntop_20 = df_q4_2017.sort_values(by=\"VALUE\", ascending=False)[[\"TICKER\", \"VALUE\"]][:40]\na = top_20.TICKER\nb = top_20.VALUE\nc = range(len(b))\n\nfig = plt.figure(figsize=(15,5))\nax = fig.add_subplot(111)\nax.bar(c, b)\n\nplt.xticks(c, a, rotation=90)\nplt.title('Top 40 Stock Holdings by Value')\nplt.xlabel('Stock Ticker')\nplt.ylabel('USD (10 MM))')\nplt.show()\n",[30,65763,65764,65782,65792,65796,65838,65850,65862,65879,65883,65907,65921,65926,65930,65944,65953,65962,65971],{"__ignoreMap":464},[151,65765,65766,65769,65771,65773,65776,65778,65780],{"class":469,"line":470},[151,65767,65768],{"class":503},"q4_2017 ",[151,65770,1876],{"class":1869},[151,65772,44767],{"class":2226},[151,65774,65775],{"class":503},"(files)[",[151,65777,12445],{"class":1869},[151,65779,6760],{"class":477},[151,65781,3691],{"class":503},[151,65783,65784,65787,65789],{"class":469,"line":488},[151,65785,65786],{"class":503},"df_q4_2017 ",[151,65788,1876],{"class":1869},[151,65790,65791],{"class":503}," scrape_13f(q4_2017)\n",[151,65793,65794],{"class":469,"line":500},[151,65795,1090],{"emptyLinePlaceholder":609},[151,65797,65798,65801,65803,65806,65809,65811,65813,65815,65818,65820,65822,65825,65827,65829,65831,65834,65836],{"class":469,"line":509},[151,65799,65800],{"class":503},"top_20 ",[151,65802,1876],{"class":1869},[151,65804,65805],{"class":503}," df_q4_2017.sort_values(",[151,65807,65808],{"class":15210},"by",[151,65810,1876],{"class":1869},[151,65812,64931],{"class":481},[151,65814,106],{"class":503},[151,65816,65817],{"class":15210},"ascending",[151,65819,1876],{"class":1869},[151,65821,39461],{"class":477},[151,65823,65824],{"class":503},")[[",[151,65826,65706],{"class":481},[151,65828,106],{"class":503},[151,65830,64931],{"class":481},[151,65832,65833],{"class":503},"]][:",[151,65835,44365],{"class":477},[151,65837,3691],{"class":503},[151,65839,65840,65842,65844,65847],{"class":469,"line":517},[151,65841,61268],{"class":503},[151,65843,1876],{"class":1869},[151,65845,65846],{"class":503}," top_20.",[151,65848,65849],{"class":477},"TICKER\n",[151,65851,65852,65855,65857,65859],{"class":469,"line":534},[151,65853,65854],{"class":503},"b ",[151,65856,1876],{"class":1869},[151,65858,65846],{"class":503},[151,65860,65861],{"class":477},"VALUE\n",[151,65863,65864,65867,65869,65871,65873,65876],{"class":469,"line":1413},[151,65865,65866],{"class":503},"c ",[151,65868,1876],{"class":1869},[151,65870,2793],{"class":2226},[151,65872,12386],{"class":503},[151,65874,65875],{"class":2226},"len",[151,65877,65878],{"class":503},"(b))\n",[151,65880,65881],{"class":469,"line":1418},[151,65882,1090],{"emptyLinePlaceholder":609},[151,65884,65885,65888,65890,65893,65895,65897,65899,65901,65903,65905],{"class":469,"line":2462},[151,65886,65887],{"class":503},"fig ",[151,65889,1876],{"class":1869},[151,65891,65892],{"class":503}," plt.figure(",[151,65894,44358],{"class":15210},[151,65896,1876],{"class":1869},[151,65898,12386],{"class":503},[151,65900,42310],{"class":477},[151,65902,3634],{"class":503},[151,65904,24380],{"class":477},[151,65906,12451],{"class":503},[151,65908,65909,65912,65914,65917,65919],{"class":469,"line":2471},[151,65910,65911],{"class":503},"ax ",[151,65913,1876],{"class":1869},[151,65915,65916],{"class":503}," fig.add_subplot(",[151,65918,45392],{"class":477},[151,65920,3640],{"class":503},[151,65922,65923],{"class":469,"line":2480},[151,65924,65925],{"class":503},"ax.bar(c, b)\n",[151,65927,65928],{"class":469,"line":2489},[151,65929,1090],{"emptyLinePlaceholder":609},[151,65931,65932,65935,65937,65939,65942],{"class":469,"line":2497},[151,65933,65934],{"class":503},"plt.xticks(c, a, ",[151,65936,65166],{"class":15210},[151,65938,1876],{"class":1869},[151,65940,65941],{"class":477},"90",[151,65943,3640],{"class":503},[151,65945,65946,65948,65951],{"class":469,"line":3140},[151,65947,65123],{"class":503},[151,65949,65950],{"class":481},"'Top 40 Stock Holdings by Value'",[151,65952,3640],{"class":503},[151,65954,65955,65957,65960],{"class":469,"line":3149},[151,65956,65133],{"class":503},[151,65958,65959],{"class":481},"'Stock Ticker'",[151,65961,3640],{"class":503},[151,65963,65964,65966,65969],{"class":469,"line":3158},[151,65965,65143],{"class":503},[151,65967,65968],{"class":481},"'USD (10 MM))'",[151,65970,3640],{"class":503},[151,65972,65973],{"class":469,"line":3167},[151,65974,44415],{"class":503},[11,65976,65977],{},[2718,65978],{"alt":20386,"src":65979},"/static/sec/2017_filing.png",[589,65981,65982],{},"html pre.shiki code .sMOD_, html code.shiki .sMOD_{--shiki-default:#24292E;--shiki-dark:#E1E4E8;--shiki-sepia:#F8F8F2}html pre.shiki code .sC2Qs, html code.shiki .sC2Qs{--shiki-default:#D73A49;--shiki-dark:#F97583;--shiki-sepia:#F92672}html pre.shiki code .sstjo, html code.shiki .sstjo{--shiki-default:#032F62;--shiki-dark:#9ECBFF;--shiki-sepia:#E6DB74}html pre.shiki code .sTrkL, html code.shiki .sTrkL{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#66D9EF}html pre.shiki code .sTHNf, html code.shiki .sTHNf{--shiki-default:#E36209;--shiki-default-font-style:inherit;--shiki-dark:#FFAB70;--shiki-dark-font-style:inherit;--shiki-sepia:#FD971F;--shiki-sepia-font-style:italic}html pre.shiki code .s7F3e, html code.shiki .s7F3e{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#AE81FF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html .sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html.sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html pre.shiki code .sq6CD, html code.shiki .sq6CD{--shiki-default:#D73A49;--shiki-default-font-style:inherit;--shiki-dark:#F97583;--shiki-dark-font-style:inherit;--shiki-sepia:#66D9EF;--shiki-sepia-font-style:italic}html pre.shiki code .srTi1, html code.shiki .srTi1{--shiki-default:#6F42C1;--shiki-dark:#B392F0;--shiki-sepia:#A6E22E}html pre.shiki code .so59x, html code.shiki .so59x{--shiki-default:#24292E;--shiki-default-font-style:inherit;--shiki-dark:#E1E4E8;--shiki-dark-font-style:inherit;--shiki-sepia:#FD971F;--shiki-sepia-font-style:italic}html pre.shiki code .sOrwc, html code.shiki .sOrwc{--shiki-default:#E36209;--shiki-dark:#FFAB70;--shiki-sepia:#F8F8F2}html pre.shiki code .s-m8C, html code.shiki .s-m8C{--shiki-default:#005CC5;--shiki-default-font-style:inherit;--shiki-dark:#79B8FF;--shiki-dark-font-style:inherit;--shiki-sepia:#66D9EF;--shiki-sepia-font-style:italic}",{"title":464,"searchDepth":488,"depth":488,"links":65984},[65985],{"id":65181,"depth":488,"text":65182},"2018-01-30","How to read SEC filing data with Python","/static/sec/sec.jpg",{"layout":48045},"/2018/01/30/reading-13f-sec-filings-with-python",{"title":64610,"description":65987},"2018/01/30/reading-13f-sec-filings-with-python",[65994,12886,12355,46089],"sec","VdxsAoHhWBei2IJRIzWg7z1CA1MC-hir5JSJrsRh5nI",{"id":65997,"title":65998,"body":65999,"comments":609,"date":66571,"description":66003,"draft":602,"extension":605,"external":606,"image":66572,"meta":66573,"navigation":609,"path":66575,"seo":66576,"stem":66577,"tags":66578,"__hash__":66581},"blog/2018/01/28/simple-sql-tasks-with-mariadb.md","How to do simple SQL tasks with MariaDB",{"type":8,"value":66000,"toc":66562},[66001,66004,66007,66015,66019,66022,66025,66036,66041,66052,66056,66064,66068,66075,66081,66089,66092,66098,66104,66110,66125,66129,66136,66142,66145,66151,66157,66163,66166,66169,66171,66263,66269,66272,66274,66433,66439,66442,66444,66542,66548,66551,66556,66559],[11,66002,66003],{},"In this article I'll look at another interview takehome assignment. This assignment included SQL, python and some logic puzzles. If you read to the end I'll share a fun \"trick question\" that was also included.",[11,66005,66006],{},"I mostly want to talk about the SQL portion of the test. Since I mostly work with postgresql through the Django ORM, this was a good refresher. I'll show how I approached some simple SQL tasks, and how to use MariaDB.",[210,66008,66009],{},[11,66010,66011,66014],{},[15,66012,66013],{},"MariaDB"," is the default implementation of MySQL in Arch Linux, provided with the mariadb package.",[56,66016,66018],{"id":66017},"the-task","The Task",[11,66020,66021],{},"Here are the three questions I was asked:",[11,66023,66024],{},"Please reference the 3 tables below.",[700,66026,66027,66030,66033],{},[79,66028,66029],{},"Using SQL please write the code to generate a table that includes all individuals on the file and contains the following fields: ID, Congressional District, and Gender",[79,66031,66032],{},"Using SQL please write the code to generate a table that includes only individuals with a gender on file that have a DistrictID of 3. Also please convert the values for Gender from “M” and “F” to “Male” and “Female” respectively. Your final table should include only ID and the converted Gender.",[79,66034,66035],{},"Using SQL please generate the code to run a count of gender by Congressional District.",[11,66037,66038],{},[2718,66039],{"alt":20386,"src":66040},"/static/sql.png",[76,66042,66043,66046,66049],{},[79,66044,66045],{},"ID is a unique ID that is applied to each individual on file",[79,66047,66048],{},"DistrictID is a unique ID that is applied to each district on file",[79,66050,66051],{},"Gender is a value that is recorded on some records on file",[736,66053,66055],{"id":66054},"install-mysqlmariadb","Install MySQL/MariaDB",[11,66057,66058,66059,643],{},"First, we need to install MariaDB. As usual, just ",[20,66060,66063],{"href":66061,"rel":66062},"https://wiki.archlinux.org/index.php/MySQL",[24],"follow the Arch Wiki",[736,66065,66067],{"id":66066},"run-mysql","Run mysql",[11,66069,66070,66071,66074],{},"Next, we can run ",[30,66072,66073],{},"mysql"," in the terminal:",[459,66076,66079],{"className":66077,"code":66078,"language":997},[995]," $ mysql\nWelcome to the MariaDB monitor.  Commands end with ; or \\g.\nYour MariaDB connection id is 18\nServer version: 10.1.30-MariaDB MariaDB Server\n\nCopyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others.\n\nType 'help;' or '\\h' for help. Type '\\c' to clear the current input statement.\n",[30,66080,66078],{"__ignoreMap":464},[736,66082,66084,66085,66088],{"id":66083},"create-and-set-a-database-to-use","Create ",[51,66086,66087],{},"and set"," a database to use",[11,66090,66091],{},"The next step is to create a database. We can do the following:",[459,66093,66096],{"className":66094,"code":66095,"language":997},[995],"MariaDB [(none)]> create database sample_db default character set utf8 default collate utf8_bin;\nERROR 1044 (42000): Access denied for user ''@'localhost' to database 'sample_db'\nMariaDB [(none)]> Ctrl-C -- exit!\nAborted\n",[30,66097,66095],{"__ignoreMap":464},[11,66099,66100,66101,66103],{},"If you see this error, it is because we aren't using the correct user for SQL. We can use the default root user with no password instead. So here we will rerun the ",[30,66102,66073],{}," command with additional arguments:",[459,66105,66108],{"className":66106,"code":66107,"language":997},[995]," $ mysql -u root\nWelcome to the MariaDB monitor.  Commands end with ; or \\g.\nYour MariaDB connection id is 19\nServer version: 10.1.30-MariaDB MariaDB Server\n\nCopyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others.\n\nType 'help;' or '\\h' for help. Type '\\c' to clear the current input statement.\n\nMariaDB [(none)]> create database sample_db default character set utf8 default collate utf8_bin;\nQuery OK, 1 row affected (0.01 sec)\n\nMariaDB [(none)]> use sample_db\nDatabase changed\nMariaDB [sample_db]>\n",[30,66109,66107],{"__ignoreMap":464},[11,66111,66112,66113,66116,66117,66120,66121,66124],{},"Great, no we are using the blank database called ",[30,66114,66115],{},"sample_db"," and we are using it, notice that ",[30,66118,66119],{},"[sample_db]>"," has repaced ",[30,66122,66123],{},"[(none)]>"," in the MariaDB prompt.",[736,66126,66128],{"id":66127},"create-tables-and-insert-data-into-the-tables","Create tables and insert data into the tables",[11,66130,66131,66132,66135],{},"Now we are read to get going with our questions. Before we write our queries, we need to get the data from the question into our database. To do this I wrote ",[30,66133,66134],{},"insert.sql"," and ran it in MariaDB. Here is the script:",[459,66137,66140],{"className":66138,"code":66139,"language":997},[995],"CREATE TABLE table_A (\n  ID int,\n  District_ID int\n  );\n\nCREATE TABLE table_B (\n  District_ID int,\n  Congressional_District int\n  );\n\nCREATE TABLE table_C (\n  ID int,\n  Gender CHAR(1)\n  );\n\nINSERT INTO table_A (ID, District_ID) VALUES ('1', '3');\nINSERT INTO table_A (ID, District_ID) VALUES ('2', '3');\nINSERT INTO table_A (ID, District_ID) VALUES ('3', '4');\nINSERT INTO table_A (ID, District_ID) VALUES ('4', '4');\nINSERT INTO table_A (ID, District_ID) VALUES ('5', '3');\nINSERT INTO table_A (ID, District_ID) VALUES ('6', '4');\nINSERT INTO table_A (ID, District_ID) VALUES ('7', '4');\nINSERT INTO table_A (ID, District_ID) VALUES ('8', '3');\nINSERT INTO table_A (ID, District_ID) VALUES ('9', '4');\nINSERT INTO table_A (ID, District_ID) VALUES ('10', '3');\n\nINSERT INTO table_B (District_ID, Congressional_District) VALUES ('1', '8');\nINSERT INTO table_B (District_ID, Congressional_District) VALUES ('2', '2');\nINSERT INTO table_B (District_ID, Congressional_District) VALUES ('3', '14');\nINSERT INTO table_B (District_ID, Congressional_District) VALUES ('4', '7');\nINSERT INTO table_B (District_ID, Congressional_District) VALUES ('5', '11');\n\nINSERT INTO table_C (ID, Gender) VALUES ('1', 'M');\nINSERT INTO table_C (ID, Gender) VALUES ('3', 'F');\nINSERT INTO table_C (ID, Gender) VALUES ('4', 'M');\nINSERT INTO table_C (ID, Gender) VALUES ('5', 'F');\nINSERT INTO table_C (ID, Gender) VALUES ('6', 'F');\nINSERT INTO table_C (ID, Gender) VALUES ('7', 'F');\nINSERT INTO table_C (ID, Gender) VALUES ('8', 'M');\nINSERT INTO table_C (ID, Gender) VALUES ('9', 'F');\nINSERT INTO table_C (ID, Gender) VALUES ('10', 'M');\n",[30,66141,66139],{"__ignoreMap":464},[11,66143,66144],{},"Now we can run the script with the following command:",[459,66146,66149],{"className":66147,"code":66148,"language":997},[995],"MariaDB [sample_db]> source my_sql_script.sql\nQuery OK, 0 rows affected (0.08 sec)\n\nQuery OK, 0 rows affected (0.07 sec)\n\nQuery OK, 0 rows affected (0.06 sec)\n\nQuery OK, 1 row affected (0.02 sec)\n\nQuery OK, 1 row affected (0.00 sec)\n\nQuery OK, 1 row affected (0.01 sec)\n\nQuery OK, 1 row affected (0.03 sec)\n\nQuery OK, 1 row affected (0.01 sec)\n\nQuery OK, 1 row affected (0.00 sec)\n\nQuery OK, 1 row affected (0.01 sec)\n\nQuery OK, 1 row affected (0.01 sec)\n\nQuery OK, 1 row affected (0.01 sec)\n\nQuery OK, 1 row affected (0.01 sec)\n\nQuery OK, 1 row affected (0.01 sec)\n\nQuery OK, 1 row affected (0.01 sec)\n\nQuery OK, 1 row affected (0.01 sec)\n\nQuery OK, 1 row affected (0.01 sec)\n\nQuery OK, 1 row affected (0.01 sec)\n\nQuery OK, 1 row affected (0.01 sec)\n\nQuery OK, 1 row affected (0.01 sec)\n\nQuery OK, 1 row affected (0.02 sec)\n\nQuery OK, 1 row affected (0.01 sec)\n\nQuery OK, 1 row affected (0.01 sec)\n\nQuery OK, 1 row affected (0.01 sec)\n\nQuery OK, 1 row affected (0.01 sec)\n\nQuery OK, 1 row affected (0.01 sec)\n\nQuery OK, 1 row affected (0.01 sec)\n\nMariaDB [sample_db]>\n",[30,66150,66148],{"__ignoreMap":464},[11,66152,66153,66154,33226],{},"Great, now let's test it out with a simple ",[30,66155,66156],{},"select * from table_name",[459,66158,66161],{"className":66159,"code":66160,"language":997},[995],"MariaDB [sample_db]> select * from table_A;\n+------+-------------+\n| ID   | District_ID |\n+------+-------------+\n|    1 |           3 |\n|    2 |           3 |\n|    3 |           4 |\n|    4 |           4 |\n|    5 |           3 |\n|    6 |           4 |\n|    7 |           4 |\n|    8 |           3 |\n|    9 |           4 |\n|   10 |           3 |\n+------+-------------+\n10 rows in set (0.00 sec)\n",[30,66162,66160],{"__ignoreMap":464},[11,66164,66165],{},"This matches what we were given, so now let's move on to the first task, creating one table from the three we are given.",[54716,66167,6760],{"id":66168},"_1",[11,66170,66029],{},[459,66172,66176],{"className":66173,"code":66174,"language":66175,"meta":464,"style":464},"language-sql shiki shiki-themes github-light github-dark monokai","select table_A.ID, Congressional_District, Gender\nfrom table_A\n  left join table_B\n    on table_A.District_ID = table_B.District_ID\n  left join table_C\n    on table_A.ID = table_C.ID order by ID;\n","sql",[30,66177,66178,66194,66201,66209,66231,66238],{"__ignoreMap":464},[151,66179,66180,66183,66186,66188,66191],{"class":469,"line":470},[151,66181,66182],{"class":1869},"select",[151,66184,66185],{"class":477}," table_A",[151,66187,643],{"class":503},[151,66189,66190],{"class":477},"ID",[151,66192,66193],{"class":503},", Congressional_District, Gender\n",[151,66195,66196,66198],{"class":469,"line":488},[151,66197,16853],{"class":1869},[151,66199,66200],{"class":503}," table_A\n",[151,66202,66203,66206],{"class":469,"line":500},[151,66204,66205],{"class":1869},"  left join",[151,66207,66208],{"class":503}," table_B\n",[151,66210,66211,66214,66216,66218,66221,66223,66226,66228],{"class":469,"line":509},[151,66212,66213],{"class":1869},"    on",[151,66215,66185],{"class":477},[151,66217,643],{"class":503},[151,66219,66220],{"class":477},"District_ID",[151,66222,19865],{"class":1869},[151,66224,66225],{"class":477}," table_B",[151,66227,643],{"class":503},[151,66229,66230],{"class":477},"District_ID\n",[151,66232,66233,66235],{"class":469,"line":517},[151,66234,66205],{"class":1869},[151,66236,66237],{"class":503}," table_C\n",[151,66239,66240,66242,66244,66246,66248,66250,66253,66255,66257,66260],{"class":469,"line":534},[151,66241,66213],{"class":1869},[151,66243,66185],{"class":477},[151,66245,643],{"class":503},[151,66247,66190],{"class":477},[151,66249,19865],{"class":1869},[151,66251,66252],{"class":477}," table_C",[151,66254,643],{"class":503},[151,66256,66190],{"class":477},[151,66258,66259],{"class":1869}," order by",[151,66261,66262],{"class":503}," ID;\n",[459,66264,66267],{"className":66265,"code":66266,"language":997},[995],"+------+------------------------+--------+\n| ID   | Congressional_District | Gender |\n+------+------------------------+--------+\n|    1 |                     14 | M      |\n|    2 |                     14 | NULL   |\n|    3 |                      7 | F      |\n|    4 |                      7 | M      |\n|    5 |                     14 | F      |\n|    6 |                      7 | F      |\n|    7 |                      7 | F      |\n|    8 |                     14 | M      |\n|    9 |                      7 | F      |\n|   10 |                     14 | M      |\n+------+------------------------+--------+\n10 rows in set (0.00 sec)\n",[30,66268,66266],{"__ignoreMap":464},[54716,66270,6619],{"id":66271},"_2",[11,66273,66032],{},[459,66275,66277],{"className":66173,"code":66276,"language":66175,"meta":464,"style":464},"select\n  table_A.ID,\n  case\n    when Gender = \"M\" then \"Male\"\n    when Gender = \"F\" then \"Female\"\n  end as Gender\nfrom table_A\n  left join table_B\n    on table_A.District_ID = table_B.District_ID\n  left join table_C on table_A.ID = table_C.ID\nwhere table_A.District_ID = 3 and table_C.Gender is not null\norder by ID;\n",[30,66278,66279,66284,66295,66300,66319,66335,66345,66351,66357,66375,66399,66426],{"__ignoreMap":464},[151,66280,66281],{"class":469,"line":470},[151,66282,66283],{"class":1869},"select\n",[151,66285,66286,66289,66291,66293],{"class":469,"line":488},[151,66287,66288],{"class":477},"  table_A",[151,66290,643],{"class":503},[151,66292,66190],{"class":477},[151,66294,9417],{"class":503},[151,66296,66297],{"class":469,"line":500},[151,66298,66299],{"class":1869},"  case\n",[151,66301,66302,66305,66308,66310,66313,66316],{"class":469,"line":509},[151,66303,66304],{"class":1869},"    when",[151,66306,66307],{"class":503}," Gender ",[151,66309,1876],{"class":1869},[151,66311,66312],{"class":481}," \"M\"",[151,66314,66315],{"class":1869}," then",[151,66317,66318],{"class":481}," \"Male\"\n",[151,66320,66321,66323,66325,66327,66330,66332],{"class":469,"line":517},[151,66322,66304],{"class":1869},[151,66324,66307],{"class":503},[151,66326,1876],{"class":1869},[151,66328,66329],{"class":481}," \"F\"",[151,66331,66315],{"class":1869},[151,66333,66334],{"class":481}," \"Female\"\n",[151,66336,66337,66340,66342],{"class":469,"line":534},[151,66338,66339],{"class":1869},"  end",[151,66341,18347],{"class":1869},[151,66343,66344],{"class":503}," Gender\n",[151,66346,66347,66349],{"class":469,"line":1413},[151,66348,16853],{"class":1869},[151,66350,66200],{"class":503},[151,66352,66353,66355],{"class":469,"line":1418},[151,66354,66205],{"class":1869},[151,66356,66208],{"class":503},[151,66358,66359,66361,66363,66365,66367,66369,66371,66373],{"class":469,"line":2462},[151,66360,66213],{"class":1869},[151,66362,66185],{"class":477},[151,66364,643],{"class":503},[151,66366,66220],{"class":477},[151,66368,19865],{"class":1869},[151,66370,66225],{"class":477},[151,66372,643],{"class":503},[151,66374,66230],{"class":477},[151,66376,66377,66379,66382,66384,66386,66388,66390,66392,66394,66396],{"class":469,"line":2471},[151,66378,66205],{"class":1869},[151,66380,66381],{"class":503}," table_C ",[151,66383,20429],{"class":1869},[151,66385,66185],{"class":477},[151,66387,643],{"class":503},[151,66389,66190],{"class":477},[151,66391,19865],{"class":1869},[151,66393,66252],{"class":477},[151,66395,643],{"class":503},[151,66397,66398],{"class":477},"ID\n",[151,66400,66401,66404,66406,66408,66410,66412,66414,66416,66418,66420,66423],{"class":469,"line":2480},[151,66402,66403],{"class":1869},"where",[151,66405,66185],{"class":477},[151,66407,643],{"class":503},[151,66409,66220],{"class":477},[151,66411,19865],{"class":1869},[151,66413,3650],{"class":477},[151,66415,2181],{"class":1869},[151,66417,66252],{"class":477},[151,66419,643],{"class":503},[151,66421,66422],{"class":477},"Gender",[151,66424,66425],{"class":1869}," is not null\n",[151,66427,66428,66431],{"class":469,"line":2489},[151,66429,66430],{"class":1869},"order by",[151,66432,66262],{"class":503},[459,66434,66437],{"className":66435,"code":66436,"language":997},[995],"+------+--------+\n| ID   | Gender |\n+------+--------+\n|    1 | Male   |\n|    5 | Female |\n|    8 | Male   |\n|   10 | Male   |\n+------+--------+\n4 rows in set (0.00 sec)\n",[30,66438,66436],{"__ignoreMap":464},[54716,66440,6557],{"id":66441},"_3",[11,66443,66035],{},[459,66445,66447],{"className":66173,"code":66446,"language":66175,"meta":464,"style":464},"select\n  count(table_C.Gender) Count,\n  Congressional_District,\n  Gender\nfrom table_A\n  left join table_B\n    on table_A.District_ID = table_B.District_ID\n  left join table_C\n    on table_A.ID = table_C.ID\ngroup by Congressional_District, Gender;\n",[30,66448,66449,66453,66470,66475,66480,66486,66492,66510,66516,66534],{"__ignoreMap":464},[151,66450,66451],{"class":469,"line":470},[151,66452,66283],{"class":1869},[151,66454,66455,66458,66460,66463,66465,66467],{"class":469,"line":488},[151,66456,66457],{"class":2226},"  count",[151,66459,12386],{"class":503},[151,66461,66462],{"class":477},"table_C",[151,66464,643],{"class":503},[151,66466,66422],{"class":477},[151,66468,66469],{"class":503},") Count,\n",[151,66471,66472],{"class":469,"line":500},[151,66473,66474],{"class":503},"  Congressional_District,\n",[151,66476,66477],{"class":469,"line":509},[151,66478,66479],{"class":503},"  Gender\n",[151,66481,66482,66484],{"class":469,"line":517},[151,66483,16853],{"class":1869},[151,66485,66200],{"class":503},[151,66487,66488,66490],{"class":469,"line":534},[151,66489,66205],{"class":1869},[151,66491,66208],{"class":503},[151,66493,66494,66496,66498,66500,66502,66504,66506,66508],{"class":469,"line":1413},[151,66495,66213],{"class":1869},[151,66497,66185],{"class":477},[151,66499,643],{"class":503},[151,66501,66220],{"class":477},[151,66503,19865],{"class":1869},[151,66505,66225],{"class":477},[151,66507,643],{"class":503},[151,66509,66230],{"class":477},[151,66511,66512,66514],{"class":469,"line":1418},[151,66513,66205],{"class":1869},[151,66515,66237],{"class":503},[151,66517,66518,66520,66522,66524,66526,66528,66530,66532],{"class":469,"line":2462},[151,66519,66213],{"class":1869},[151,66521,66185],{"class":477},[151,66523,643],{"class":503},[151,66525,66190],{"class":477},[151,66527,19865],{"class":1869},[151,66529,66252],{"class":477},[151,66531,643],{"class":503},[151,66533,66398],{"class":477},[151,66535,66536,66539],{"class":469,"line":2471},[151,66537,66538],{"class":1869},"group by",[151,66540,66541],{"class":503}," Congressional_District, Gender;\n",[459,66543,66546],{"className":66544,"code":66545,"language":997},[995],"+-------+------------------------+--------+\n| Count | Congressional_District | Gender |\n+-------+------------------------+--------+\n|     4 |                      7 | F      |\n|     1 |                      7 | M      |\n|     0 |                     14 | NULL   |\n|     1 |                     14 | F      |\n|     3 |                     14 | M      |\n+-------+------------------------+--------+\n5 rows in set (0.00 sec)\n",[30,66547,66545],{"__ignoreMap":464},[11,66549,66550],{},"That's it for the three SQL tasks. Here's the bonus question:",[210,66552,66553],{},[11,66554,66555],{},"Assume there are 6,000,000,000 (6 billion) people on Earth. What would you estimate to be the result, if you multiply together the number of fingers on every person's left-hands? (For the purposes of this exercise, thumbs count as fingers.)",[11,66557,66558],{},"Think about it for a minute. I have hidden my answer at the end of the URL for this article.",[589,66560,66561],{},"html pre.shiki code .sC2Qs, html code.shiki .sC2Qs{--shiki-default:#D73A49;--shiki-dark:#F97583;--shiki-sepia:#F92672}html pre.shiki code .s7F3e, html code.shiki .s7F3e{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#AE81FF}html pre.shiki code .sMOD_, html code.shiki .sMOD_{--shiki-default:#24292E;--shiki-dark:#E1E4E8;--shiki-sepia:#F8F8F2}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html .sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html.sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html pre.shiki code .sstjo, html code.shiki .sstjo{--shiki-default:#032F62;--shiki-dark:#9ECBFF;--shiki-sepia:#E6DB74}html pre.shiki code .sTrkL, html code.shiki .sTrkL{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#66D9EF}",{"title":464,"searchDepth":488,"depth":488,"links":66563},[66564],{"id":66017,"depth":488,"text":66018,"children":66565},[66566,66567,66568,66570],{"id":66054,"depth":500,"text":66055},{"id":66066,"depth":500,"text":66067},{"id":66083,"depth":500,"text":66569},"Create and set a database to use",{"id":66127,"depth":500,"text":66128},"2018-01-28","/static/mariadb.png",{"layout":48045,"permalink":66574},"/2018/01/28/how-to-do-simple-sql-tasks-with-mariadb-0","/2018/01/28/simple-sql-tasks-with-mariadb",{"title":65998,"description":66003},"2018/01/28/simple-sql-tasks-with-mariadb",[66579,66175,66580],"maria-db","databases","5Djrn-JBuWs6_wh7SRAS-EBpA3NkzX9X4t0YTz9q_n4",{"id":66583,"title":66584,"body":66585,"comments":609,"date":68502,"description":464,"draft":602,"extension":605,"external":606,"image":66591,"meta":68503,"navigation":609,"path":68504,"seo":68505,"stem":68506,"tags":68507,"__hash__":68509},"blog/2018/01/02/checking-poker-hands-with-python.md","Finding the best poker hand in five-card draw with python",{"type":8,"value":66586,"toc":68496},[66587,66592,66595,66599,66618,66622,66628,66633,66636,66641,66644,66650,66654,66657,66742,66746,66749,66752,66791,66800,66803,66807,66890,66893,66900,66907,66998,67005,67009,67012,68006,68011,68015,68018,68032,68307,68311,68314,68409,68412,68418,68421,68427,68431,68445,68448,68451,68457,68465,68490,68493],[11,66588,66589],{},[2718,66590],{"alt":20386,"src":66591},"/static/poker.jpg",[11,66593,66594],{},"I recently took a Hackerrank challenge for a job application that involved poker. I'm not a poker player, so I had a brief moment of panic as I read over the problem the description. In this article I want to do some reflection on how I approached the problem.",[14063,66596,66598],{"id":66597},"the-problem","The Problem",[11,66600,66601,66602,66607,66608,66610,66611,66614,66615,66617],{},"The hackerrank question asked me to write a program that would determine the best poker hand possible in ",[20,66603,66606],{"href":66604,"rel":66605},"https://en.wikipedia.org/wiki/Five-card_draw",[24],"five-card draw"," poker. We are given 10 cards, the first 5 are the current hand, and the second 5 are the next five cards in the deck. We assume that we can see the next five cards (they are not hidden). We want to exchange any ",[30,66609,8521],{}," number of cards (where ",[30,66612,66613],{},"n \u003C= 5",") in our hand for the next ",[30,66616,8521],{}," cards in the deck. For example, we can take out any combination of 2 cards from the hand we are given, but we must replace these two cards with the next two cards from the deck (we can't pick any two cards from the deck).",[14063,66619,66621],{"id":66620},"evaluating-hands","Evaluating hands",[11,66623,66624,66625,643],{},"Suit and value make up the value of playing cards. For example, you can have a 3 of clubs. 3 is the value, clubs is the suit. We can represent this as ",[30,66626,66627],{},"3C",[11,66629,66630],{},[15,66631,66632],{},"Suits",[11,66634,66635],{},"Clubs C\nSpades S\nHeart H\nDiamonds D",[11,66637,66638],{},[15,66639,66640],{},"Value (Rank)",[11,66642,66643],{},"2, 3, 4, 5, 6, 7, 8, 9, 10, Jack, Queen, King, Ace",[459,66645,66648],{"className":66646,"code":66647,"language":997},[995],"values = {\"2\":2, \"3\":3, \"4\":4, \"5\":5, \"6\":6, \"7\":7, \"8\":8, \"9\":9, \"10\":10, \"J\":11, \"Q\":12, \"K\":13, \"A\":14}\n",[30,66649,66647],{"__ignoreMap":464},[56,66651,66653],{"id":66652},"hands","Hands",[11,66655,66656],{},"Here are the hands of poker",[700,66658,66659,66667,66678,66686,66694,66702,66710,66718,66726,66734],{},[79,66660,66661,66662],{},"Royal flush (the problem didn't ask me to consider Royal Flush)",[210,66663,66664],{},[11,66665,66666],{},"A, K, Q, J, 10, all the same suit.",[79,66668,66669,66670],{},"Straight flush",[210,66671,66672],{},[11,66673,66674,66675],{},"Five cards in a sequence, all in the same suit. ",[15,66676,66677],{},"Ace can either come before 2 or come after King.",[79,66679,66680,66681],{},"Four of a kind",[210,66682,66683],{},[11,66684,66685],{},"All four cards of the same rank.",[79,66687,66688,66689],{},"Full house",[210,66690,66691],{},[11,66692,66693],{},"Three of a kind with a pair.",[79,66695,66696,66697],{},"Flush",[210,66698,66699],{},[11,66700,66701],{},"Any five cards of the same suit, but not in a sequence.",[79,66703,66704,66705],{},"Straight",[210,66706,66707],{},[11,66708,66709],{},"Five cards in a sequence, but not of the same suit.",[79,66711,66712,66713],{},"Three of a kind",[210,66714,66715],{},[11,66716,66717],{},"Three cards of the same rank.",[79,66719,66720,66721],{},"Two pair",[210,66722,66723],{},[11,66724,66725],{},"Two different pairs.",[79,66727,66728,66729],{},"Pair",[210,66730,66731],{},[11,66732,66733],{},"Two cards of the same rank.",[79,66735,66736,66737],{},"High Card",[210,66738,66739],{},[11,66740,66741],{},"When you haven't made any of the hands above, the highest card plays.\nIn the example below, the jack plays as the highest card.",[56,66743,66745],{"id":66744},"evaluating-a-hand-of-cards","Evaluating a hand of cards",[11,66747,66748],{},"A hand is five cards. The first thing I did was write out functions to evaluate if a group of 5 cards satisfies the conditions of one of the ten hands.",[11,66750,66751],{},"Here's a sample hand:",[459,66753,66755],{"className":13136,"code":66754,"language":12886,"meta":464,"style":464},"hand = [\"3S\", \"JC\", \"QD\", \"5D\", \"AH\"]\n",[30,66756,66757],{"__ignoreMap":464},[151,66758,66759,66762,66764,66766,66769,66771,66774,66776,66779,66781,66784,66786,66789],{"class":469,"line":470},[151,66760,66761],{"class":503},"hand ",[151,66763,1876],{"class":1869},[151,66765,6604],{"class":503},[151,66767,66768],{"class":481},"\"3S\"",[151,66770,106],{"class":503},[151,66772,66773],{"class":481},"\"JC\"",[151,66775,106],{"class":503},[151,66777,66778],{"class":481},"\"QD\"",[151,66780,106],{"class":503},[151,66782,66783],{"class":481},"\"5D\"",[151,66785,106],{"class":503},[151,66787,66788],{"class":481},"\"AH\"",[151,66790,3691],{"class":503},[11,66792,66793,66794,187,66797,643],{},"To write functions, I reached for using 2 important python features: ",[30,66795,66796],{},"set",[30,66798,66799],{},"defaultdict",[11,66801,66802],{},"Here's an example of a simple function to detect a flush, a hand with cards of all the same suit:",[56,66804,66806],{"id":66805},"checking-a-flush","Checking a flush",[459,66808,66810],{"className":13136,"code":66809,"language":12886,"meta":464,"style":464},"def check_flush(hand):\n    suits = [h[1] for h in hand]\n    if len(set(suits)) == 1:\n      return True\n    else:\n      return False\n",[30,66811,66812,66826,66850,66869,66877,66883],{"__ignoreMap":464},[151,66813,66814,66816,66819,66821,66824],{"class":469,"line":470},[151,66815,16925],{"class":12347},[151,66817,66818],{"class":473}," check_flush",[151,66820,12386],{"class":503},[151,66822,66823],{"class":15232},"hand",[151,66825,15264],{"class":503},[151,66827,66828,66831,66833,66836,66838,66840,66842,66845,66847],{"class":469,"line":488},[151,66829,66830],{"class":503},"    suits ",[151,66832,1876],{"class":1869},[151,66834,66835],{"class":503}," [h[",[151,66837,6760],{"class":477},[151,66839,16654],{"class":503},[151,66841,16732],{"class":1869},[151,66843,66844],{"class":503}," h ",[151,66846,16417],{"class":1869},[151,66848,66849],{"class":503}," hand]\n",[151,66851,66852,66854,66856,66858,66860,66863,66865,66867],{"class":469,"line":500},[151,66853,23327],{"class":1869},[151,66855,45035],{"class":2226},[151,66857,12386],{"class":503},[151,66859,66796],{"class":6205},[151,66861,66862],{"class":503},"(suits)) ",[151,66864,17223],{"class":1869},[151,66866,12448],{"class":477},[151,66868,14372],{"class":503},[151,66870,66871,66874],{"class":469,"line":509},[151,66872,66873],{"class":1869},"      return",[151,66875,66876],{"class":477}," True\n",[151,66878,66879,66881],{"class":469,"line":517},[151,66880,38878],{"class":1869},[151,66882,14372],{"class":503},[151,66884,66885,66887],{"class":469,"line":534},[151,66886,66873],{"class":1869},[151,66888,66889],{"class":477}," False\n",[11,66891,66892],{},"This function creates a list of the suits in our hand, and then counts the unique elements in that list by making it a set. If the length of the set is 1, then all the cards in the hand must be of the same suit.",[11,66894,66895,66896,66899],{},"But wait, what if we have a straight flush? Also, a hand that satisfies a flush could also be described as a two pair hand. The problem asked me to find the highest possible hand for a given set of cards, so I tried to keep things simple by writing a ",[30,66897,66898],{},"check_hand()"," function that checks each hand starting from straight flush down to high card. As soon as a condition for a hand was satisfied, I returned a number that corresponded to the strength of the hand (1 for high card up to 10 for straight flush). The problem didn't include Royal flush, so I will not include that here.",[11,66901,66902,66903,66906],{},"Here's the ",[30,66904,66905],{},"check_hand"," function:",[459,66908,66910],{"className":13136,"code":66909,"language":12886,"meta":464,"style":464},"def check_hand(hand):\n    if check_straight_flush(hand):\n        return 9\n    if check_four_of_a_kind(hand):\n        return 8\n\n    [...]\n    if check_two_pair(hand):\n        return 3\n    if check_pair(hand):\n        return 2\n    return 1\n",[30,66911,66912,66925,66932,66939,66946,66953,66957,66965,66972,66978,66985,66992],{"__ignoreMap":464},[151,66913,66914,66916,66919,66921,66923],{"class":469,"line":470},[151,66915,16925],{"class":12347},[151,66917,66918],{"class":473}," check_hand",[151,66920,12386],{"class":503},[151,66922,66823],{"class":15232},[151,66924,15264],{"class":503},[151,66926,66927,66929],{"class":469,"line":488},[151,66928,23327],{"class":1869},[151,66930,66931],{"class":503}," check_straight_flush(hand):\n",[151,66933,66934,66936],{"class":469,"line":500},[151,66935,16833],{"class":1869},[151,66937,66938],{"class":477}," 9\n",[151,66940,66941,66943],{"class":469,"line":509},[151,66942,23327],{"class":1869},[151,66944,66945],{"class":503}," check_four_of_a_kind(hand):\n",[151,66947,66948,66950],{"class":469,"line":517},[151,66949,16833],{"class":1869},[151,66951,66952],{"class":477}," 8\n",[151,66954,66955],{"class":469,"line":534},[151,66956,1090],{"emptyLinePlaceholder":609},[151,66958,66959,66961,66963],{"class":469,"line":1413},[151,66960,33774],{"class":503},[151,66962,27455],{"class":477},[151,66964,3691],{"class":503},[151,66966,66967,66969],{"class":469,"line":1418},[151,66968,23327],{"class":1869},[151,66970,66971],{"class":503}," check_two_pair(hand):\n",[151,66973,66974,66976],{"class":469,"line":2462},[151,66975,16833],{"class":1869},[151,66977,24461],{"class":477},[151,66979,66980,66982],{"class":469,"line":2471},[151,66981,23327],{"class":1869},[151,66983,66984],{"class":503}," check_pair(hand):\n",[151,66986,66987,66989],{"class":469,"line":2480},[151,66988,16833],{"class":1869},[151,66990,66991],{"class":477}," 2\n",[151,66993,66994,66996],{"class":469,"line":2489},[151,66995,17496],{"class":1869},[151,66997,3181],{"class":477},[11,66999,67000,67001,67004],{},"This function starts checking the most valuable hands. After it checks the second to lowest hand (pair), it returns a value of 1. This value of 1 corresponds to the \"highest card\" hand. Since I'm not comparing the relative value of hands, it doesn't matter what the highest card is, so the number just represents the ",[51,67002,67003],{},"type"," of hand that is the strongest.",[56,67006,67008],{"id":67007},"other-hands","Other hands",[11,67010,67011],{},"Here are the all of the functions I used to detect hands:",[459,67013,67015],{"className":13136,"code":67014,"language":12886,"meta":464,"style":464},"card_order_dict = {\"2\":2, \"3\":3, \"4\":4, \"5\":5, \"6\":6, \"7\":7, \"8\":8, \"9\":9, \"T\":10,\"J\":11, \"Q\":12, \"K\":13, \"A\":14}\n\ndef check_straight_flush(hand):\n    if check_flush(hand) and check_straight(hand):\n        return True\n    else:\n        return False\n\ndef check_four_of_a_kind(hand):\n    values = [i[0] for i in hand]\n    value_counts = defaultdict(lambda:0)\n    for v in values:\n        value_counts[v]+=1\n    if sorted(value_counts.values()) == [1,4]:\n        return True\n    return False\n\ndef check_full_house(hand):\n    values = [i[0] for i in hand]\n    value_counts = defaultdict(lambda:0)\n    for v in values:\n        value_counts[v]+=1\n    if sorted(value_counts.values()) == [2,3]:\n        return True\n    return False\n\ndef check_flush(hand):\n    suits = [i[1] for i in hand]\n    if len(set(suits))==1:\n        return True\n    else:\n        return False\n\ndef check_straight(hand):\n    values = [i[0] for i in hand]\n    value_counts = defaultdict(lambda:0)\n    for v in values:\n        value_counts[v] += 1\n    rank_values = [card_order_dict[i] for i in values]\n    value_range = max(rank_values) - min(rank_values)\n    if len(set(value_counts.values())) == 1 and (value_range==4):\n        return True\n    else:\n        #check straight with low Ace\n        if set(values) == set([\"A\", \"2\", \"3\", \"4\", \"5\"]):\n            return True\n        return False\n\ndef check_three_of_a_kind(hand):\n    values = [i[0] for i in hand]\n    value_counts = defaultdict(lambda:0)\n    for v in values:\n        value_counts[v]+=1\n    if set(value_counts.values()) == set([3,1]):\n        return True\n    else:\n        return False\n\ndef check_two_pairs(hand):\n    values = [i[0] for i in hand]\n    value_counts = defaultdict(lambda:0)\n    for v in values:\n        value_counts[v]+=1\n    if sorted(value_counts.values())==[1,2,2]:\n        return True\n    else:\n        return False\n\ndef check_one_pairs(hand):\n    values = [i[0] for i in hand]\n    value_counts = defaultdict(lambda:0)\n    for v in values:\n        value_counts[v]+=1\n    if 2 in value_counts.values():\n        return True\n    else:\n        return False\n",[30,67016,67017,67143,67147,67160,67172,67178,67184,67190,67194,67207,67230,67247,67259,67268,67289,67295,67301,67305,67318,67338,67354,67364,67372,67392,67398,67404,67408,67420,67440,67459,67465,67471,67477,67481,67494,67514,67530,67540,67549,67568,67588,67616,67622,67628,67633,67669,67675,67681,67685,67698,67718,67734,67744,67752,67774,67780,67786,67792,67796,67809,67829,67845,67855,67863,67888,67894,67900,67906,67910,67923,67943,67959,67969,67977,67988,67994,68000],{"__ignoreMap":464},[151,67018,67019,67022,67024,67026,67029,67031,67033,67035,67038,67040,67042,67044,67047,67049,67051,67053,67056,67058,67060,67062,67065,67067,67069,67071,67074,67076,67078,67080,67083,67085,67087,67089,67092,67094,67096,67098,67101,67103,67105,67107,67110,67112,67114,67116,67119,67121,67123,67125,67128,67130,67132,67134,67136,67138,67141],{"class":469,"line":470},[151,67020,67021],{"class":503},"card_order_dict ",[151,67023,1876],{"class":1869},[151,67025,52023],{"class":503},[151,67027,67028],{"class":481},"\"2\"",[151,67030,208],{"class":503},[151,67032,6619],{"class":477},[151,67034,106],{"class":503},[151,67036,67037],{"class":481},"\"3\"",[151,67039,208],{"class":503},[151,67041,6557],{"class":477},[151,67043,106],{"class":503},[151,67045,67046],{"class":481},"\"4\"",[151,67048,208],{"class":503},[151,67050,9187],{"class":477},[151,67052,106],{"class":503},[151,67054,67055],{"class":481},"\"5\"",[151,67057,208],{"class":503},[151,67059,24380],{"class":477},[151,67061,106],{"class":503},[151,67063,67064],{"class":481},"\"6\"",[151,67066,208],{"class":503},[151,67068,25038],{"class":477},[151,67070,106],{"class":503},[151,67072,67073],{"class":481},"\"7\"",[151,67075,208],{"class":503},[151,67077,25043],{"class":477},[151,67079,106],{"class":503},[151,67081,67082],{"class":481},"\"8\"",[151,67084,208],{"class":503},[151,67086,24369],{"class":477},[151,67088,106],{"class":503},[151,67090,67091],{"class":481},"\"9\"",[151,67093,208],{"class":503},[151,67095,7918],{"class":477},[151,67097,106],{"class":503},[151,67099,67100],{"class":481},"\"T\"",[151,67102,208],{"class":503},[151,67104,12423],{"class":477},[151,67106,3634],{"class":503},[151,67108,67109],{"class":481},"\"J\"",[151,67111,208],{"class":503},[151,67113,42377],{"class":477},[151,67115,106],{"class":503},[151,67117,67118],{"class":481},"\"Q\"",[151,67120,208],{"class":503},[151,67122,42360],{"class":477},[151,67124,106],{"class":503},[151,67126,67127],{"class":481},"\"K\"",[151,67129,208],{"class":503},[151,67131,42327],{"class":477},[151,67133,106],{"class":503},[151,67135,58394],{"class":481},[151,67137,208],{"class":503},[151,67139,67140],{"class":477},"14",[151,67142,6274],{"class":503},[151,67144,67145],{"class":469,"line":488},[151,67146,1090],{"emptyLinePlaceholder":609},[151,67148,67149,67151,67154,67156,67158],{"class":469,"line":500},[151,67150,16925],{"class":12347},[151,67152,67153],{"class":473}," check_straight_flush",[151,67155,12386],{"class":503},[151,67157,66823],{"class":15232},[151,67159,15264],{"class":503},[151,67161,67162,67164,67167,67169],{"class":469,"line":509},[151,67163,23327],{"class":1869},[151,67165,67166],{"class":503}," check_flush(hand) ",[151,67168,40499],{"class":1869},[151,67170,67171],{"class":503}," check_straight(hand):\n",[151,67173,67174,67176],{"class":469,"line":517},[151,67175,16833],{"class":1869},[151,67177,66876],{"class":477},[151,67179,67180,67182],{"class":469,"line":534},[151,67181,38878],{"class":1869},[151,67183,14372],{"class":503},[151,67185,67186,67188],{"class":469,"line":1413},[151,67187,16833],{"class":1869},[151,67189,66889],{"class":477},[151,67191,67192],{"class":469,"line":1418},[151,67193,1090],{"emptyLinePlaceholder":609},[151,67195,67196,67198,67201,67203,67205],{"class":469,"line":2462},[151,67197,16925],{"class":12347},[151,67199,67200],{"class":473}," check_four_of_a_kind",[151,67202,12386],{"class":503},[151,67204,66823],{"class":15232},[151,67206,15264],{"class":503},[151,67208,67209,67212,67214,67217,67219,67221,67223,67226,67228],{"class":469,"line":2471},[151,67210,67211],{"class":503},"    values ",[151,67213,1876],{"class":1869},[151,67215,67216],{"class":503}," [i[",[151,67218,9181],{"class":477},[151,67220,16654],{"class":503},[151,67222,16732],{"class":1869},[151,67224,67225],{"class":503}," i ",[151,67227,16417],{"class":1869},[151,67229,66849],{"class":503},[151,67231,67232,67235,67237,67239,67241,67243,67245],{"class":469,"line":2480},[151,67233,67234],{"class":503},"    value_counts ",[151,67236,1876],{"class":1869},[151,67238,43770],{"class":503},[151,67240,43773],{"class":12347},[151,67242,208],{"class":503},[151,67244,9181],{"class":477},[151,67246,3640],{"class":503},[151,67248,67249,67251,67254,67256],{"class":469,"line":2489},[151,67250,16411],{"class":1869},[151,67252,67253],{"class":503}," v ",[151,67255,16417],{"class":1869},[151,67257,67258],{"class":503}," values:\n",[151,67260,67261,67264,67266],{"class":469,"line":2497},[151,67262,67263],{"class":503},"        value_counts[v]",[151,67265,24780],{"class":1869},[151,67267,1963],{"class":477},[151,67269,67270,67272,67274,67277,67279,67281,67283,67285,67287],{"class":469,"line":3140},[151,67271,23327],{"class":1869},[151,67273,44767],{"class":2226},[151,67275,67276],{"class":503},"(value_counts.values()) ",[151,67278,17223],{"class":1869},[151,67280,6604],{"class":503},[151,67282,6760],{"class":477},[151,67284,3634],{"class":503},[151,67286,9187],{"class":477},[151,67288,17073],{"class":503},[151,67290,67291,67293],{"class":469,"line":3149},[151,67292,16833],{"class":1869},[151,67294,66876],{"class":477},[151,67296,67297,67299],{"class":469,"line":3158},[151,67298,17496],{"class":1869},[151,67300,66889],{"class":477},[151,67302,67303],{"class":469,"line":3167},[151,67304,1090],{"emptyLinePlaceholder":609},[151,67306,67307,67309,67312,67314,67316],{"class":469,"line":3175},[151,67308,16925],{"class":12347},[151,67310,67311],{"class":473}," check_full_house",[151,67313,12386],{"class":503},[151,67315,66823],{"class":15232},[151,67317,15264],{"class":503},[151,67319,67320,67322,67324,67326,67328,67330,67332,67334,67336],{"class":469,"line":3184},[151,67321,67211],{"class":503},[151,67323,1876],{"class":1869},[151,67325,67216],{"class":503},[151,67327,9181],{"class":477},[151,67329,16654],{"class":503},[151,67331,16732],{"class":1869},[151,67333,67225],{"class":503},[151,67335,16417],{"class":1869},[151,67337,66849],{"class":503},[151,67339,67340,67342,67344,67346,67348,67350,67352],{"class":469,"line":3193},[151,67341,67234],{"class":503},[151,67343,1876],{"class":1869},[151,67345,43770],{"class":503},[151,67347,43773],{"class":12347},[151,67349,208],{"class":503},[151,67351,9181],{"class":477},[151,67353,3640],{"class":503},[151,67355,67356,67358,67360,67362],{"class":469,"line":3720},[151,67357,16411],{"class":1869},[151,67359,67253],{"class":503},[151,67361,16417],{"class":1869},[151,67363,67258],{"class":503},[151,67365,67366,67368,67370],{"class":469,"line":3729},[151,67367,67263],{"class":503},[151,67369,24780],{"class":1869},[151,67371,1963],{"class":477},[151,67373,67374,67376,67378,67380,67382,67384,67386,67388,67390],{"class":469,"line":3735},[151,67375,23327],{"class":1869},[151,67377,44767],{"class":2226},[151,67379,67276],{"class":503},[151,67381,17223],{"class":1869},[151,67383,6604],{"class":503},[151,67385,6619],{"class":477},[151,67387,3634],{"class":503},[151,67389,6557],{"class":477},[151,67391,17073],{"class":503},[151,67393,67394,67396],{"class":469,"line":3745},[151,67395,16833],{"class":1869},[151,67397,66876],{"class":477},[151,67399,67400,67402],{"class":469,"line":3754},[151,67401,17496],{"class":1869},[151,67403,66889],{"class":477},[151,67405,67406],{"class":469,"line":3760},[151,67407,1090],{"emptyLinePlaceholder":609},[151,67409,67410,67412,67414,67416,67418],{"class":469,"line":3773},[151,67411,16925],{"class":12347},[151,67413,66818],{"class":473},[151,67415,12386],{"class":503},[151,67417,66823],{"class":15232},[151,67419,15264],{"class":503},[151,67421,67422,67424,67426,67428,67430,67432,67434,67436,67438],{"class":469,"line":3782},[151,67423,66830],{"class":503},[151,67425,1876],{"class":1869},[151,67427,67216],{"class":503},[151,67429,6760],{"class":477},[151,67431,16654],{"class":503},[151,67433,16732],{"class":1869},[151,67435,67225],{"class":503},[151,67437,16417],{"class":1869},[151,67439,66849],{"class":503},[151,67441,67442,67444,67446,67448,67450,67453,67455,67457],{"class":469,"line":3791},[151,67443,23327],{"class":1869},[151,67445,45035],{"class":2226},[151,67447,12386],{"class":503},[151,67449,66796],{"class":6205},[151,67451,67452],{"class":503},"(suits))",[151,67454,17223],{"class":1869},[151,67456,6760],{"class":477},[151,67458,14372],{"class":503},[151,67460,67461,67463],{"class":469,"line":3803},[151,67462,16833],{"class":1869},[151,67464,66876],{"class":477},[151,67466,67467,67469],{"class":469,"line":3811},[151,67468,38878],{"class":1869},[151,67470,14372],{"class":503},[151,67472,67473,67475],{"class":469,"line":3820},[151,67474,16833],{"class":1869},[151,67476,66889],{"class":477},[151,67478,67479],{"class":469,"line":7084},[151,67480,1090],{"emptyLinePlaceholder":609},[151,67482,67483,67485,67488,67490,67492],{"class":469,"line":7148},[151,67484,16925],{"class":12347},[151,67486,67487],{"class":473}," check_straight",[151,67489,12386],{"class":503},[151,67491,66823],{"class":15232},[151,67493,15264],{"class":503},[151,67495,67496,67498,67500,67502,67504,67506,67508,67510,67512],{"class":469,"line":7211},[151,67497,67211],{"class":503},[151,67499,1876],{"class":1869},[151,67501,67216],{"class":503},[151,67503,9181],{"class":477},[151,67505,16654],{"class":503},[151,67507,16732],{"class":1869},[151,67509,67225],{"class":503},[151,67511,16417],{"class":1869},[151,67513,66849],{"class":503},[151,67515,67516,67518,67520,67522,67524,67526,67528],{"class":469,"line":7273},[151,67517,67234],{"class":503},[151,67519,1876],{"class":1869},[151,67521,43770],{"class":503},[151,67523,43773],{"class":12347},[151,67525,208],{"class":503},[151,67527,9181],{"class":477},[151,67529,3640],{"class":503},[151,67531,67532,67534,67536,67538],{"class":469,"line":7335},[151,67533,16411],{"class":1869},[151,67535,67253],{"class":503},[151,67537,16417],{"class":1869},[151,67539,67258],{"class":503},[151,67541,67542,67545,67547],{"class":469,"line":7398},[151,67543,67544],{"class":503},"        value_counts[v] ",[151,67546,24780],{"class":1869},[151,67548,3181],{"class":477},[151,67550,67551,67554,67556,67559,67561,67563,67565],{"class":469,"line":7462},[151,67552,67553],{"class":503},"    rank_values ",[151,67555,1876],{"class":1869},[151,67557,67558],{"class":503}," [card_order_dict[i] ",[151,67560,16732],{"class":1869},[151,67562,67225],{"class":503},[151,67564,16417],{"class":1869},[151,67566,67567],{"class":503}," values]\n",[151,67569,67570,67573,67575,67578,67581,67583,67585],{"class":469,"line":7467},[151,67571,67572],{"class":503},"    value_range ",[151,67574,1876],{"class":1869},[151,67576,67577],{"class":2226}," max",[151,67579,67580],{"class":503},"(rank_values) ",[151,67582,12445],{"class":1869},[151,67584,4046],{"class":2226},[151,67586,67587],{"class":503},"(rank_values)\n",[151,67589,67590,67592,67594,67596,67598,67601,67603,67605,67607,67610,67612,67614],{"class":469,"line":7532},[151,67591,23327],{"class":1869},[151,67593,45035],{"class":2226},[151,67595,12386],{"class":503},[151,67597,66796],{"class":6205},[151,67599,67600],{"class":503},"(value_counts.values())) ",[151,67602,17223],{"class":1869},[151,67604,12448],{"class":477},[151,67606,2181],{"class":1869},[151,67608,67609],{"class":503}," (value_range",[151,67611,17223],{"class":1869},[151,67613,9187],{"class":477},[151,67615,15264],{"class":503},[151,67617,67618,67620],{"class":469,"line":7537},[151,67619,16833],{"class":1869},[151,67621,66876],{"class":477},[151,67623,67624,67626],{"class":469,"line":7603},[151,67625,38878],{"class":1869},[151,67627,14372],{"class":503},[151,67629,67630],{"class":469,"line":7608},[151,67631,67632],{"class":1527},"        #check straight with low Ace\n",[151,67634,67635,67637,67639,67642,67644,67646,67648,67650,67652,67654,67656,67658,67660,67662,67664,67666],{"class":469,"line":7673},[151,67636,23357],{"class":1869},[151,67638,2309],{"class":6205},[151,67640,67641],{"class":503},"(values) ",[151,67643,17223],{"class":1869},[151,67645,2309],{"class":6205},[151,67647,60170],{"class":503},[151,67649,58394],{"class":481},[151,67651,106],{"class":503},[151,67653,67028],{"class":481},[151,67655,106],{"class":503},[151,67657,67037],{"class":481},[151,67659,106],{"class":503},[151,67661,67046],{"class":481},[151,67663,106],{"class":503},[151,67665,67055],{"class":481},[151,67667,67668],{"class":503},"]):\n",[151,67670,67671,67673],{"class":469,"line":7678},[151,67672,15386],{"class":1869},[151,67674,66876],{"class":477},[151,67676,67677,67679],{"class":469,"line":7708},[151,67678,16833],{"class":1869},[151,67680,66889],{"class":477},[151,67682,67683],{"class":469,"line":7713},[151,67684,1090],{"emptyLinePlaceholder":609},[151,67686,67687,67689,67692,67694,67696],{"class":469,"line":7746},[151,67688,16925],{"class":12347},[151,67690,67691],{"class":473}," check_three_of_a_kind",[151,67693,12386],{"class":503},[151,67695,66823],{"class":15232},[151,67697,15264],{"class":503},[151,67699,67700,67702,67704,67706,67708,67710,67712,67714,67716],{"class":469,"line":7751},[151,67701,67211],{"class":503},[151,67703,1876],{"class":1869},[151,67705,67216],{"class":503},[151,67707,9181],{"class":477},[151,67709,16654],{"class":503},[151,67711,16732],{"class":1869},[151,67713,67225],{"class":503},[151,67715,16417],{"class":1869},[151,67717,66849],{"class":503},[151,67719,67720,67722,67724,67726,67728,67730,67732],{"class":469,"line":7816},[151,67721,67234],{"class":503},[151,67723,1876],{"class":1869},[151,67725,43770],{"class":503},[151,67727,43773],{"class":12347},[151,67729,208],{"class":503},[151,67731,9181],{"class":477},[151,67733,3640],{"class":503},[151,67735,67736,67738,67740,67742],{"class":469,"line":7821},[151,67737,16411],{"class":1869},[151,67739,67253],{"class":503},[151,67741,16417],{"class":1869},[151,67743,67258],{"class":503},[151,67745,67746,67748,67750],{"class":469,"line":7847},[151,67747,67263],{"class":503},[151,67749,24780],{"class":1869},[151,67751,1963],{"class":477},[151,67753,67754,67756,67758,67760,67762,67764,67766,67768,67770,67772],{"class":469,"line":7852},[151,67755,23327],{"class":1869},[151,67757,2309],{"class":6205},[151,67759,67276],{"class":503},[151,67761,17223],{"class":1869},[151,67763,2309],{"class":6205},[151,67765,60170],{"class":503},[151,67767,6557],{"class":477},[151,67769,3634],{"class":503},[151,67771,6760],{"class":477},[151,67773,67668],{"class":503},[151,67775,67776,67778],{"class":469,"line":7887},[151,67777,16833],{"class":1869},[151,67779,66876],{"class":477},[151,67781,67782,67784],{"class":469,"line":7892},[151,67783,38878],{"class":1869},[151,67785,14372],{"class":503},[151,67787,67788,67790],{"class":469,"line":7924},[151,67789,16833],{"class":1869},[151,67791,66889],{"class":477},[151,67793,67794],{"class":469,"line":7929},[151,67795,1090],{"emptyLinePlaceholder":609},[151,67797,67798,67800,67803,67805,67807],{"class":469,"line":7991},[151,67799,16925],{"class":12347},[151,67801,67802],{"class":473}," check_two_pairs",[151,67804,12386],{"class":503},[151,67806,66823],{"class":15232},[151,67808,15264],{"class":503},[151,67810,67811,67813,67815,67817,67819,67821,67823,67825,67827],{"class":469,"line":7996},[151,67812,67211],{"class":503},[151,67814,1876],{"class":1869},[151,67816,67216],{"class":503},[151,67818,9181],{"class":477},[151,67820,16654],{"class":503},[151,67822,16732],{"class":1869},[151,67824,67225],{"class":503},[151,67826,16417],{"class":1869},[151,67828,66849],{"class":503},[151,67830,67831,67833,67835,67837,67839,67841,67843],{"class":469,"line":8078},[151,67832,67234],{"class":503},[151,67834,1876],{"class":1869},[151,67836,43770],{"class":503},[151,67838,43773],{"class":12347},[151,67840,208],{"class":503},[151,67842,9181],{"class":477},[151,67844,3640],{"class":503},[151,67846,67847,67849,67851,67853],{"class":469,"line":8140},[151,67848,16411],{"class":1869},[151,67850,67253],{"class":503},[151,67852,16417],{"class":1869},[151,67854,67258],{"class":503},[151,67856,67857,67859,67861],{"class":469,"line":8145},[151,67858,67263],{"class":503},[151,67860,24780],{"class":1869},[151,67862,1963],{"class":477},[151,67864,67865,67867,67869,67872,67874,67876,67878,67880,67882,67884,67886],{"class":469,"line":8259},[151,67866,23327],{"class":1869},[151,67868,44767],{"class":2226},[151,67870,67871],{"class":503},"(value_counts.values())",[151,67873,17223],{"class":1869},[151,67875,6698],{"class":503},[151,67877,6760],{"class":477},[151,67879,3634],{"class":503},[151,67881,6619],{"class":477},[151,67883,3634],{"class":503},[151,67885,6619],{"class":477},[151,67887,17073],{"class":503},[151,67889,67890,67892],{"class":469,"line":8264},[151,67891,16833],{"class":1869},[151,67893,66876],{"class":477},[151,67895,67896,67898],{"class":469,"line":8613},[151,67897,38878],{"class":1869},[151,67899,14372],{"class":503},[151,67901,67902,67904],{"class":469,"line":8678},[151,67903,16833],{"class":1869},[151,67905,66889],{"class":477},[151,67907,67908],{"class":469,"line":8742},[151,67909,1090],{"emptyLinePlaceholder":609},[151,67911,67912,67914,67917,67919,67921],{"class":469,"line":8806},[151,67913,16925],{"class":12347},[151,67915,67916],{"class":473}," check_one_pairs",[151,67918,12386],{"class":503},[151,67920,66823],{"class":15232},[151,67922,15264],{"class":503},[151,67924,67925,67927,67929,67931,67933,67935,67937,67939,67941],{"class":469,"line":8870},[151,67926,67211],{"class":503},[151,67928,1876],{"class":1869},[151,67930,67216],{"class":503},[151,67932,9181],{"class":477},[151,67934,16654],{"class":503},[151,67936,16732],{"class":1869},[151,67938,67225],{"class":503},[151,67940,16417],{"class":1869},[151,67942,66849],{"class":503},[151,67944,67945,67947,67949,67951,67953,67955,67957],{"class":469,"line":8875},[151,67946,67234],{"class":503},[151,67948,1876],{"class":1869},[151,67950,43770],{"class":503},[151,67952,43773],{"class":12347},[151,67954,208],{"class":503},[151,67956,9181],{"class":477},[151,67958,3640],{"class":503},[151,67960,67961,67963,67965,67967],{"class":469,"line":8881},[151,67962,16411],{"class":1869},[151,67964,67253],{"class":503},[151,67966,16417],{"class":1869},[151,67968,67258],{"class":503},[151,67970,67971,67973,67975],{"class":469,"line":8886},[151,67972,67263],{"class":503},[151,67974,24780],{"class":1869},[151,67976,1963],{"class":477},[151,67978,67979,67981,67983,67985],{"class":469,"line":8892},[151,67980,23327],{"class":1869},[151,67982,59070],{"class":477},[151,67984,2820],{"class":1869},[151,67986,67987],{"class":503}," value_counts.values():\n",[151,67989,67990,67992],{"class":469,"line":8963},[151,67991,16833],{"class":1869},[151,67993,66876],{"class":477},[151,67995,67996,67998],{"class":469,"line":8969},[151,67997,38878],{"class":1869},[151,67999,14372],{"class":503},[151,68001,68002,68004],{"class":469,"line":15001},[151,68003,16833],{"class":1869},[151,68005,66889],{"class":477},[11,68007,68008,68010],{},[30,68009,66799],{}," is a great built-in that is good to use when you don't know what elements will be in your dictionary, but you know what the initial values of any key that could be added should be. We don't need it here, but the alternative would be to write a very long dictionary where keys are the possible card values and the values of each key is 0.",[14063,68012,68014],{"id":68013},"finding-the-best-hand","Finding the best hand",[11,68016,68017],{},"It would certainly be cleaner and more efficient to write out the above functions into one large function, but I wanted to keep things simple as I was under time constraints.",[11,68019,68020,68021,68024,68025,68028,68029,68031],{},"The next step in the problem is to determine the best possible hand we can get given the hand we are dealt and the 5 cards on top of the deck. I decided to first solve this problem with brute force. Here was my logic for this part: use ",[30,68022,68023],{},"itertools"," to get all combinations of groups of 0, 1, 2, 3, 4 and 5 cards from my hand and add the first ",[30,68026,68027],{},"5 - n"," cards from the deck so we get a five card deck. For each combination of cards we can run ",[30,68030,66898],{}," and keep track of the highest rank hand, and then return that hand as the best hand. Here's the code I wrote for this part of the problem:",[459,68033,68035],{"className":13136,"code":68034,"language":12886,"meta":464,"style":464},"from itertools import combinations\n\nhand_dict = {9:\"straight-flush\", 8:\"four-of-a-kind\", 7:\"full-house\", 6:\"flush\", 5:\"straight\", 4:\"three-of-a-kind\", 3:\"two-pairs\", 2:\"one-pair\", 1:\"highest-card\"}\n\n#exhaustive search using itertools.combinations\ndef play(cards):\n    hand = cards[:5]\n    deck = cards[5:]\n    best_hand = 0\n    for i in range(6):\n        possible_combos = combinations(hand, 5-i)\n        for c in possible_combos:\n            current_hand = list(c) + deck[:i]\n            hand_value = check_hand(current_hand)\n            if hand_value > best_hand:\n                best_hand = hand_value\n\n    return hand_dict[best_hand]\n",[30,68036,68037,68049,68053,68143,68147,68152,68166,68180,68194,68203,68219,68236,68247,68264,68274,68286,68296,68300],{"__ignoreMap":464},[151,68038,68039,68041,68044,68046],{"class":469,"line":470},[151,68040,16853],{"class":1869},[151,68042,68043],{"class":503}," itertools ",[151,68045,16859],{"class":1869},[151,68047,68048],{"class":503}," combinations\n",[151,68050,68051],{"class":469,"line":488},[151,68052,1090],{"emptyLinePlaceholder":609},[151,68054,68055,68058,68060,68062,68064,68066,68069,68071,68073,68075,68078,68080,68082,68084,68087,68089,68091,68093,68096,68098,68100,68102,68105,68107,68109,68111,68114,68116,68118,68120,68123,68125,68127,68129,68132,68134,68136,68138,68141],{"class":469,"line":500},[151,68056,68057],{"class":503},"hand_dict ",[151,68059,1876],{"class":1869},[151,68061,52023],{"class":503},[151,68063,7918],{"class":477},[151,68065,208],{"class":503},[151,68067,68068],{"class":481},"\"straight-flush\"",[151,68070,106],{"class":503},[151,68072,24369],{"class":477},[151,68074,208],{"class":503},[151,68076,68077],{"class":481},"\"four-of-a-kind\"",[151,68079,106],{"class":503},[151,68081,25043],{"class":477},[151,68083,208],{"class":503},[151,68085,68086],{"class":481},"\"full-house\"",[151,68088,106],{"class":503},[151,68090,25038],{"class":477},[151,68092,208],{"class":503},[151,68094,68095],{"class":481},"\"flush\"",[151,68097,106],{"class":503},[151,68099,24380],{"class":477},[151,68101,208],{"class":503},[151,68103,68104],{"class":481},"\"straight\"",[151,68106,106],{"class":503},[151,68108,9187],{"class":477},[151,68110,208],{"class":503},[151,68112,68113],{"class":481},"\"three-of-a-kind\"",[151,68115,106],{"class":503},[151,68117,6557],{"class":477},[151,68119,208],{"class":503},[151,68121,68122],{"class":481},"\"two-pairs\"",[151,68124,106],{"class":503},[151,68126,6619],{"class":477},[151,68128,208],{"class":503},[151,68130,68131],{"class":481},"\"one-pair\"",[151,68133,106],{"class":503},[151,68135,6760],{"class":477},[151,68137,208],{"class":503},[151,68139,68140],{"class":481},"\"highest-card\"",[151,68142,6274],{"class":503},[151,68144,68145],{"class":469,"line":509},[151,68146,1090],{"emptyLinePlaceholder":609},[151,68148,68149],{"class":469,"line":517},[151,68150,68151],{"class":1527},"#exhaustive search using itertools.combinations\n",[151,68153,68154,68156,68159,68161,68164],{"class":469,"line":534},[151,68155,16925],{"class":12347},[151,68157,68158],{"class":473}," play",[151,68160,12386],{"class":503},[151,68162,68163],{"class":15232},"cards",[151,68165,15264],{"class":503},[151,68167,68168,68171,68173,68176,68178],{"class":469,"line":1413},[151,68169,68170],{"class":503},"    hand ",[151,68172,1876],{"class":1869},[151,68174,68175],{"class":503}," cards[:",[151,68177,24380],{"class":477},[151,68179,3691],{"class":503},[151,68181,68182,68185,68187,68190,68192],{"class":469,"line":1418},[151,68183,68184],{"class":503},"    deck ",[151,68186,1876],{"class":1869},[151,68188,68189],{"class":503}," cards[",[151,68191,24380],{"class":477},[151,68193,59118],{"class":503},[151,68195,68196,68199,68201],{"class":469,"line":2462},[151,68197,68198],{"class":503},"    best_hand ",[151,68200,1876],{"class":1869},[151,68202,57871],{"class":477},[151,68204,68205,68207,68209,68211,68213,68215,68217],{"class":469,"line":2471},[151,68206,16411],{"class":1869},[151,68208,67225],{"class":503},[151,68210,16417],{"class":1869},[151,68212,2793],{"class":2226},[151,68214,12386],{"class":503},[151,68216,25038],{"class":477},[151,68218,15264],{"class":503},[151,68220,68221,68224,68226,68229,68231,68233],{"class":469,"line":2480},[151,68222,68223],{"class":503},"        possible_combos ",[151,68225,1876],{"class":1869},[151,68227,68228],{"class":503}," combinations(hand, ",[151,68230,24380],{"class":477},[151,68232,12445],{"class":1869},[151,68234,68235],{"class":503},"i)\n",[151,68237,68238,68240,68242,68244],{"class":469,"line":2489},[151,68239,16616],{"class":1869},[151,68241,63611],{"class":503},[151,68243,16417],{"class":1869},[151,68245,68246],{"class":503}," possible_combos:\n",[151,68248,68249,68252,68254,68256,68259,68261],{"class":469,"line":2497},[151,68250,68251],{"class":503},"            current_hand ",[151,68253,1876],{"class":1869},[151,68255,59145],{"class":6205},[151,68257,68258],{"class":503},"(c) ",[151,68260,22885],{"class":1869},[151,68262,68263],{"class":503}," deck[:i]\n",[151,68265,68266,68269,68271],{"class":469,"line":3140},[151,68267,68268],{"class":503},"            hand_value ",[151,68270,1876],{"class":1869},[151,68272,68273],{"class":503}," check_hand(current_hand)\n",[151,68275,68276,68278,68281,68283],{"class":469,"line":3149},[151,68277,40442],{"class":1869},[151,68279,68280],{"class":503}," hand_value ",[151,68282,3663],{"class":1869},[151,68284,68285],{"class":503}," best_hand:\n",[151,68287,68288,68291,68293],{"class":469,"line":3158},[151,68289,68290],{"class":503},"                best_hand ",[151,68292,1876],{"class":1869},[151,68294,68295],{"class":503}," hand_value\n",[151,68297,68298],{"class":469,"line":3167},[151,68299,1090],{"emptyLinePlaceholder":609},[151,68301,68302,68304],{"class":469,"line":3175},[151,68303,17496],{"class":1869},[151,68305,68306],{"class":503}," hand_dict[best_hand]\n",[14063,68308,68310],{"id":68309},"checking-test-cases","Checking test cases",[11,68312,68313],{},"Lastly, I need to check each hand and print out the best hand possible. Here's the loop I wrote to do this:",[459,68315,68317],{"className":13136,"code":68316,"language":12886,"meta":464,"style":464},"for i in sys.stdin.readlines():\n    cards = list(map(lambda x:x, i.split()))\n    hand = cards[:5]\n    deck = cards[5:]\n    print(\"Hand:\", \" \".join(hand), \"Deck:\", \" \".join(deck), \"Best hand:\", play(cards))\n",[30,68318,68319,68330,68353,68365,68377],{"__ignoreMap":464},[151,68320,68321,68323,68325,68327],{"class":469,"line":470},[151,68322,16732],{"class":1869},[151,68324,67225],{"class":503},[151,68326,16417],{"class":1869},[151,68328,68329],{"class":503}," sys.stdin.readlines():\n",[151,68331,68332,68335,68337,68339,68341,68344,68346,68348,68350],{"class":469,"line":488},[151,68333,68334],{"class":503},"    cards ",[151,68336,1876],{"class":1869},[151,68338,59145],{"class":6205},[151,68340,12386],{"class":503},[151,68342,68343],{"class":2226},"map",[151,68345,12386],{"class":503},[151,68347,43773],{"class":12347},[151,68349,27729],{"class":15232},[151,68351,68352],{"class":503},":x, i.split()))\n",[151,68354,68355,68357,68359,68361,68363],{"class":469,"line":500},[151,68356,68170],{"class":503},[151,68358,1876],{"class":1869},[151,68360,68175],{"class":503},[151,68362,24380],{"class":477},[151,68364,3691],{"class":503},[151,68366,68367,68369,68371,68373,68375],{"class":469,"line":509},[151,68368,68184],{"class":503},[151,68370,1876],{"class":1869},[151,68372,68189],{"class":503},[151,68374,24380],{"class":477},[151,68376,59118],{"class":503},[151,68378,68379,68381,68383,68386,68388,68390,68393,68396,68398,68400,68403,68406],{"class":469,"line":517},[151,68380,24285],{"class":2226},[151,68382,12386],{"class":503},[151,68384,68385],{"class":481},"\"Hand:\"",[151,68387,106],{"class":503},[151,68389,24311],{"class":481},[151,68391,68392],{"class":503},".join(hand), ",[151,68394,68395],{"class":481},"\"Deck:\"",[151,68397,106],{"class":503},[151,68399,24311],{"class":481},[151,68401,68402],{"class":503},".join(deck), ",[151,68404,68405],{"class":481},"\"Best hand:\"",[151,68407,68408],{"class":503},", play(cards))\n",[11,68410,68411],{},"This will accept one round of cards per line:",[459,68413,68416],{"className":68414,"code":68415,"language":997},[995],"2C 3D 4S 5D 7H KD QH 6C JH 2D\n",[30,68417,68415],{"__ignoreMap":464},[11,68419,68420],{},"and it will output the following:",[459,68422,68425],{"className":68423,"code":68424,"language":997},[995],"Hand: 2C 3D 4S 5D 7H Deck: KD QH 6C JH 2D Best hand: straight\n",[30,68426,68424],{"__ignoreMap":464},[14063,68428,68430],{"id":68429},"optimization","Optimization",[11,68432,68433,68434,68437,68438,68440,68441,68444],{},"This was an interesting problem to deal with as the solution contained several parts that worked together. While solving the problem I aimed worked through to the end leaving some parts to come back to that I felt confident in solving. Instead of writing each function to check differnt hands at the beginning, I filled most of these functions with ",[30,68435,68436],{},"pass"," and moved on to write the next part that involves checking each different combination of cards. Recently having worked through python's ",[30,68439,68023],{}," exercises on Hackerrank, the ",[30,68442,68443],{},"combinations"," functions was fresh in my mind.",[11,68446,68447],{},"While I was able to arrive at a solution that satisfied the test cases, I did not have time to think about the efficiency or Big O analysis of the problem.",[11,68449,68450],{},"There is obviously some refactoring that I could do to make things cleaner. With more time I would take an object oriented approach by making classes for cards and hands, and adding class methods to evaluate the hands.",[11,68452,68453,68454,68456],{},"For each round, we have to run ",[30,68455,66898],{}," on each hand combination. Let's think about how many hands we have to evaluate:",[11,68458,68459,68460,68462,68463,643],{},"We have to consider combinations of cards formed by taking out groups of 0, 1, 2, 3, 4 and 5 cards and adding the next number of cards in the deck that bring the total card count to 5, which means we have to do 5C0 + 5C1 + 5C2 + 5C3 + 5C4 + 5C5 calls to ",[30,68461,66898],{},". So the sum of total calls is 1 + 5 + 10 + 10 + 5 + 1 = ",[15,68464,9302],{},[11,68466,68467,68468,106,68471,68474,68475,68478,68479,68481,68482,68485,68486,68489],{},"For each of these 32 calls that happen when we run ",[30,68469,68470],{},"play()",[30,68472,68473],{},"check_hands()"," runs through each of the ",[30,68476,68477],{},"check_"," functions starting with the highest value hand. As soon as it finds a \"match\", ",[30,68480,68473],{}," returns a number value (",[30,68483,68484],{},"hand_value",") corresponding to straight flush, four of a kind, etc. This value is then compared with the highest value that has been previously found (",[30,68487,68488],{},"best_hand",") and replaces that value if the current hand's hand rank has a higher value.",[11,68491,68492],{},"I'm not sure if there is faster way to find the best hand than the brute force method I implemented.",[589,68494,68495],{},"html pre.shiki code .sMOD_, html code.shiki .sMOD_{--shiki-default:#24292E;--shiki-dark:#E1E4E8;--shiki-sepia:#F8F8F2}html pre.shiki code .sC2Qs, html code.shiki .sC2Qs{--shiki-default:#D73A49;--shiki-dark:#F97583;--shiki-sepia:#F92672}html pre.shiki code .sstjo, html code.shiki .sstjo{--shiki-default:#032F62;--shiki-dark:#9ECBFF;--shiki-sepia:#E6DB74}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html .sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html.sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html pre.shiki code .sq6CD, html code.shiki .sq6CD{--shiki-default:#D73A49;--shiki-default-font-style:inherit;--shiki-dark:#F97583;--shiki-dark-font-style:inherit;--shiki-sepia:#66D9EF;--shiki-sepia-font-style:italic}html pre.shiki code .srTi1, html code.shiki .srTi1{--shiki-default:#6F42C1;--shiki-dark:#B392F0;--shiki-sepia:#A6E22E}html pre.shiki code .so59x, html code.shiki .so59x{--shiki-default:#24292E;--shiki-default-font-style:inherit;--shiki-dark:#E1E4E8;--shiki-dark-font-style:inherit;--shiki-sepia:#FD971F;--shiki-sepia-font-style:italic}html pre.shiki code .s7F3e, html code.shiki .s7F3e{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#AE81FF}html pre.shiki code .sTrkL, html code.shiki .sTrkL{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#66D9EF}html pre.shiki code .s-m8C, html code.shiki .s-m8C{--shiki-default:#005CC5;--shiki-default-font-style:inherit;--shiki-dark:#79B8FF;--shiki-dark-font-style:inherit;--shiki-sepia:#66D9EF;--shiki-sepia-font-style:italic}html pre.shiki code .s8-w5, html code.shiki .s8-w5{--shiki-default:#6A737D;--shiki-dark:#6A737D;--shiki-sepia:#88846F}",{"title":464,"searchDepth":488,"depth":488,"links":68497},[68498,68499,68500,68501],{"id":66652,"depth":488,"text":66653},{"id":66744,"depth":488,"text":66745},{"id":66805,"depth":488,"text":66806},{"id":67007,"depth":488,"text":67008},"2018-01-02",{"layout":48045},"/2018/01/02/checking-poker-hands-with-python",{"title":66584,"description":464},"2018/01/02/checking-poker-hands-with-python",[12886,68508],"poker","qIdjwmcktcOacdENgNQsRXmhHSbiMCneSjD6gUJnRto",{"id":68511,"title":68512,"body":68513,"comments":609,"date":69026,"description":464,"draft":602,"extension":605,"external":606,"image":68519,"meta":69027,"navigation":609,"path":69028,"seo":69029,"stem":69030,"tags":69031,"__hash__":69033},"blog/2017/12/22/getting-started-with-twisted.md","Getting started with Python's Twisted Framework",{"type":8,"value":68514,"toc":69022},[68515,68520,68527,68531,68537,68540,68545,68549,68552,68613,68622,68626,68894,68898,69019],[11,68516,68517],{},[2718,68518],{"alt":20386,"src":68519},"/static/twisted-snakes.png",[11,68521,68522,68523,68526],{},"In this article I'm going to be exploring python's twisted framework. I'm working through the ",[51,68524,68525],{},"Twisted Network Programming Essentials"," book from O'Reilly.",[56,68528,68530],{"id":68529},"installation","Installation",[459,68532,68535],{"className":68533,"code":68534,"language":997},[995]," $ pip install twisted\n",[30,68536,68534],{"__ignoreMap":464},[11,68538,68539],{},"The main idea behind Twisted is that it gives us the parallelism of multithreading programming with the ease of reasoning of single threaded programming.",[11,68541,68542],{},[2718,68543],{"alt":20386,"src":68544},"/static/event-driven.jpg",[56,68546,68548],{"id":68547},"the-reactor","The Reactor",[11,68550,68551],{},"This is the core of Twisted. Here is a simple explanation of what the reactor does with psuedo-code:",[459,68553,68555],{"className":13136,"code":68554,"language":12886,"meta":464,"style":464},"while True:\n    timeout = timeout_until_next_timed_event()\n    events = wait_for_events(timeout)\n    events += timed_events_until(now())\n    for event in events:\n        event.process()\n",[30,68556,68557,68567,68577,68587,68596,68608],{"__ignoreMap":464},[151,68558,68559,68562,68565],{"class":469,"line":470},[151,68560,68561],{"class":1869},"while",[151,68563,68564],{"class":477}," True",[151,68566,14372],{"class":503},[151,68568,68569,68572,68574],{"class":469,"line":488},[151,68570,68571],{"class":503},"    timeout ",[151,68573,1876],{"class":1869},[151,68575,68576],{"class":503}," timeout_until_next_timed_event()\n",[151,68578,68579,68582,68584],{"class":469,"line":500},[151,68580,68581],{"class":503},"    events ",[151,68583,1876],{"class":1869},[151,68585,68586],{"class":503}," wait_for_events(timeout)\n",[151,68588,68589,68591,68593],{"class":469,"line":509},[151,68590,68581],{"class":503},[151,68592,24780],{"class":1869},[151,68594,68595],{"class":503}," timed_events_until(now())\n",[151,68597,68598,68600,68603,68605],{"class":469,"line":517},[151,68599,16411],{"class":1869},[151,68601,68602],{"class":503}," event ",[151,68604,16417],{"class":1869},[151,68606,68607],{"class":503}," events:\n",[151,68609,68610],{"class":469,"line":534},[151,68611,68612],{"class":503},"        event.process()\n",[11,68614,68615,68616,187,68619,208],{},"Here's a simple echo server/client example that illustrates how the reactor works. It is composed of ",[30,68617,68618],{},"echoclient.py",[30,68620,68621],{},"echoserver.py",[11,68623,68624],{},[51,68625,68618],{},[459,68627,68629],{"className":13136,"code":68628,"language":12886,"meta":464,"style":464},"from twisted.internet import reactor, protocol\n\nclass EchoClient(protocol.Protocol):\n    def connectionMade(self):\n        self.transport.write(u\"Hello, world!\".encode('utf-8'))\n\n    def dataReceived(self, data):\n        print(\"Server said:\", data)\n        self.transport.loseConnection()\n\nclass EchoFactory(protocol.ClientFactory):\n    def buildProtocol(self, addr):\n        return EchoClient()\n\n    def clientConnectionFailed(self, connector, reason):\n        print(\"Connection failed.\")\n        reactor.stop()\n\n    def clientConnectionLost(self, connector, reason):\n        print(\"Connection lost.\")\n        reactor.stop()\n\nreactor.connectTCP(\"localhost\", 8000, EchoFactory())\nreactor.run()\n",[30,68630,68631,68643,68647,68666,68679,68700,68704,68721,68733,68740,68744,68762,68780,68787,68791,68814,68825,68830,68834,68855,68866,68870,68874,68889],{"__ignoreMap":464},[151,68632,68633,68635,68638,68640],{"class":469,"line":470},[151,68634,16853],{"class":1869},[151,68636,68637],{"class":503}," twisted.internet ",[151,68639,16859],{"class":1869},[151,68641,68642],{"class":503}," reactor, protocol\n",[151,68644,68645],{"class":469,"line":488},[151,68646,1090],{"emptyLinePlaceholder":609},[151,68648,68649,68651,68654,68656,68659,68661,68664],{"class":469,"line":500},[151,68650,16519],{"class":12347},[151,68652,68653],{"class":15254}," EchoClient",[151,68655,12386],{"class":503},[151,68657,68658],{"class":15260},"protocol",[151,68660,643],{"class":503},[151,68662,68663],{"class":15260},"Protocol",[151,68665,15264],{"class":503},[151,68667,68668,68670,68673,68675,68677],{"class":469,"line":509},[151,68669,16566],{"class":12347},[151,68671,68672],{"class":473}," connectionMade",[151,68674,12386],{"class":503},[151,68676,15277],{"class":15232},[151,68678,15264],{"class":503},[151,68680,68681,68683,68686,68689,68692,68695,68698],{"class":469,"line":517},[151,68682,37901],{"class":15289},[151,68684,68685],{"class":503},".transport.write(",[151,68687,68688],{"class":12347},"u",[151,68690,68691],{"class":481},"\"Hello, world!\"",[151,68693,68694],{"class":503},".encode(",[151,68696,68697],{"class":481},"'utf-8'",[151,68699,12451],{"class":503},[151,68701,68702],{"class":469,"line":534},[151,68703,1090],{"emptyLinePlaceholder":609},[151,68705,68706,68708,68711,68713,68715,68717,68719],{"class":469,"line":1413},[151,68707,16566],{"class":12347},[151,68709,68710],{"class":473}," dataReceived",[151,68712,12386],{"class":503},[151,68714,15277],{"class":15232},[151,68716,106],{"class":503},[151,68718,12355],{"class":15232},[151,68720,15264],{"class":503},[151,68722,68723,68725,68727,68730],{"class":469,"line":1418},[151,68724,18355],{"class":2226},[151,68726,12386],{"class":503},[151,68728,68729],{"class":481},"\"Server said:\"",[151,68731,68732],{"class":503},", data)\n",[151,68734,68735,68737],{"class":469,"line":2462},[151,68736,37901],{"class":15289},[151,68738,68739],{"class":503},".transport.loseConnection()\n",[151,68741,68742],{"class":469,"line":2471},[151,68743,1090],{"emptyLinePlaceholder":609},[151,68745,68746,68748,68751,68753,68755,68757,68760],{"class":469,"line":2480},[151,68747,16519],{"class":12347},[151,68749,68750],{"class":15254}," EchoFactory",[151,68752,12386],{"class":503},[151,68754,68658],{"class":15260},[151,68756,643],{"class":503},[151,68758,68759],{"class":15260},"ClientFactory",[151,68761,15264],{"class":503},[151,68763,68764,68766,68769,68771,68773,68775,68778],{"class":469,"line":2489},[151,68765,16566],{"class":12347},[151,68767,68768],{"class":473}," buildProtocol",[151,68770,12386],{"class":503},[151,68772,15277],{"class":15232},[151,68774,106],{"class":503},[151,68776,68777],{"class":15232},"addr",[151,68779,15264],{"class":503},[151,68781,68782,68784],{"class":469,"line":2497},[151,68783,16833],{"class":1869},[151,68785,68786],{"class":503}," EchoClient()\n",[151,68788,68789],{"class":469,"line":3140},[151,68790,1090],{"emptyLinePlaceholder":609},[151,68792,68793,68795,68798,68800,68802,68804,68807,68809,68812],{"class":469,"line":3149},[151,68794,16566],{"class":12347},[151,68796,68797],{"class":473}," clientConnectionFailed",[151,68799,12386],{"class":503},[151,68801,15277],{"class":15232},[151,68803,106],{"class":503},[151,68805,68806],{"class":15232},"connector",[151,68808,106],{"class":503},[151,68810,68811],{"class":15232},"reason",[151,68813,15264],{"class":503},[151,68815,68816,68818,68820,68823],{"class":469,"line":3158},[151,68817,18355],{"class":2226},[151,68819,12386],{"class":503},[151,68821,68822],{"class":481},"\"Connection failed.\"",[151,68824,3640],{"class":503},[151,68826,68827],{"class":469,"line":3167},[151,68828,68829],{"class":503},"        reactor.stop()\n",[151,68831,68832],{"class":469,"line":3175},[151,68833,1090],{"emptyLinePlaceholder":609},[151,68835,68836,68838,68841,68843,68845,68847,68849,68851,68853],{"class":469,"line":3184},[151,68837,16566],{"class":12347},[151,68839,68840],{"class":473}," clientConnectionLost",[151,68842,12386],{"class":503},[151,68844,15277],{"class":15232},[151,68846,106],{"class":503},[151,68848,68806],{"class":15232},[151,68850,106],{"class":503},[151,68852,68811],{"class":15232},[151,68854,15264],{"class":503},[151,68856,68857,68859,68861,68864],{"class":469,"line":3193},[151,68858,18355],{"class":2226},[151,68860,12386],{"class":503},[151,68862,68863],{"class":481},"\"Connection lost.\"",[151,68865,3640],{"class":503},[151,68867,68868],{"class":469,"line":3720},[151,68869,68829],{"class":503},[151,68871,68872],{"class":469,"line":3729},[151,68873,1090],{"emptyLinePlaceholder":609},[151,68875,68876,68879,68882,68884,68886],{"class":469,"line":3735},[151,68877,68878],{"class":503},"reactor.connectTCP(",[151,68880,68881],{"class":481},"\"localhost\"",[151,68883,106],{"class":503},[151,68885,13103],{"class":477},[151,68887,68888],{"class":503},", EchoFactory())\n",[151,68890,68891],{"class":469,"line":3745},[151,68892,68893],{"class":503},"reactor.run()\n",[11,68895,68896],{},[51,68897,68621],{},[459,68899,68901],{"className":13136,"code":68900,"language":12886,"meta":464,"style":464},"from twisted.internet import protocol, reactor\n\nclass Echo(protocol.Protocol):\n    def dataReceived(self,data):\n        self.transport.write(data)\n\nclass EchoFactory(protocol.Factory):\n    def buildProtocol(self, addr):\n        return Echo()\n\nreactor.listenTCP(8000, EchoFactory())\nreactor.run()\n",[30,68902,68903,68914,68918,68935,68951,68958,68962,68979,68995,69002,69006,69015],{"__ignoreMap":464},[151,68904,68905,68907,68909,68911],{"class":469,"line":470},[151,68906,16853],{"class":1869},[151,68908,68637],{"class":503},[151,68910,16859],{"class":1869},[151,68912,68913],{"class":503}," protocol, reactor\n",[151,68915,68916],{"class":469,"line":488},[151,68917,1090],{"emptyLinePlaceholder":609},[151,68919,68920,68922,68925,68927,68929,68931,68933],{"class":469,"line":500},[151,68921,16519],{"class":12347},[151,68923,68924],{"class":15254}," Echo",[151,68926,12386],{"class":503},[151,68928,68658],{"class":15260},[151,68930,643],{"class":503},[151,68932,68663],{"class":15260},[151,68934,15264],{"class":503},[151,68936,68937,68939,68941,68943,68945,68947,68949],{"class":469,"line":509},[151,68938,16566],{"class":12347},[151,68940,68710],{"class":473},[151,68942,12386],{"class":503},[151,68944,15277],{"class":15232},[151,68946,3634],{"class":503},[151,68948,12355],{"class":15232},[151,68950,15264],{"class":503},[151,68952,68953,68955],{"class":469,"line":517},[151,68954,37901],{"class":15289},[151,68956,68957],{"class":503},".transport.write(data)\n",[151,68959,68960],{"class":469,"line":534},[151,68961,1090],{"emptyLinePlaceholder":609},[151,68963,68964,68966,68968,68970,68972,68974,68977],{"class":469,"line":1413},[151,68965,16519],{"class":12347},[151,68967,68750],{"class":15254},[151,68969,12386],{"class":503},[151,68971,68658],{"class":15260},[151,68973,643],{"class":503},[151,68975,68976],{"class":15260},"Factory",[151,68978,15264],{"class":503},[151,68980,68981,68983,68985,68987,68989,68991,68993],{"class":469,"line":1418},[151,68982,16566],{"class":12347},[151,68984,68768],{"class":473},[151,68986,12386],{"class":503},[151,68988,15277],{"class":15232},[151,68990,106],{"class":503},[151,68992,68777],{"class":15232},[151,68994,15264],{"class":503},[151,68996,68997,68999],{"class":469,"line":2462},[151,68998,16833],{"class":1869},[151,69000,69001],{"class":503}," Echo()\n",[151,69003,69004],{"class":469,"line":2471},[151,69005,1090],{"emptyLinePlaceholder":609},[151,69007,69008,69011,69013],{"class":469,"line":2480},[151,69009,69010],{"class":503},"reactor.listenTCP(",[151,69012,13103],{"class":477},[151,69014,68888],{"class":503},[151,69016,69017],{"class":469,"line":2489},[151,69018,68893],{"class":503},[589,69020,69021],{},"html pre.shiki code .sC2Qs, html code.shiki .sC2Qs{--shiki-default:#D73A49;--shiki-dark:#F97583;--shiki-sepia:#F92672}html pre.shiki code .s7F3e, html code.shiki .s7F3e{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#AE81FF}html pre.shiki code .sMOD_, html code.shiki .sMOD_{--shiki-default:#24292E;--shiki-dark:#E1E4E8;--shiki-sepia:#F8F8F2}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html .sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html.sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html pre.shiki code .sq6CD, html code.shiki .sq6CD{--shiki-default:#D73A49;--shiki-default-font-style:inherit;--shiki-dark:#F97583;--shiki-dark-font-style:inherit;--shiki-sepia:#66D9EF;--shiki-sepia-font-style:italic}html pre.shiki code .sz2Vg, html code.shiki .sz2Vg{--shiki-default:#6F42C1;--shiki-default-text-decoration:inherit;--shiki-dark:#B392F0;--shiki-dark-text-decoration:inherit;--shiki-sepia:#A6E22E;--shiki-sepia-text-decoration:underline}html pre.shiki code .s30JN, html code.shiki .s30JN{--shiki-default:#6F42C1;--shiki-default-font-style:inherit;--shiki-default-text-decoration:inherit;--shiki-dark:#B392F0;--shiki-dark-font-style:inherit;--shiki-dark-text-decoration:inherit;--shiki-sepia:#A6E22E;--shiki-sepia-font-style:italic;--shiki-sepia-text-decoration:underline}html pre.shiki code .srTi1, html code.shiki .srTi1{--shiki-default:#6F42C1;--shiki-dark:#B392F0;--shiki-sepia:#A6E22E}html pre.shiki code .so59x, html code.shiki .so59x{--shiki-default:#24292E;--shiki-default-font-style:inherit;--shiki-dark:#E1E4E8;--shiki-dark-font-style:inherit;--shiki-sepia:#FD971F;--shiki-sepia-font-style:italic}html pre.shiki code .sP7S_, html code.shiki .sP7S_{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#FD971F}html pre.shiki code .sstjo, html code.shiki .sstjo{--shiki-default:#032F62;--shiki-dark:#9ECBFF;--shiki-sepia:#E6DB74}html pre.shiki code .sTrkL, html code.shiki .sTrkL{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#66D9EF}",{"title":464,"searchDepth":488,"depth":488,"links":69023},[69024,69025],{"id":68529,"depth":488,"text":68530},{"id":68547,"depth":488,"text":68548},"2017-12-22",{"layout":48045},"/2017/12/22/getting-started-with-twisted",{"title":68512,"description":464},"2017/12/22/getting-started-with-twisted",[12886,69032],"twisted","JG_5bk_P-i8CdoImJJNqgaeqoJzjDkvczXhIGhTPUac",{"id":69035,"title":69036,"body":69037,"comments":609,"date":69878,"description":464,"draft":602,"extension":605,"external":606,"image":69043,"meta":69879,"navigation":609,"path":69880,"seo":69881,"stem":69882,"tags":69883,"__hash__":69884},"blog/2017/12/09/setting-up-flask-cli-with-docker.md","Setting up a Flask project with Flask CLI and Docker",{"type":8,"value":69038,"toc":69871},[69039,69044,69047,69068,69073,69076,69080,69086,69090,69096,69100,69106,69113,69117,69123,69126,69129,69141,69152,69156,69255,69258,69264,69275,69281,69288,69293,69417,69423,69438,69444,69450,69460,69463,69469,69472,69478,69490,69558,69561,69567,69573,69579,69586,69589,69594,69600,69606,69612,69618,69629,69632,69638,69643,69646,69654,69658,69664,69670,69674,69680,69683,69686,69692,69695,69701,69711,69717,69724,69730,69737,69740,69746,69749,69755,69758,69764,69774,69780,69796,69806,69812,69820,69828,69834,69837,69843,69854,69860,69865,69868],[11,69040,69041],{},[2718,69042],{"alt":20386,"src":69043},"/static/flask-docker.png",[11,69045,69046],{},"⚠️ This article is under contruction ⚠️",[11,69048,69049,69050,69055,69056,18952,69059,69061,69062,69065,69066,643],{},"I've recently been working on an awesome tutorial from ",[20,69051,69054],{"href":69052,"rel":69053},"https://testdriven.io",[24],"testdriven.io"," that covers flask, react and docker. The beginning of the project covers how to setup a basic flask app using ",[30,69057,69058],{},"flask-scripts",[30,69060,69058],{}," is a deprecated tool and the tutorial recommends using ",[30,69063,69064],{},"Flask CLI",". I have fumbled with this the first time I tried to set it up and while I was able to get it working, I couldn't get it working inside of docker. In this article I'll detail the setup of my flask project with ",[30,69067,69064],{},[210,69069,69070],{},[11,69071,69072],{},"One of the nice new features in Flask 0.11 is the built-in integration of the click command line interface. This enables a wide range of new features for the Flask ecosystem and your own applications.",[11,69074,69075],{},"OK. Let's set up a basic flask app:",[736,69077,69079],{"id":69078},"directories","Directories",[459,69081,69084],{"className":69082,"code":69083,"language":997},[995]," $ mkdir test-flask && cd test-flask\n $ mkdir users-service && cd users-service\n $ mkdir project\n\n",[30,69085,69083],{"__ignoreMap":464},[736,69087,69089],{"id":69088},"virtual-environment","Virtual Environment",[459,69091,69094],{"className":69092,"code":69093,"language":997},[995]," $ virtualenv -p python3 env\nRunning virtualenv with interpreter /home/brian/anaconda3/bin/python3\nUsing base prefix '/home/brian/anaconda3'\nNew python executable in /home/brian/Documents/flask/test-flask/users-service/env/bin/python3\nAlso creating executable in /home/brian/Documents/flask/test-flask/users-service/env/bin/python\nInstalling setuptools, pip, wheel...done.\n",[30,69095,69093],{"__ignoreMap":464},[736,69097,69099],{"id":69098},"activate-virtual-environment","Activate Virtual Environment",[459,69101,69104],{"className":69102,"code":69103,"language":997},[995]," $ source env/bin/activate\n(env) $\n",[30,69105,69103],{"__ignoreMap":464},[11,69107,69108,69109,69112],{},"We will come back to the ",[30,69110,69111],{},"activate"," script and add some environment variables to the bottom if it so we have them accessible when we activate the virtual environment.",[736,69114,69116],{"id":69115},"install-flask-in-the-virtual-environment","Install flask in the virtual environment",[459,69118,69121],{"className":69119,"code":69120,"language":997},[995]," $ pip install flask==0.12.2\nCollecting flask==0.12.2\n  Using cached Flask-0.12.2-py2.py3-none-any.whl\nCollecting itsdangerous>=0.21 (from flask==0.12.2)\nCollecting Werkzeug>=0.7 (from flask==0.12.2)\n  Using cached Werkzeug-0.13-py2.py3-none-any.whl\nCollecting click>=2.0 (from flask==0.12.2)\n  Using cached click-6.7-py2.py3-none-any.whl\nCollecting Jinja2>=2.4 (from flask==0.12.2)\n  Using cached Jinja2-2.10-py2.py3-none-any.whl\nCollecting MarkupSafe>=0.23 (from Jinja2>=2.4->flask==0.12.2)\nInstalling collected packages: itsdangerous, Werkzeug, click, MarkupSafe, Jinja2, flask\nSuccessfully installed Jinja2-2.10 MarkupSafe-1.0 Werkzeug-0.13 click-6.7 flask-0.12.2 itsdangerous-0.24\n(env) $\n",[30,69122,69120],{"__ignoreMap":464},[11,69124,69125],{},"At this point we are ready to create our flask app.",[11,69127,69128],{},"Here's a note about the CLI:",[210,69130,69131],{},[11,69132,69133,69134,69136,69137,69140],{},"For the ",[15,69135,40766],{}," script to work, an application needs to be discovered. This is achieved by exporting the ",[30,69138,69139],{},"FLASK_APP"," environment variable. It can be either set to an import path or to a filename of a Python module that contains a Flask application.",[11,69142,69143,69144,69147,69148,69151],{},"Let's add an ",[30,69145,69146],{},"app.py"," file inside the ",[30,69149,69150],{},"project"," folder:",[11,69153,69154],{},[51,69155,69146],{},[459,69157,69159],{"className":13136,"code":69158,"language":12886,"meta":464,"style":464},"from flask import Flask, jsonify\n\napp = Flask(__name__)\n\n@app.route('/users/ping', methods=['GET'])\ndef ping_pong():\n    return jsonify({\n        'message':'pong!',\n        'status':'success'\n        })\n",[30,69160,69161,69172,69176,69188,69192,69214,69223,69229,69241,69251],{"__ignoreMap":464},[151,69162,69163,69165,69167,69169],{"class":469,"line":470},[151,69164,16853],{"class":1869},[151,69166,38677],{"class":503},[151,69168,16859],{"class":1869},[151,69170,69171],{"class":503}," Flask, jsonify\n",[151,69173,69174],{"class":469,"line":488},[151,69175,1090],{"emptyLinePlaceholder":609},[151,69177,69178,69180,69182,69184,69186],{"class":469,"line":500},[151,69179,38486],{"class":503},[151,69181,1876],{"class":1869},[151,69183,38782],{"class":503},[151,69185,38785],{"class":12360},[151,69187,3640],{"class":503},[151,69189,69190],{"class":469,"line":509},[151,69191,1090],{"emptyLinePlaceholder":609},[151,69193,69194,69196,69198,69201,69203,69205,69207,69209,69212],{"class":469,"line":517},[151,69195,38800],{"class":473},[151,69197,12386],{"class":503},[151,69199,69200],{"class":481},"'/users/ping'",[151,69202,106],{"class":503},[151,69204,38810],{"class":15210},[151,69206,1876],{"class":1869},[151,69208,6698],{"class":503},[151,69210,69211],{"class":481},"'GET'",[151,69213,38820],{"class":503},[151,69215,69216,69218,69221],{"class":469,"line":534},[151,69217,16925],{"class":12347},[151,69219,69220],{"class":473}," ping_pong",[151,69222,16931],{"class":503},[151,69224,69225,69227],{"class":469,"line":1413},[151,69226,17496],{"class":1869},[151,69228,39434],{"class":503},[151,69230,69231,69234,69236,69239],{"class":469,"line":1418},[151,69232,69233],{"class":481},"        'message'",[151,69235,208],{"class":503},[151,69237,69238],{"class":481},"'pong!'",[151,69240,9417],{"class":503},[151,69242,69243,69246,69248],{"class":469,"line":2462},[151,69244,69245],{"class":481},"        'status'",[151,69247,208],{"class":503},[151,69249,69250],{"class":481},"'success'\n",[151,69252,69253],{"class":469,"line":2471},[151,69254,39503],{"class":503},[11,69256,69257],{},"And now let's add an environment variable to tell the Flask CLI where our app is located:",[459,69259,69262],{"className":69260,"code":69261,"language":997},[995]," $ export FLASK_APP=/home/brian/Documents/flask/test-flask/users-service/project/app.py\n",[30,69263,69261],{"__ignoreMap":464},[11,69265,69266,69267,69270,69271,69274],{},"OK, now let's try to run ",[30,69268,69269],{},"flask run"," and navigate to ",[30,69272,69273],{},"/users/ping"," and see what happens:",[459,69276,69279],{"className":69277,"code":69278,"language":997},[995]," $ flask run\n * Serving Flask app \"app\"\n * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)\n127.0.0.1 - - [09/Dec/2017 19:20:14] \"GET /users/ping HTTP/1.1\" 200 -\n",[30,69280,69278],{"__ignoreMap":464},[11,69282,69283,69284,69287],{},"Great! We see our ",[30,69285,69286],{},"pong!"," message returned in the browser. Next, let's configure our settings:",[11,69289,69290],{},[51,69291,69292],{},"project/config.py",[459,69294,69296],{"className":13136,"code":69295,"language":12886,"meta":464,"style":464},"class BaseConfig:\n    \"\"\"Base configuration\"\"\"\n    DEBUG = False\n    TESTING = False\nclass DevelopmentConfig(BaseConfig):\n    \"\"\"Development configuration\"\"\"\n    DEBUG = True\nclass TestingConfig(BaseConfig):\n    \"\"\"Testing configuration\"\"\"\n    DEBUG = True\n    TESTING = True\nclass ProductionConfig(BaseConfig):\n    \"\"\"Production configuration\"\"\"\n    DEBUG = False\n",[30,69297,69298,69307,69312,69321,69330,69344,69349,69357,69370,69375,69383,69391,69404,69409],{"__ignoreMap":464},[151,69299,69300,69302,69305],{"class":469,"line":470},[151,69301,16519],{"class":12347},[151,69303,69304],{"class":15254}," BaseConfig",[151,69306,14372],{"class":503},[151,69308,69309],{"class":469,"line":488},[151,69310,69311],{"class":481},"    \"\"\"Base configuration\"\"\"\n",[151,69313,69314,69317,69319],{"class":469,"line":500},[151,69315,69316],{"class":477},"    DEBUG",[151,69318,19865],{"class":1869},[151,69320,66889],{"class":477},[151,69322,69323,69326,69328],{"class":469,"line":509},[151,69324,69325],{"class":477},"    TESTING",[151,69327,19865],{"class":1869},[151,69329,66889],{"class":477},[151,69331,69332,69334,69337,69339,69342],{"class":469,"line":517},[151,69333,16519],{"class":12347},[151,69335,69336],{"class":15254}," DevelopmentConfig",[151,69338,12386],{"class":503},[151,69340,69341],{"class":15260},"BaseConfig",[151,69343,15264],{"class":503},[151,69345,69346],{"class":469,"line":534},[151,69347,69348],{"class":481},"    \"\"\"Development configuration\"\"\"\n",[151,69350,69351,69353,69355],{"class":469,"line":1413},[151,69352,69316],{"class":477},[151,69354,19865],{"class":1869},[151,69356,66876],{"class":477},[151,69358,69359,69361,69364,69366,69368],{"class":469,"line":1418},[151,69360,16519],{"class":12347},[151,69362,69363],{"class":15254}," TestingConfig",[151,69365,12386],{"class":503},[151,69367,69341],{"class":15260},[151,69369,15264],{"class":503},[151,69371,69372],{"class":469,"line":2462},[151,69373,69374],{"class":481},"    \"\"\"Testing configuration\"\"\"\n",[151,69376,69377,69379,69381],{"class":469,"line":2471},[151,69378,69316],{"class":477},[151,69380,19865],{"class":1869},[151,69382,66876],{"class":477},[151,69384,69385,69387,69389],{"class":469,"line":2480},[151,69386,69325],{"class":477},[151,69388,19865],{"class":1869},[151,69390,66876],{"class":477},[151,69392,69393,69395,69398,69400,69402],{"class":469,"line":2489},[151,69394,16519],{"class":12347},[151,69396,69397],{"class":15254}," ProductionConfig",[151,69399,12386],{"class":503},[151,69401,69341],{"class":15260},[151,69403,15264],{"class":503},[151,69405,69406],{"class":469,"line":2497},[151,69407,69408],{"class":481},"    \"\"\"Production configuration\"\"\"\n",[151,69410,69411,69413,69415],{"class":469,"line":3140},[151,69412,69316],{"class":477},[151,69414,19865],{"class":1869},[151,69416,66889],{"class":477},[11,69418,69419,69420,208],{},"And now we can add the following line right below where we define ",[30,69421,69422],{},"app = Flask(__name__)",[459,69424,69426],{"className":13136,"code":69425,"language":12886,"meta":464,"style":464},"app.config.from_object('project.config.DevelopmentConfig')\n",[30,69427,69428],{"__ignoreMap":464},[151,69429,69430,69433,69436],{"class":469,"line":470},[151,69431,69432],{"class":503},"app.config.from_object(",[151,69434,69435],{"class":481},"'project.config.DevelopmentConfig'",[151,69437,3640],{"class":503},[11,69439,69440,69441,69443],{},"When we run ",[30,69442,69269],{},", we get a long error message including:",[459,69445,69448],{"className":69446,"code":69447,"language":997},[995],"Debugged import:\n\n- 'project' not found.\n\nOriginal exception:\n\nImportStringError: import_string() failed for 'project.config'. Possible reasons are:\n\n- missing __init__.py in a package;\n- package or module path not included in sys.path;\n- duplicated package or module name taking precedence in sys.path;\n- missing module, class, function or variable;\n",[30,69449,69447],{"__ignoreMap":464},[11,69451,69452,69453,69456,69457,69459],{},"Let's try to add ",[30,69454,69455],{},"__init__.py"," to our ",[30,69458,69150],{}," folder.",[11,69461,69462],{},"Once we do this, we are able to run the app successfully, but we don't see any special message about Debug mode being on:",[459,69464,69467],{"className":69465,"code":69466,"language":997},[995]," $ flask run\n * Serving Flask app \"project.app\"\n * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)\n",[30,69468,69466],{"__ignoreMap":464},[11,69470,69471],{},"The documentation mentions that we can turn on Debug mode with:",[459,69473,69476],{"className":69474,"code":69475,"language":997},[995],"export FLASK_DEBUG=1\n",[30,69477,69475],{"__ignoreMap":464},[11,69479,69480,69481,22326,69484,69487,69488,208],{},"We can check the debug mode by printing ",[30,69482,69483],{},"app.config",[30,69485,69486],{},"ping_pong()"," function that is returned when we hit ",[30,69489,69273],{},[459,69491,69493],{"className":13136,"code":69492,"language":12886,"meta":464,"style":464},"@app.route('/users/ping', methods=['GET'])\ndef ping_pong():\n    print(app.config)\n    return jsonify({\n        'message':'pong!',\n        'status':'success'\n        })\n",[30,69494,69495,69515,69523,69530,69536,69546,69554],{"__ignoreMap":464},[151,69496,69497,69499,69501,69503,69505,69507,69509,69511,69513],{"class":469,"line":470},[151,69498,38800],{"class":473},[151,69500,12386],{"class":503},[151,69502,69200],{"class":481},[151,69504,106],{"class":503},[151,69506,38810],{"class":15210},[151,69508,1876],{"class":1869},[151,69510,6698],{"class":503},[151,69512,69211],{"class":481},[151,69514,38820],{"class":503},[151,69516,69517,69519,69521],{"class":469,"line":488},[151,69518,16925],{"class":12347},[151,69520,69220],{"class":473},[151,69522,16931],{"class":503},[151,69524,69525,69527],{"class":469,"line":500},[151,69526,24285],{"class":2226},[151,69528,69529],{"class":503},"(app.config)\n",[151,69531,69532,69534],{"class":469,"line":509},[151,69533,17496],{"class":1869},[151,69535,39434],{"class":503},[151,69537,69538,69540,69542,69544],{"class":469,"line":517},[151,69539,69233],{"class":481},[151,69541,208],{"class":503},[151,69543,69238],{"class":481},[151,69545,9417],{"class":503},[151,69547,69548,69550,69552],{"class":469,"line":534},[151,69549,69245],{"class":481},[151,69551,208],{"class":503},[151,69553,69250],{"class":481},[151,69555,69556],{"class":469,"line":1413},[151,69557,39503],{"class":503},[11,69559,69560],{},"Here's what we see in the terminal:",[459,69562,69565],{"className":69563,"code":69564,"language":997},[995],"\u003CConfig {'DEBUG': True, 'TESTING': False, 'PROPAGATE_EXCEPTIONS': None, 'PRESERVE_CONTEXT_ON_EXCEPTION': None, 'SECRET_KEY': None, 'PERMANENT_SESSION_LIFETIME': datetime.timedelta(31), 'USE_X_SENDFILE': False, 'LOGGER_NAME': 'project.app', 'LOGGER_HANDLER_POLICY': 'always', 'SERVER_NAME': None, 'APPLICATION_ROOT': None, 'SESSION_COOKIE_NAME': 'session', 'SESSION_COOKIE_DOMAIN': None, 'SESSION_COOKIE_PATH': None, 'SESSION_COOKIE_HTTPONLY': True, 'SESSION_COOKIE_SECURE': False, 'SESSION_REFRESH_EACH_REQUEST': True, 'MAX_CONTENT_LENGTH': None, 'SEND_FILE_MAX_AGE_DEFAULT': datetime.timedelta(0, 43200), 'TRAP_BAD_REQUEST_ERRORS': False, 'TRAP_HTTP_EXCEPTIONS': False, 'EXPLAIN_TEMPLATE_LOADING': False, 'PREFERRED_URL_SCHEME': 'http', 'JSON_AS_ASCII': True, 'JSON_SORT_KEYS': True, 'JSONIFY_PRETTYPRINT_REGULAR': True, 'JSONIFY_MIMETYPE': 'application/json', 'TEMPLATES_AUTO_RELOAD': None}>\n127.0.0.1 - - [09/Dec/2017 19:42:46] \"GET /users/ping HTTP/1.1\" 200 -\n",[30,69566,69564],{"__ignoreMap":464},[11,69568,69569,69570,208],{},"Just to be sure this is working correctly, let's try another config setting, ",[30,69571,69572],{},"ProductionConfig",[459,69574,69577],{"className":69575,"code":69576,"language":997},[995],"\u003CConfig {'DEBUG': False, 'TESTING': False, 'PROPAGATE_EXCEPTIONS': None, 'PRESERVE_CONTEXT_ON_EXCEPTION': None, 'SECRET_KEY': None, 'PERMANENT_SESSION_LIFETIME': datetime.timedelta(31), 'USE_X_SENDFILE': False, 'LOGGER_NAME': 'project.app', 'LOGGER_HANDLER_POLICY': 'always', 'SERVER_NAME': None, 'APPLICATION_ROOT': None, 'SESSION_COOKIE_NAME': 'session', 'SESSION_COOKIE_DOMAIN': None, 'SESSION_COOKIE_PATH': None, 'SESSION_COOKIE_HTTPONLY': True, 'SESSION_COOKIE_SECURE': False, 'SESSION_REFRESH_EACH_REQUEST': True, 'MAX_CONTENT_LENGTH': None, 'SEND_FILE_MAX_AGE_DEFAULT': datetime.timedelta(0, 43200), 'TRAP_BAD_REQUEST_ERRORS': False, 'TRAP_HTTP_EXCEPTIONS': False, 'EXPLAIN_TEMPLATE_LOADING': False, 'PREFERRED_URL_SCHEME': 'http', 'JSON_AS_ASCII': True, 'JSON_SORT_KEYS': True, 'JSONIFY_PRETTYPRINT_REGULAR': True, 'JSONIFY_MIMETYPE': 'application/json', 'TEMPLATES_AUTO_RELOAD': None}>\n127.0.0.1 - - [09/Dec/2017 19:45:29] \"GET /users/ping HTTP/1.1\" 200 -\n",[30,69578,69576],{"__ignoreMap":464},[11,69580,69581,69582,69585],{},"OK, so far so good! I think that ",[30,69583,69584],{},"FLASK_DEBUG"," may give us some additional information in the terminal.",[11,69587,69588],{},"Let's run:",[459,69590,69592],{"className":69591,"code":69475,"language":997},[995],[30,69593,69475],{"__ignoreMap":464},[11,69595,69596,69597,69599],{},"and run our app in ",[30,69598,69572],{}," mode:",[459,69601,69604],{"className":69602,"code":69603,"language":997},[995]," $ flask run\n * Serving Flask app \"project.app\"\n * Forcing debug mode on\n * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)\n * Restarting with stat\n * Debugger is active!\n * Debugger PIN: 775-946-486\n",[30,69605,69603],{"__ignoreMap":464},[11,69607,69608,69609,69611],{},"and when we hit ",[30,69610,69273],{}," we get:",[459,69613,69616],{"className":69614,"code":69615,"language":997},[995],"\u003CConfig {'DEBUG': True, 'TESTING': False, 'PROPAGATE_EXCEPTIONS': None, 'PRESERVE_CONTEXT_ON_EXCEPTION': None, 'SECRET_KEY': None, 'PERMANENT_SESSION_LIFETIME': datetime.timedelta(31), 'USE_X_SENDFILE': False, 'LOGGER_NAME': 'project.app', 'LOGGER_HANDLER_POLICY': 'always', 'SERVER_NAME': None, 'APPLICATION_ROOT': None, 'SESSION_COOKIE_NAME': 'session', 'SESSION_COOKIE_DOMAIN': None, 'SESSION_COOKIE_PATH': None, 'SESSION_COOKIE_HTTPONLY': True, 'SESSION_COOKIE_SECURE': False, 'SESSION_REFRESH_EACH_REQUEST': True, 'MAX_CONTENT_LENGTH': None, 'SEND_FILE_MAX_AGE_DEFAULT': datetime.timedelta(0, 43200), 'TRAP_BAD_REQUEST_ERRORS': False, 'TRAP_HTTP_EXCEPTIONS': False, 'EXPLAIN_TEMPLATE_LOADING': False, 'PREFERRED_URL_SCHEME': 'http', 'JSON_AS_ASCII': True, 'JSON_SORT_KEYS': True, 'JSONIFY_PRETTYPRINT_REGULAR': True, 'JSONIFY_MIMETYPE': 'application/json', 'TEMPLATES_AUTO_RELOAD': None}>\n127.0.0.1 - - [09/Dec/2017 19:51:17] \"GET /users/ping HTTP/1.1\" 200 -\n",[30,69617,69615],{"__ignoreMap":464},[11,69619,69620,69621,306,69623,69625,69626,643],{},"Here we can see that ",[30,69622,49418],{},[30,69624,36962],{},", which was forced when we set ",[30,69627,69628],{},"FLASK_DEBUG=1",[11,69630,69631],{},"For clarity, let's review the directory structure of our project:",[459,69633,69636],{"className":69634,"code":69635,"language":997},[995]," $ tree project/\nproject/\n├── app.py\n├── config.py\n└── __init__.py\n",[30,69637,69635],{"__ignoreMap":464},[11,69639,69640,69642],{},[30,69641,69455],{}," is just an empty file at this point, but in the testdriven.io tutorial it is the file that contains our app.",[11,69644,69645],{},"OK, we have a very simple flask app that we can control with the flask cli. Let's get ready to dockerize this simple project.",[11,69647,69648,69649,69651,69652,69151],{},"We need a ",[30,69650,38577],{}," file. So far we just have flask. We want to place this file on the same level as our ",[30,69653,69150],{},[11,69655,69656],{},[51,69657,38577],{},[459,69659,69662],{"className":69660,"code":69661,"language":997},[995],"Flask==0.12.1\n",[30,69663,69661],{"__ignoreMap":464},[11,69665,69666,69667,69669],{},"And we can add a ",[30,69668,12546],{}," file as well at the same level:",[11,69671,69672],{},[51,69673,12546],{},[459,69675,69678],{"className":69676,"code":69677,"language":997},[995],"__pycache__\nenv\n",[30,69679,69677],{"__ignoreMap":464},[56,69681,69682],{"id":30129},"Docker",[11,69684,69685],{},"Here are the versions of docker applications I have installed:",[459,69687,69690],{"className":69688,"code":69689,"language":997},[995]," $ docker -v && docker-compose -v && docker-machine -v\nDocker version 17.10.0-ce, build f4ffd2511c\ndocker-compose version 1.17.1, build unknown\ndocker-machine version 0.13.0, build HEAD\n",[30,69691,69689],{"__ignoreMap":464},[11,69693,69694],{},"Currently I don't have any docker machines, images or containers. Here is the status of the docker service:",[459,69696,69699],{"className":69697,"code":69698,"language":997},[995]," $ systemctl status docker\n● docker.service - Docker Application Container Engine\n   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)\n   Active: active (running) since Fri 2017-12-08 19:13:09 EST; 24h ago\n     Docs: https://docs.docker.com\n Main PID: 5486 (dockerd)\n    Tasks: 26 (limit: 4915)\n   CGroup: /system.slice/docker.service\n           ├─5486 /usr/bin/dockerd -g /home/brian/docker -H fd://\n           └─5492 docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --shim docker-containerd-shim --runtime docker-runc\n\nDec 08 19:13:08 archthinkpad dockerd[5486]: time=\"2017-12-08T19:13:08.126834430-05:00\" level=warning msg=\"Your kernel does not support cgroup rt period\"\nDec 08 19:13:08 archthinkpad dockerd[5486]: time=\"2017-12-08T19:13:08.126850093-05:00\" level=warning msg=\"Your kernel does not support cgroup rt runtime\"\nDec 08 19:13:08 archthinkpad dockerd[5486]: time=\"2017-12-08T19:13:08.127216425-05:00\" level=info msg=\"Loading containers: start.\"\nDec 08 19:13:08 archthinkpad dockerd[5486]: time=\"2017-12-08T19:13:08.776371797-05:00\" level=info msg=\"Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address\"\nDec 08 19:13:09 archthinkpad dockerd[5486]: time=\"2017-12-08T19:13:09.013651222-05:00\" level=info msg=\"Loading containers: done.\"\nDec 08 19:13:09 archthinkpad dockerd[5486]: time=\"2017-12-08T19:13:09.041318847-05:00\" level=warning msg=\"Not using native diff for overlay2, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled\"\nDec 08 19:13:09 archthinkpad dockerd[5486]: time=\"2017-12-08T19:13:09.075633462-05:00\" level=info msg=\"Docker daemon\" commit=f4ffd2511c graphdriver(s)=overlay2 version=17.10.0-ce\nDec 08 19:13:09 archthinkpad dockerd[5486]: time=\"2017-12-08T19:13:09.076162900-05:00\" level=info msg=\"Daemon has completed initialization\"\nDec 08 19:13:09 archthinkpad dockerd[5486]: time=\"2017-12-08T19:13:09.082056339-05:00\" level=info msg=\"API listen on /var/run/docker.sock\"\nDec 08 19:13:09 archthinkpad systemd[1]: Started Docker Application Container Engine.\n",[30,69700,69698],{"__ignoreMap":464},[11,69702,69703,69704,69707,69708,33226],{},"OK, docker seems to be working fine. Now we need to create a Docker host and point the docker client at it. What does this mean? From what I understand, we will be running docker containers not on our local machine but in an instance of ",[30,69705,69706],{},"virtualbox"," on our local machine. To do this, we will use the ",[30,69709,69710],{},"docker-machine",[459,69712,69715],{"className":69713,"code":69714,"language":997},[995]," $ docker-machine create -d virtualbox testdriven-dev\nRunning pre-create checks...\nCreating machine...\n(testdriven-dev) Copying /home/brian/.docker/machine/cache/boot2docker.iso to /home/brian/.docker/machine/machines/testdriven-dev/boot2docker.iso...\n(testdriven-dev) Creating VirtualBox VM...\n(testdriven-dev) Creating SSH key...\n(testdriven-dev) Starting the VM...\n(testdriven-dev) Check network to re-create if needed...\n(testdriven-dev) Waiting for an IP...\nWaiting for machine to be running, this may take a few minutes...\nDetecting operating system of created instance...\nWaiting for SSH to be available...\nDetecting the provisioner...\nProvisioning with boot2docker...\nCopying certs to the local machine directory...\nCopying certs to the remote machine...\nSetting Docker configuration on the remote daemon...\n\nThis machine has been allocated an IP address, but Docker Machine could not reach it successfully.\n\nSSH for the machine should still work, but connecting to exposed ports, such as the Docker daemon port (usually \u003Cip>:2376), may not work properly.\n\nYou may need to add the route manually, or use another related workaround.\n\nThis could be due to a VPN, proxy, or host file configuration issue.\n\nYou also might want to clear any VirtualBox host only interfaces you are not using.\nChecking connection to Docker...\nDocker is up and running!\nTo see how to connect your Docker Client to the Docker Engine running on this virtual machine, run: docker-machine env testdriven-dev\n",[30,69716,69714],{"__ignoreMap":464},[11,69718,69719,69720,69723],{},"We see that ",[30,69721,69722],{},"Docker is up and running!",", but notice this line:",[459,69725,69728],{"className":69726,"code":69727,"language":997},[995],"This machine has been allocated an IP address, but Docker Machine could not reach it successfully.\n",[30,69729,69727],{"__ignoreMap":464},[11,69731,69732,69733,69736],{},"This might be a problem. I think that the ",[30,69734,69735],{},"docker-machine env \u003Cmachine-name>"," command fixes this:",[11,69738,69739],{},"When you run this command you get the following:",[459,69741,69744],{"className":69742,"code":69743,"language":997},[995]," $ docker-machine env testdriven-dev\nexport DOCKER_TLS_VERIFY=\"1\"\nexport DOCKER_HOST=\"tcp://192.168.99.100:2376\"\nexport DOCKER_CERT_PATH=\"/home/brian/.docker/machine/machines/testdriven-dev\"\nexport DOCKER_MACHINE_NAME=\"testdriven-dev\"\n# Run this command to configure your shell:\n# eval $(docker-machine env testdriven-dev)\n",[30,69745,69743],{"__ignoreMap":464},[11,69747,69748],{},"The result of this command tells us to run `eval $(docker-machine env testdriven-dev):",[459,69750,69753],{"className":69751,"code":69752,"language":997},[995],"eval \"$(docker-machine env testdriven-dev)\"\n",[30,69754,69752],{"__ignoreMap":464},[11,69756,69757],{},"To clarify, running the above commands puts adds some environment variables:",[459,69759,69762],{"className":69760,"code":69761,"language":997},[995]," $ env | grep DOCKER\nDOCKER_MACHINE_NAME=testdriven-dev\nDOCKER_CERT_PATH=/home/brian/.docker/machine/machines/testdriven-dev\nDOCKER_TLS_VERIFY=1\nDOCKER_HOST=tcp://192.168.99.100:2376\n",[30,69763,69761],{"__ignoreMap":464},[11,69765,69766,69767,69770,69771,69773],{},"Next we need to add a Dockerfile. We will call it ",[30,69768,69769],{},"Dockerfile-dev",". Let's look at ",[30,69772,69769],{}," from the tutorial and see how we may need to modify it for the way we set up our project:",[459,69775,69778],{"className":69776,"code":69777,"language":997},[995],"FROM python:3.6.3\n\n# set working directory\nRUN mkdir -p /usr/src/app\nWORKDIR /usr/src/app\n\n# add requirements\nADD ./requirements.txt /usr/src/app/requirements.txt\n\n# install requirements\nRUN pip install -r requirements.txt\n\n# add app\nADD . /usr/src/app\n\n# run server\nCMD python manage.py runserver -h 0.0.0.0\n",[30,69779,69777],{"__ignoreMap":464},[11,69781,69782,69783,69785,69786,69788,69789,69792,69793,643],{},"We start by defining a base image with the ",[30,69784,51041],{}," line which will give us the correct version of python. We then set folders and the current working directory in docker. Next we install flask add the ",[30,69787,38577],{}," file and install flask with the ",[30,69790,69791],{},"RUN"," line. We then add the directory (on our local machine) with ",[30,69794,69795],{},"ADD . /usr/src/app",[11,69797,69798,69799,69802,69803,69805],{},"This should all be fine up until the last line where we see a ",[30,69800,69801],{},"manage.py"," file. We never created this file since we wish to use the Flask CLI. We could try replicating the process did locally inside our Dockerfile. We need to add the the ",[30,69804,69139],{}," environment variable, and its value should be the script that has just been added to the docker image. Let's try:",[459,69807,69810],{"className":69808,"code":69809,"language":997},[995],"[...]\n\nENV FLASK_APP /usr/src/app/project/app.py\n\nCMD flask run\n",[30,69811,69809],{"__ignoreMap":464},[11,69813,69814,69815,18952,69817,69819],{},"Next we need a script for ",[30,69816,49902],{},[30,69818,49902],{}," is a tool for defining and running multi-container Docker applications. Again, let's look at what was included in the tutorial and then see if we need to make any adjustments:",[11,69821,69822,69825,69826,748],{},[51,69823,69824],{},"docker-compose-dev.yml"," (this file goes in the root directory, one level up from where ",[30,69827,69769],{},[459,69829,69832],{"className":69830,"code":69831,"language":997},[995],"version: '3.3'\n\nservices:\n\n  users-service:\n    container_name: users-service\n      build:\n        context: ./users-service\n        dockerfile: Dockerfile-dev\n      volumes:\n        - './users-service:/usr/src/app'\n      ports:\n        - 5001:5000\n",[30,69833,69831],{"__ignoreMap":464},[11,69835,69836],{},"OK, this looks good! Let's give it a try. We can run the following command:",[459,69838,69841],{"className":69839,"code":69840,"language":997},[995]," $ docker-compose -f docker-compose-dev.yml build\nBuilding users-service\n85b1f47fba49: Pull complete\nba6bd283713a: Pull complete\n817c8cd48a09: Pull complete\n47cc0ed96dc3: Pull complete\n4a36819a59dc: Pull complete\ndb9a0221399f: Pull complete\n7a511a7689b6: Pull complete\n1223757f6914: Pull complete\nDigest: sha256:db9d8546f3ff74e96702abe0a78a0e0454df6ea898de8f124feba81deea416d7\nStatus: Downloaded newer image for python:3.6.3\n ---> 79e1dc9af1c1\nStep 2/8 : RUN mkdir -p /usr/src/app\n ---> Running in 808f6c0497d3\n ---> 8873e8e0d526\nRemoving intermediate container 808f6c0497d3\nStep 3/8 : WORKDIR /usr/src/app\n ---> 76ef8912b4d5\nRemoving intermediate container a3ca419fe9c1\nStep 4/8 : ADD ./requirements.txt /usr/src/app/requirements.txt\n ---> eb513314527a\nStep 5/8 : RUN pip install -r requirements.txt\n ---> Running in 1a708ec3b565\nCollecting Flask==0.12.1 (from -r requirements.txt (line 1))\n  Downloading Flask-0.12.1-py2.py3-none-any.whl (82kB)\nCollecting Jinja2>=2.4 (from Flask==0.12.1->-r requirements.txt (line 1))\n  Downloading Jinja2-2.10-py2.py3-none-any.whl (126kB)\nCollecting click>=2.0 (from Flask==0.12.1->-r requirements.txt (line 1))\n  Downloading click-6.7-py2.py3-none-any.whl (71kB)\nCollecting Werkzeug>=0.7 (from Flask==0.12.1->-r requirements.txt (line 1))\n  Downloading Werkzeug-0.13-py2.py3-none-any.whl (311kB)\nCollecting itsdangerous>=0.21 (from Flask==0.12.1->-r requirements.txt (line 1))\n  Downloading itsdangerous-0.24.tar.gz (46kB)\nCollecting MarkupSafe>=0.23 (from Jinja2>=2.4->Flask==0.12.1->-r requirements.txt (line 1))\n  Downloading MarkupSafe-1.0.tar.gz\nBuilding wheels for collected packages: itsdangerous, MarkupSafe\n  Running setup.py bdist_wheel for itsdangerous: started\n  Running setup.py bdist_wheel for itsdangerous: finished with status 'done'\n  Stored in directory: /root/.cache/pip/wheels/fc/a8/66/24d655233c757e178d45dea2de22a04c6d92766abfb741129a\n  Running setup.py bdist_wheel for MarkupSafe: started\n  Running setup.py bdist_wheel for MarkupSafe: finished with status 'done'\n  Stored in directory: /root/.cache/pip/wheels/88/a7/30/e39a54a87bcbe25308fa3ca64e8ddc75d9b3e5afa21ee32d57\nSuccessfully built itsdangerous MarkupSafe\nInstalling collected packages: MarkupSafe, Jinja2, click, Werkzeug, itsdangerous, Flask\nSuccessfully installed Flask-0.12.1 Jinja2-2.10 MarkupSafe-1.0 Werkzeug-0.13 click-6.7 itsdangerous-0.24\n ---> d828a0518114\nRemoving intermediate container 1a708ec3b565\nStep 6/8 : ADD . /usr/src/app\n ---> 8e0efae73a47\nStep 7/8 : ENV FLASK_APP /usr/src/app/project/app.py\n ---> Running in 959581952d05\n ---> 20aaec61b615\nRemoving intermediate container 959581952d05\nStep 8/8 : CMD flask run\n ---> Running in 4f2fc701ba14\n ---> 1d5b59f3cef2\nRemoving intermediate container 4f2fc701ba14\n\nSuccessfully built 1d5b59f3cef2\nSuccessfully tagged testflask_users-service:latest\n\n",[30,69842,69840],{"__ignoreMap":464},[11,69844,69845,69846,69849,69850,69853],{},"That seemed to work. Next the tutorial says to run ",[30,69847,69848],{},"docker-compose -f docker-compose-dev.yml up -d",". I'll run this without the ",[30,69851,69852],{},"-d"," flag so we can see if there are any errors. Moment of truth!",[459,69855,69858],{"className":69856,"code":69857,"language":997},[995]," $ docker-compose -f docker-compose-dev.yml up\nCreating network \"testflask_default\" with the default driver\nCreating users-service ...\nCreating users-service ... done\nAttaching to users-service\nusers-service    | Usage: flask run [OPTIONS]\nusers-service    |\nusers-service    | Error: The file/path provided (/usr/src/app/project/app.py) does not appear to exist.  Please verify the path is correct.  If app is not on PYTHONPATH, ensure the extension is .py\nusers-service exited with code 2\n",[30,69859,69857],{"__ignoreMap":464},[11,69861,69862,69863,10552],{},"OK, we have an error that seems to have come from our ",[30,69864,69269],{},[11,69866,69867],{},"To bo continued...",[589,69869,69870],{},"html pre.shiki code .sC2Qs, html code.shiki .sC2Qs{--shiki-default:#D73A49;--shiki-dark:#F97583;--shiki-sepia:#F92672}html pre.shiki code .sMOD_, html code.shiki .sMOD_{--shiki-default:#24292E;--shiki-dark:#E1E4E8;--shiki-sepia:#F8F8F2}html pre.shiki code .sXSQT, html code.shiki .sXSQT{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#F8F8F2}html pre.shiki code .srTi1, html code.shiki .srTi1{--shiki-default:#6F42C1;--shiki-dark:#B392F0;--shiki-sepia:#A6E22E}html pre.shiki code .sstjo, html code.shiki .sstjo{--shiki-default:#032F62;--shiki-dark:#9ECBFF;--shiki-sepia:#E6DB74}html pre.shiki code .sTHNf, html code.shiki .sTHNf{--shiki-default:#E36209;--shiki-default-font-style:inherit;--shiki-dark:#FFAB70;--shiki-dark-font-style:inherit;--shiki-sepia:#FD971F;--shiki-sepia-font-style:italic}html pre.shiki code .sq6CD, html code.shiki .sq6CD{--shiki-default:#D73A49;--shiki-default-font-style:inherit;--shiki-dark:#F97583;--shiki-dark-font-style:inherit;--shiki-sepia:#66D9EF;--shiki-sepia-font-style:italic}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html .sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html.sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html pre.shiki code .sz2Vg, html code.shiki .sz2Vg{--shiki-default:#6F42C1;--shiki-default-text-decoration:inherit;--shiki-dark:#B392F0;--shiki-dark-text-decoration:inherit;--shiki-sepia:#A6E22E;--shiki-sepia-text-decoration:underline}html pre.shiki code .s7F3e, html code.shiki .s7F3e{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#AE81FF}html pre.shiki code .s30JN, html code.shiki .s30JN{--shiki-default:#6F42C1;--shiki-default-font-style:inherit;--shiki-default-text-decoration:inherit;--shiki-dark:#B392F0;--shiki-dark-font-style:inherit;--shiki-dark-text-decoration:inherit;--shiki-sepia:#A6E22E;--shiki-sepia-font-style:italic;--shiki-sepia-text-decoration:underline}html pre.shiki code .sTrkL, html code.shiki .sTrkL{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#66D9EF}",{"title":464,"searchDepth":488,"depth":488,"links":69872},[69873,69874,69875,69876,69877],{"id":69078,"depth":500,"text":69079},{"id":69088,"depth":500,"text":69089},{"id":69098,"depth":500,"text":69099},{"id":69115,"depth":500,"text":69116},{"id":30129,"depth":488,"text":69682},"2017-12-09",{"layout":48045},"/2017/12/09/setting-up-flask-cli-with-docker",{"title":69036,"description":464},"2017/12/09/setting-up-flask-cli-with-docker",[40766,30129],"B5J-wjk1V6-ZJKckKfrE9eaGegSEPcBFXGn4teqR9YI",{"id":69886,"title":69887,"body":69888,"comments":609,"date":70661,"description":464,"draft":602,"extension":605,"external":606,"image":69894,"meta":70662,"navigation":609,"path":70663,"seo":70664,"stem":70665,"tags":70666,"__hash__":70668},"blog/2017/12/03/the-twelve-factor-app-and-my-experience-developing-web-apps.md","Reflecting on my web-app development process after reading The Twelve-Factor App",{"type":8,"value":69889,"toc":70647},[69890,69895,69907,69911,69916,69934,69940,70006,70010,70015,70022,70028,70031,70041,70057,70060,70065,70078,70082,70087,70090,70096,70103,70109,70113,70118,70121,70126,70135,70234,70247,70253,70269,70272,70369,70373,70378,70381,70387,70391,70396,70399,70405,70408,70421,70426,70433,70493,70497,70502,70505,70511,70516,70519,70525,70529,70534,70537,70543,70546,70551,70558,70562,70567,70572,70584,70589,70593,70598,70601,70604,70608,70613,70618,70624,70627,70631,70636,70641,70644],[11,69891,69892],{},[2718,69893],{"alt":20386,"src":69894},"/static/the-12-factor-app.png",[11,69896,69897,69898,69901,69902,69906],{},"I recently had a conversation with a developer on the topic of bridging application development and production. From this conversation I was recommneded to have a look at ",[51,69899,69900],{},"The Twelve-Factor App",", a high level guide for building modern, production-ready web applications. In this article I thought it would be interesting to go through each of the twelve sections and reflect on my current development process and how it follows and/or deviates from these factors. I also want to talk about the new technologies and techniques I have been learning from ",[20,69903,69054],{"href":69904,"rel":69905},"https://testdriven.io/",[24]," and how I hope they can benefit me on the next stage of my learning.",[56,69908,69910],{"id":69909},"i-codebase","I. Codebase",[11,69912,69913],{},[15,69914,69915],{},"One codebase tracked in revision control, many deploys",[11,69917,69918,69919,69924,69925,12445,69928,12445,69930,69933],{},"I have been using git to track my projects since starting ",[20,69920,69923],{"href":69921,"rel":69922},"https://www.obeythetestinggoat.com/",[24],"Obey the Testing Goat!"," and have been gradually exploring many of the different features beyond a linear ",[30,69926,69927],{},"add",[30,69929,34092],{},[30,69931,69932],{},"push"," loop. Using Visual Studio Code makes resolving merge conflicts very easy. On this blog I have accepted at least one pull requests from a helpful readers to correct outdated information.",[11,69935,69936,69937,69939],{},"The pattern I have been following for Django uses ",[30,69938,12546],{}," to keep local application settings out of the production codebase using the following logic:",[459,69941,69943],{"className":13136,"code":69942,"language":12886,"meta":464,"style":464},"from .base import *\n\nfrom .production import *\n\ntry:\n    from .local import *\nexcept:\n    pass\n",[30,69944,69945,69957,69961,69972,69976,69983,69994,70001],{"__ignoreMap":464},[151,69946,69947,69949,69952,69954],{"class":469,"line":470},[151,69948,16853],{"class":1869},[151,69950,69951],{"class":503}," .base ",[151,69953,16859],{"class":1869},[151,69955,69956],{"class":1869}," *\n",[151,69958,69959],{"class":469,"line":488},[151,69960,1090],{"emptyLinePlaceholder":609},[151,69962,69963,69965,69968,69970],{"class":469,"line":500},[151,69964,16853],{"class":1869},[151,69966,69967],{"class":503}," .production ",[151,69969,16859],{"class":1869},[151,69971,69956],{"class":1869},[151,69973,69974],{"class":469,"line":509},[151,69975,1090],{"emptyLinePlaceholder":609},[151,69977,69978,69981],{"class":469,"line":517},[151,69979,69980],{"class":1869},"try",[151,69982,14372],{"class":503},[151,69984,69985,69987,69990,69992],{"class":469,"line":534},[151,69986,40344],{"class":1869},[151,69988,69989],{"class":503}," .local ",[151,69991,16859],{"class":1869},[151,69993,69956],{"class":1869},[151,69995,69996,69999],{"class":469,"line":1413},[151,69997,69998],{"class":1869},"except",[151,70000,14372],{"class":503},[151,70002,70003],{"class":469,"line":1418},[151,70004,70005],{"class":1869},"    pass\n",[56,70007,70009],{"id":70008},"ii-dependencies","II. Dependencies",[11,70011,70012],{},[15,70013,70014],{},"Explicitly declare and isolate dependencies",[11,70016,70017,70018,70021],{},"Using python makes this easy and I have had success running ",[30,70019,70020],{},"pip install -r requirements.txt",", or:",[459,70023,70026],{"className":70024,"code":70025,"language":997},[995],"ADD ./requirements.txt /usr/src/app/requirements.txt\nRUN pip install -r requirements.txt\n",[30,70027,70025],{"__ignoreMap":464},[11,70029,70030],{},"when running Docker.",[11,70032,70033,70034,70037,70038,70040],{},"I have ran into minor dependency issues in experimenting with my personal website. I shouldn't have been doing it this way, but I made a virtual environment with ",[30,70035,70036],{},"conda create"," and included all of the packages in the environment in my production ",[30,70039,38577],{},". The Heroku build process gave me a tricky error that I was able to trace to a dependency that was failing to install on the Heroku instance.",[11,70042,70043,70044,106,70047,187,70050,70053,70054,70056],{},"With Python 3 there are a few different options for creating virtual environments: ",[30,70045,70046],{},"virtualenv",[30,70048,70049],{},"venv",[30,70051,70052],{},"conda"," are three that I have used, and ",[30,70055,70046],{}," is a reliable tool for building webapps that I have seen used widely.",[11,70058,70059],{},"Another interesting tip from this section relates to common system tools:",[210,70061,70062],{},[11,70063,70064],{},"Twelve-factor apps also do not rely on the implicit existence of any system tools. Examples include shelling out to ImageMagick or curl.",[11,70066,70067,70068,70071,70072,70074,70075,643],{},"This is something that I have been grappling with in Docker. Tools like ",[30,70069,70070],{},"wget"," aren't part of \"base images\" and need to be intalled in the ",[30,70073,49743],{}," or in scripts called from ",[30,70076,70077],{},"docker-compose.yml",[56,70079,70081],{"id":70080},"iii-config","III. Config",[11,70083,70084],{},[15,70085,70086],{},"Store config in the environment",[11,70088,70089],{},"Heroku's command line utilities make setting production environments very easy. I think I can improve the way I organize environment variables locally since I often have many different projects it would be easy for projects to accidentally share the same variable name. One idea I have thought about would be to have an untracked bash script that sets environment variables that I run when starting the development environment, something like:",[459,70091,70094],{"className":70092,"code":70093,"language":997},[995],"export SECRET_KEY=\"my_secret_key\"\nexport DB_URL=\"postgres://my_db_url\"\n",[30,70095,70093],{"__ignoreMap":464},[11,70097,70098,70099,70102],{},"Docker wins points again on this factor because environemnt variables can simple be defined in a development and production ",[30,70100,70101],{},"docker-compose-*.yml"," files:",[459,70104,70107],{"className":70105,"code":70106,"language":997},[995],"    environment:\n      - APP_SETTINGS=project.config.DevelopmentConfig\n      - DATABASE_URL=postgres://postgres:postgres@users-db:5432/users_dev\n      - DATABASE_TEST_URL=postgres://postgres:postgres@users-db:5432/users_test\n",[30,70108,70106],{"__ignoreMap":464},[56,70110,70112],{"id":70111},"iv-backing-services","IV. Backing services",[11,70114,70115],{},[15,70116,70117],{},"Treat backing services as attached resources",[11,70119,70120],{},"The key takeaway for backing services (databases, message/queuing systems, etc.) is:",[210,70122,70123],{},[11,70124,70125],{},"The code for a twelve-factor app makes no distinction between local and third party services.",[11,70127,70128,70129,70134],{},"This factor made me think of an interesting syntax that I saw for the first time when learning the microservices architecture with docker in ",[20,70130,70133],{"href":70131,"rel":70132},"https://github.com/jakewright/tutorials/tree/master/docker/02-docker-compose",[24],"this Docker example"," that uses two Flask apps to provide 1) front-end templates and 2) API backend. When the front end calls the API backend, it does so by referencing the service name in the URL. Here is an example with PHP, but it would work similarly in any other framework:",[459,70136,70140],{"className":70137,"code":70138,"language":70139,"meta":464,"style":464},"language-php shiki shiki-themes github-light github-dark monokai","\u003C?php\n    $json = file_get_contents('http://product-service/');\n    $obj = json_decode($json);\n    $products = $obj->products;\n    foreach ($products as $product) {\n        echo \"\u003Cli>$product\u003C/li>\";\n    }\n?>\n","php",[30,70141,70142,70150,70167,70180,70196,70209,70225,70229],{"__ignoreMap":464},[151,70143,70144,70147],{"class":469,"line":470},[151,70145,70146],{"class":1869},"\u003C?",[151,70148,70149],{"class":477},"php\n",[151,70151,70152,70155,70157,70160,70162,70165],{"class":469,"line":488},[151,70153,70154],{"class":503},"    $json ",[151,70156,1876],{"class":1869},[151,70158,70159],{"class":2226}," file_get_contents",[151,70161,12386],{"class":503},[151,70163,70164],{"class":481},"'http://product-service/'",[151,70166,20129],{"class":503},[151,70168,70169,70172,70174,70177],{"class":469,"line":500},[151,70170,70171],{"class":503},"    $obj ",[151,70173,1876],{"class":1869},[151,70175,70176],{"class":2226}," json_decode",[151,70178,70179],{"class":503},"($json);\n",[151,70181,70182,70185,70187,70190,70193],{"class":469,"line":509},[151,70183,70184],{"class":503},"    $products ",[151,70186,1876],{"class":1869},[151,70188,70189],{"class":503}," $obj",[151,70191,70192],{"class":1869},"->",[151,70194,70195],{"class":503},"products;\n",[151,70197,70198,70201,70204,70206],{"class":469,"line":517},[151,70199,70200],{"class":1869},"    foreach",[151,70202,70203],{"class":503}," ($products ",[151,70205,16998],{"class":1869},[151,70207,70208],{"class":503}," $product) {\n",[151,70210,70211,70214,70217,70220,70223],{"class":469,"line":534},[151,70212,70213],{"class":2226},"        echo",[151,70215,70216],{"class":481}," \"\u003Cli>",[151,70218,70219],{"class":503},"$product",[151,70221,70222],{"class":481},"\u003C/li>\"",[151,70224,20086],{"class":503},[151,70226,70227],{"class":469,"line":1413},[151,70228,9461],{"class":503},[151,70230,70231],{"class":469,"line":1418},[151,70232,70233],{"class":1869},"?>\n",[11,70235,70236,70237,70240,70241,70244,70245,208],{},"This front-end code hits the backend API endpoint ",[30,70238,70239],{},"http://product-service/",", where ",[30,70242,70243],{},"product-service"," is the name of a service included in ",[30,70246,70077],{},[459,70248,70251],{"className":70249,"code":70250,"language":997},[995],"version: '3'\n\nservices:\n  product-service:\n    build: ./product\n    volumes:\n      - ./product:/usr/src/app\n    ports:\n      - 5001:80\n\n  website:\n    image: php:apache\n    volumes:\n      - ./website:/var/www/html\n    ports:\n      - 5000:80\n    depends_on:\n      - product-service\n",[30,70252,70250],{"__ignoreMap":464},[11,70254,70255,70256,187,70258,70260,70261,70264,70265,70268],{},"This is docker-compose file creates a network that includes both ",[30,70257,63215],{},[30,70259,70243],{}," that can be accessed by simply creating a URL with the name of the service in the domain. Coming back to the fourth factor, ",[15,70262,70263],{},"The code for a twelve-factor app makes no distinction between local and third party services",", multiple docker containers can be thought of as ",[51,70266,70267],{},"separate services"," even though they may be running on the same virtual environment in either development or production, and the unique domain cooresponding to the service name seems to reinforce this concept.",[11,70270,70271],{},"This URL could also be referenced with environment variables:",[459,70273,70275],{"className":19459,"code":70274,"language":19461,"meta":464,"style":464},"getUsers() {\n  axios.get(`${process.env.REACT_APP_USERS_SERVICE_URL}/users`)\n  .then((res) => { console.log(res); })\n  .catch((err) => { console.log(err); })\n}\n",[30,70276,70277,70285,70318,70343,70365],{"__ignoreMap":464},[151,70278,70279,70282],{"class":469,"line":470},[151,70280,70281],{"class":473},"getUsers",[151,70283,70284],{"class":503},"() {\n",[151,70286,70287,70290,70293,70295,70297,70299,70302,70304,70306,70308,70311,70313,70316],{"class":469,"line":488},[151,70288,70289],{"class":503},"  axios.",[151,70291,70292],{"class":473},"get",[151,70294,12386],{"class":503},[151,70296,2798],{"class":481},[151,70298,19871],{"class":19870},[151,70300,70301],{"class":503},"process",[151,70303,643],{"class":4828},[151,70305,64084],{"class":503},[151,70307,643],{"class":4828},[151,70309,70310],{"class":12360},"REACT_APP_USERS_SERVICE_URL",[151,70312,2001],{"class":19870},[151,70314,70315],{"class":481},"/users`",[151,70317,3640],{"class":503},[151,70319,70320,70323,70325,70327,70330,70332,70334,70337,70340],{"class":469,"line":500},[151,70321,70322],{"class":503},"  .",[151,70324,34208],{"class":473},[151,70326,34211],{"class":503},[151,70328,70329],{"class":15210},"res",[151,70331,16995],{"class":503},[151,70333,17166],{"class":12347},[151,70335,70336],{"class":503}," { console.",[151,70338,70339],{"class":473},"log",[151,70341,70342],{"class":503},"(res); })\n",[151,70344,70345,70347,70349,70351,70354,70356,70358,70360,70362],{"class":469,"line":509},[151,70346,70322],{"class":503},[151,70348,34337],{"class":473},[151,70350,34211],{"class":503},[151,70352,70353],{"class":15210},"err",[151,70355,16995],{"class":503},[151,70357,17166],{"class":12347},[151,70359,70336],{"class":503},[151,70361,70339],{"class":473},[151,70363,70364],{"class":503},"(err); })\n",[151,70366,70367],{"class":469,"line":517},[151,70368,6274],{"class":503},[56,70370,70372],{"id":70371},"v-build-release-run","V. Build, release, run",[11,70374,70375],{},[15,70376,70377],{},"Strictly separate build and run stages",[11,70379,70380],{},"This is an important stage in getting from development to production. It is the handoff from a local repo to a live, running web application. Here's the process live in action:",[459,70382,70385],{"className":70383,"code":70384,"language":997},[995]," $ git push heroku master\nCounting objects: 3, done.\nDelta compression using up to 4 threads.\nCompressing objects: 100% (3/3), done.\nWriting objects: 100% (3/3), 303 bytes | 303.00 KiB/s, done.\nTotal 3 (delta 2), reused 0 (delta 0)\nremote: Compressing source files... done.\nremote: Building source:\nremote:\nremote: -----> Python app detected\nremote: -----> Installing requirements with pip\nremote:\nremote: -----> Discovering process types\nremote:        Procfile declares types -> web\nremote:\nremote: -----> Compressing...\nremote:        Done: 193.7M\nremote: -----> Launching...\nremote:        Released v410\nremote:        https://briancaffey.herokuapp.com/ deployed to Heroku\nremote:\nremote: Verifying deploy... done.\nTo https://git.heroku.com/briancaffey.git\n   cd6ce76..d8981a3  master -> master\n(briancaffey) brian@archthinkpad ~/Documents/github/briancaffey/src\n",[30,70386,70384],{"__ignoreMap":464},[56,70388,70390],{"id":70389},"vi-processes","VI. Processes",[11,70392,70393],{},[15,70394,70395],{},"Execute the app as one or more stateless processes",[11,70397,70398],{},"This is handled in Heroku by the Procfile. For simple Django apps on Heroku this is usually always the same one line:",[459,70400,70403],{"className":70401,"code":70402,"language":997},[995],"web: gunicorn projectname.wsgi --log-file -\n",[30,70404,70402],{"__ignoreMap":464},[11,70406,70407],{},"Here's what the Django project says about using gunicorn:",[210,70409,70410,70413,70416],{},[11,70411,70412],{},"When Gunicorn is installed, a gunicorn command is available which starts the Gunicorn server process. At its simplest, gunicorn just needs to be called with the location of a module containing a WSGI application object named application.",[11,70414,70415],{},"So for a typical Django project, invoking gunicorn would look like:",[11,70417,70418],{},[30,70419,70420],{},"gunicorn myproject.wsgi",[210,70422,70423],{},[11,70424,70425],{},"This will start one process running one thread listening on 127.0.0.1:8000. It requires that your project be on the Python path; the simplest way to ensure that is to run this command from the same directory as your manage.py file.",[11,70427,70428,70429,70432],{},"In a Django project, the ",[30,70430,70431],{},"wsgi.py"," file in the main folder of the project root directory has the following contents:",[459,70434,70436],{"className":13136,"code":70435,"language":12886,"meta":464,"style":464},"import os\n\nfrom django.core.wsgi import get_wsgi_application\n\nos.environ.setdefault(\"DJANGO_SETTINGS_MODULE\", \"brianblog.settings\")\n\napplication = get_wsgi_application()\n",[30,70437,70438,70444,70448,70460,70464,70479,70483],{"__ignoreMap":464},[151,70439,70440,70442],{"class":469,"line":470},[151,70441,16859],{"class":1869},[151,70443,24070],{"class":503},[151,70445,70446],{"class":469,"line":488},[151,70447,1090],{"emptyLinePlaceholder":609},[151,70449,70450,70452,70455,70457],{"class":469,"line":500},[151,70451,16853],{"class":1869},[151,70453,70454],{"class":503}," django.core.wsgi ",[151,70456,16859],{"class":1869},[151,70458,70459],{"class":503}," get_wsgi_application\n",[151,70461,70462],{"class":469,"line":509},[151,70463,1090],{"emptyLinePlaceholder":609},[151,70465,70466,70469,70472,70474,70477],{"class":469,"line":517},[151,70467,70468],{"class":503},"os.environ.setdefault(",[151,70470,70471],{"class":481},"\"DJANGO_SETTINGS_MODULE\"",[151,70473,106],{"class":503},[151,70475,70476],{"class":481},"\"brianblog.settings\"",[151,70478,3640],{"class":503},[151,70480,70481],{"class":469,"line":534},[151,70482,1090],{"emptyLinePlaceholder":609},[151,70484,70485,70488,70490],{"class":469,"line":1413},[151,70486,70487],{"class":503},"application ",[151,70489,1876],{"class":1869},[151,70491,70492],{"class":503}," get_wsgi_application()\n",[56,70494,70496],{"id":70495},"vi-port-binding","VI. Port binding",[11,70498,70499],{},[15,70500,70501],{},"Export services via port binding",[11,70503,70504],{},"This is an area that I'm trying to learn more about. I feel like I have a pretty good grasp of what is going on regarding port binding in the microservice architecture with docker I have seen.",[11,70506,70507,70508,208],{},"From the docker-compose docs, the \"short syntax\" for mapping ports between hosts and containers is ",[30,70509,70510],{},"HOST:CONTAINER",[210,70512,70513],{},[11,70514,70515],{},"Either specify both ports (HOST:CONTAINER), or just the container port (a random host port will be chosen).",[11,70517,70518],{},"The following are examples of how this could work:",[459,70520,70523],{"className":70521,"code":70522,"language":997},[995],"ports:\n - \"3000\"\n - \"3000-3005\"\n - \"8000:8000\"\n - \"9090-9091:8080-8081\"\n - \"49100:22\"\n - \"127.0.0.1:8001:8001\"\n - \"127.0.0.1:5000-5010:5000-5010\"\n - \"6060:6060/udp\"\n",[30,70524,70522],{"__ignoreMap":464},[56,70526,70528],{"id":70527},"viii-concurrency","VIII. Concurrency",[11,70530,70531],{},[15,70532,70533],{},"Scale out via the process model",[11,70535,70536],{},"This is an important area, but it is something I haven't had to be aware of since the apps I have developed don't require scaling processes. I belive that Heroku makes this fairly simple by allowing you to increase the number of web or worker processes through the CLI:",[459,70538,70541],{"className":70539,"code":70540,"language":997},[995],"heroku ps:scale web=1 worker=5\n",[30,70542,70540],{"__ignoreMap":464},[11,70544,70545],{},"I haven't covered Part 5 of testdriven.io yet, but it has a section on Elastic Load Balancing with EC2 which should cover this area.",[210,70547,70548],{},[11,70549,70550],{},"Twelve-factor app processes should never daemonize or write PID files. Instead, rely on the operating system’s process manager to manage output streams, respond to crashed processes, and handle user-initiated restarts and shutdowns.",[11,70552,70553,70554,70557],{},"I have been learning more about ",[30,70555,70556],{},"systemd"," and customizing",[56,70559,70561],{"id":70560},"ix-disposability","IX. Disposability",[11,70563,70564],{},[15,70565,70566],{},"Maximize robustness with fast startup and graceful shutdown",[210,70568,70569],{},[11,70570,70571],{},"The twelve-factor app’s processes are disposable, meaning they can be started or stopped at a moment’s notice. This facilitates fast elastic scaling, rapid deployment of code or config changes, and robustness of production deploys.",[11,70573,70574,70575,70580,70581,33226],{},"I have used some Heroku tools to start and stop web workers, and docker commands make this factor fairly easy to do correctly. Here's an excerpt from ",[20,70576,70579],{"href":70577,"rel":70578},"https://www.ctl.io/developers/blog/post/gracefully-stopping-docker-containers/",[24],"Century Link"," about the ",[30,70582,70583],{},"docker stop",[210,70585,70586],{},[11,70587,70588],{},"The docker stop command attempts to stop a running container first by sending a SIGTERM signal to the root process (PID 1) in the container. If the process hasn't exited within the timeout period a SIGKILL signal will be sent.",[56,70590,70592],{"id":70591},"x-devprod-parity","X. Dev/prod parity",[210,70594,70595],{},[11,70596,70597],{},"Keep development, staging, and production as similar as possible",[11,70599,70600],{},"This is exactly why I'm so interested in using Docker.",[11,70602,70603],{},"In one of my personal projects I did with Heroku I was relying on a feature of Postgres that is not available in sqlite3, the default database that comes with Django. This produced friction that I wouldn't have had to deal with if I was using Docker. I could have set up a local postgres server, but it would have been much easier to run a docker container that ran the server.",[56,70605,70607],{"id":70606},"xi-logs","XI. Logs",[11,70609,70610],{},[15,70611,70612],{},"Treat logs as event streams",[210,70614,70615],{},[11,70616,70617],{},"A twelve-factor app never concerns itself with routing or storage of its output stream. It should not attempt to write to or manage logfiles. Instead, each running process writes its event stream, unbuffered, to stdout. During local development, the developer will view this stream in the foreground of their terminal to observe the app’s behavior.",[11,70619,50000,70620,70623],{},[30,70621,70622],{},"heroku log"," has been helpful in debugging deployment issues.",[11,70625,70626],{},"Docker also produces helpful logs for all the containers currently running.",[56,70628,70630],{"id":70629},"xii-admin-processes","XII. Admin processes",[11,70632,70633],{},[15,70634,70635],{},"Run admin/management tasks as one-off processes",[210,70637,70638],{},[11,70639,70640],{},"One-off admin processes should be run in an identical environment as the regular long-running processes of the app. They run against a release, using the same codebase and config as any process run against that release. Admin code must ship with application code to avoid synchronization issues.",[11,70642,70643],{},"This seems to be true about the way I run admin processes on my Django apps.",[589,70645,70646],{},"html pre.shiki code .sC2Qs, html code.shiki .sC2Qs{--shiki-default:#D73A49;--shiki-dark:#F97583;--shiki-sepia:#F92672}html pre.shiki code .sMOD_, html code.shiki .sMOD_{--shiki-default:#24292E;--shiki-dark:#E1E4E8;--shiki-sepia:#F8F8F2}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html .sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html.sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html pre.shiki code .s7F3e, html code.shiki .s7F3e{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#AE81FF}html pre.shiki code .sTrkL, html code.shiki .sTrkL{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#66D9EF}html pre.shiki code .sstjo, html code.shiki .sstjo{--shiki-default:#032F62;--shiki-dark:#9ECBFF;--shiki-sepia:#E6DB74}html pre.shiki code .srTi1, html code.shiki .srTi1{--shiki-default:#6F42C1;--shiki-dark:#B392F0;--shiki-sepia:#A6E22E}html pre.shiki code .s1EfO, html code.shiki .s1EfO{--shiki-default:#032F62;--shiki-dark:#9ECBFF;--shiki-sepia:#F92672}html pre.shiki code .sinWB, html code.shiki .sinWB{--shiki-default:#032F62;--shiki-dark:#9ECBFF;--shiki-sepia:#F8F8F2}html pre.shiki code .sXSQT, html code.shiki .sXSQT{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#F8F8F2}html pre.shiki code .sTHNf, html code.shiki .sTHNf{--shiki-default:#E36209;--shiki-default-font-style:inherit;--shiki-dark:#FFAB70;--shiki-dark-font-style:inherit;--shiki-sepia:#FD971F;--shiki-sepia-font-style:italic}html pre.shiki code .sq6CD, html code.shiki .sq6CD{--shiki-default:#D73A49;--shiki-default-font-style:inherit;--shiki-dark:#F97583;--shiki-dark-font-style:inherit;--shiki-sepia:#66D9EF;--shiki-sepia-font-style:italic}",{"title":464,"searchDepth":488,"depth":488,"links":70648},[70649,70650,70651,70652,70653,70654,70655,70656,70657,70658,70659,70660],{"id":69909,"depth":488,"text":69910},{"id":70008,"depth":488,"text":70009},{"id":70080,"depth":488,"text":70081},{"id":70111,"depth":488,"text":70112},{"id":70371,"depth":488,"text":70372},{"id":70389,"depth":488,"text":70390},{"id":70495,"depth":488,"text":70496},{"id":70527,"depth":488,"text":70528},{"id":70560,"depth":488,"text":70561},{"id":70591,"depth":488,"text":70592},{"id":70606,"depth":488,"text":70607},{"id":70629,"depth":488,"text":70630},"2017-12-03",{"layout":48045},"/2017/12/03/the-twelve-factor-app-and-my-experience-developing-web-apps",{"title":69887,"description":464},"2017/12/03/the-twelve-factor-app-and-my-experience-developing-web-apps",[70667,30129],"development","mkFCn7h9TfIveSO9laiVJy15JJCRjpthWFqxT0isXew",{"id":70670,"title":70671,"body":70672,"comments":609,"date":72174,"description":72175,"draft":602,"extension":605,"external":606,"image":70869,"meta":72176,"navigation":609,"path":72177,"seo":72178,"stem":72179,"tags":72180,"__hash__":72182},"blog/2017/12/02/arch-linux-package-data-analysis.md","Analysis of AUR and Official Arch Repository data",{"type":8,"value":70673,"toc":72167},[70674,70681,70690,70694,70701,70706,70855,70858,70862,70865,70870,70873,70881,70898,70918,70921,70929,70932,70935,70959,70962,70967,71014,71019,71034,71039,71066,71071,71128,71133,71166,71186,71260,71263,71267,71282,71292,71295,71298,71658,71662,71665,71670,71673,72001,72005,72008,72013,72157,72164],[11,70675,70676,70677,70680],{},"Arch Linux provides packages through the official Arch Linux repositories and the Arch User Repository (AUR). I recently gathered data on ~50,000 packages from these repositories on ",[20,70678,70679],{"href":70679},"archlinux.org"," to better understand the makeup of the packages. In this article I will share some visualizations I made as well as some key takeaways about the data set I gathered.",[11,70682,70683,70684,70689],{},"The repo with all of the data I collected as well as the code I used to do so is available in ",[20,70685,70688],{"href":70686,"rel":70687},"https://github.com/briancaffey/AUR-data",[24],"this repository"," on my Github account.",[56,70691,70693],{"id":70692},"growth-of-the-aur","Growth of the AUR",[11,70695,70696,70697,70700],{},"The first questions I had about the dataset were about visualizing the growth of the AUR over time. Each package in the AUR has a ",[30,70698,70699],{},"First Submitted"," date, so I was able to put this together easily:",[11,70702,70703],{},[2718,70704],{"alt":20386,"src":70705},"/static/aur/aur_packages.png",[459,70707,70709],{"className":13136,"code":70708,"language":12886,"meta":464,"style":464},"sns.set()\ndf = df[df['First Submitted'].notnull()]\ndf[\"First Submitted\"] = pd.to_datetime(df['First Submitted'])\nlist_of_dates = df[\"First Submitted\"].sort_values()\ncounts = np.arange(0, len(list_of_dates))\nplt.figure(figsize=(10, 5))\n_ = plt.plot(list_of_dates, counts)\n_ = plt.title('AUR Packages over time')\n_ = plt.xlabel('Date')\n_ = plt.ylabel('Packages')\n",[30,70710,70711,70716,70732,70751,70766,70785,70803,70813,70827,70841],{"__ignoreMap":464},[151,70712,70713],{"class":469,"line":470},[151,70714,70715],{"class":503},"sns.set()\n",[151,70717,70718,70721,70723,70726,70729],{"class":469,"line":488},[151,70719,70720],{"class":503},"df ",[151,70722,1876],{"class":1869},[151,70724,70725],{"class":503}," df[df[",[151,70727,70728],{"class":481},"'First Submitted'",[151,70730,70731],{"class":503},"].notnull()]\n",[151,70733,70734,70737,70740,70742,70744,70747,70749],{"class":469,"line":500},[151,70735,70736],{"class":503},"df[",[151,70738,70739],{"class":481},"\"First Submitted\"",[151,70741,16654],{"class":503},[151,70743,1876],{"class":1869},[151,70745,70746],{"class":503}," pd.to_datetime(df[",[151,70748,70728],{"class":481},[151,70750,38820],{"class":503},[151,70752,70753,70756,70758,70761,70763],{"class":469,"line":509},[151,70754,70755],{"class":503},"list_of_dates ",[151,70757,1876],{"class":1869},[151,70759,70760],{"class":503}," df[",[151,70762,70739],{"class":481},[151,70764,70765],{"class":503},"].sort_values()\n",[151,70767,70768,70771,70773,70776,70778,70780,70782],{"class":469,"line":517},[151,70769,70770],{"class":503},"counts ",[151,70772,1876],{"class":1869},[151,70774,70775],{"class":503}," np.arange(",[151,70777,9181],{"class":477},[151,70779,106],{"class":503},[151,70781,65875],{"class":2226},[151,70783,70784],{"class":503},"(list_of_dates))\n",[151,70786,70787,70789,70791,70793,70795,70797,70799,70801],{"class":469,"line":534},[151,70788,44355],{"class":503},[151,70790,44358],{"class":15210},[151,70792,1876],{"class":1869},[151,70794,12386],{"class":503},[151,70796,12423],{"class":477},[151,70798,106],{"class":503},[151,70800,24380],{"class":477},[151,70802,12451],{"class":503},[151,70804,70805,70808,70810],{"class":469,"line":1413},[151,70806,70807],{"class":503},"_ ",[151,70809,1876],{"class":1869},[151,70811,70812],{"class":503}," plt.plot(list_of_dates, counts)\n",[151,70814,70815,70817,70819,70822,70825],{"class":469,"line":1418},[151,70816,70807],{"class":503},[151,70818,1876],{"class":1869},[151,70820,70821],{"class":503}," plt.title(",[151,70823,70824],{"class":481},"'AUR Packages over time'",[151,70826,3640],{"class":503},[151,70828,70829,70831,70833,70836,70839],{"class":469,"line":2462},[151,70830,70807],{"class":503},[151,70832,1876],{"class":1869},[151,70834,70835],{"class":503}," plt.xlabel(",[151,70837,70838],{"class":481},"'Date'",[151,70840,3640],{"class":503},[151,70842,70843,70845,70847,70850,70853],{"class":469,"line":2471},[151,70844,70807],{"class":503},[151,70846,1876],{"class":1869},[151,70848,70849],{"class":503}," plt.ylabel(",[151,70851,70852],{"class":481},"'Packages'",[151,70854,3640],{"class":503},[11,70856,70857],{},"It looks like there was a big boost in the number of packages submitted in mid-2015 and that number has been growing consistantly since then.",[56,70859,70861],{"id":70860},"official-repositories","Official Repositories",[11,70863,70864],{},"The official repositories contain just under 10,000 packages. Here is a force-directed graph (undirected graph) in D3.js that shows these packages as nodes and package dependencies as edges. This image shows a sample of about 1,000 packages and it is somewhat representative of the whole graph show further below.",[11,70866,70867],{},[2718,70868],{"alt":20386,"src":70869},"/static/aur/official_repos.png",[11,70871,70872],{},"There are four main repo in the official repositories:",[76,70874,70875],{},[79,70876,70877,70880],{},[15,70878,70879],{},"core"," contains packages for:",[76,70882,70883,70886,70889,70892,70895],{},[79,70884,70885],{},"booting Arch Linux",[79,70887,70888],{},"connecting to the Internet",[79,70890,70891],{},"building packages",[79,70893,70894],{},"management and repair of supported file systems",[79,70896,70897],{},"the system setup process (e.g. openssh)",[76,70899,70900,70906,70912],{},[79,70901,70902,70905],{},[15,70903,70904],{},"extra"," contains all packages that do not fit in core. Example: Xorg, window managers, web browsers, media players, tools for working with languages such as Python and Ruby, and a lot more.",[79,70907,70908,70911],{},[15,70909,70910],{},"community"," contains packages that have been adopted by Trusted Users from the Arch User Repository. Some of these packages may eventually make the transition to the core or extra repositories as the developers consider them crucial to the distribution.",[79,70913,70914,70917],{},[15,70915,70916],{},"multilib"," contains 32 bit software and libraries that can be used to run and build 32 bit applications on 64 bit installs (e.g. wine, steam, etc).",[11,70919,70920],{},"Here is an SVG showing all packages in the official repository. Click on the image to explore the SVG in more detail, and you can hover over nodes to see which packages they represent.",[11,70922,70923],{},[20,70924,70926],{"href":70925},"/static/aur/graph.svg",[2718,70927],{"alt":70928,"src":70925},"svg",[11,70930,70931],{},"Turn off the lights and you can see a ring of packages orbiting in a circle!",[70933,70934],"color-mode-picker",{},[11,70936,70937,70938,18952,70941,106,70943,38574,70946,70948,70949,70952,70953,187,70956,643],{},"If you look at this file in detail you can find some interesting clusters of packages. The \"island\" in the top left includes mostly Haskell packages and ",[30,70939,70940],{},"pandoc",[30,70942,12886],{},[30,70944,70945],{},"python2",[30,70947,1513],{}," are three of the main central hubs in the middle cluster. ",[30,70950,70951],{},"perl"," and other related packages make of most of the bottom right cluster and you can see \"flowers\" of package dependencies mostly for internationalization for popular programs like ",[30,70954,70955],{},"firefox",[30,70957,70958],{},"thunderbird",[11,70960,70961],{},"To make this interactive graph and SVG images I did the following:",[76,70963,70964],{},[79,70965,70966],{},"Create a dictionary from my data base with packages keys and a list of dependencies as values:",[459,70968,70970],{"className":13136,"code":70969,"language":12886,"meta":464,"style":464},"graph_dict = {}\nfor _, i in df.iterrows():\n    graph_dict[i[\"package_name\"]] = i[\"pkgdeps\"]\n",[30,70971,70972,70981,70993],{"__ignoreMap":464},[151,70973,70974,70977,70979],{"class":469,"line":470},[151,70975,70976],{"class":503},"graph_dict ",[151,70978,1876],{"class":1869},[151,70980,16634],{"class":503},[151,70982,70983,70985,70988,70990],{"class":469,"line":488},[151,70984,16732],{"class":1869},[151,70986,70987],{"class":503}," _, i ",[151,70989,16417],{"class":1869},[151,70991,70992],{"class":503}," df.iterrows():\n",[151,70994,70995,70998,71001,71004,71006,71009,71012],{"class":469,"line":500},[151,70996,70997],{"class":503},"    graph_dict[i[",[151,70999,71000],{"class":481},"\"package_name\"",[151,71002,71003],{"class":503},"]] ",[151,71005,1876],{"class":1869},[151,71007,71008],{"class":503}," i[",[151,71010,71011],{"class":481},"\"pkgdeps\"",[151,71013,3691],{"class":503},[76,71015,71016],{},[79,71017,71018],{},"Create a NetworkX graph with the dictionary created in the previous step:",[459,71020,71022],{"className":13136,"code":71021,"language":12886,"meta":464,"style":464},"G = nx.Graph(graph_dict)\n",[30,71023,71024],{"__ignoreMap":464},[151,71025,71026,71029,71031],{"class":469,"line":470},[151,71027,71028],{"class":503},"G ",[151,71030,1876],{"class":1869},[151,71032,71033],{"class":503}," nx.Graph(graph_dict)\n",[76,71035,71036],{},[79,71037,71038],{},"Export the NetworkX graph to JSON using a built-in NetworkX function:",[459,71040,71042],{"className":13136,"code":71041,"language":12886,"meta":464,"style":464},"from networkx.readwrite import json_graph\ndata = json_graph.node_link_data(G)\n",[30,71043,71044,71056],{"__ignoreMap":464},[151,71045,71046,71048,71051,71053],{"class":469,"line":470},[151,71047,16853],{"class":1869},[151,71049,71050],{"class":503}," networkx.readwrite ",[151,71052,16859],{"class":1869},[151,71054,71055],{"class":503}," json_graph\n",[151,71057,71058,71061,71063],{"class":469,"line":488},[151,71059,71060],{"class":503},"data ",[151,71062,1876],{"class":1869},[151,71064,71065],{"class":503}," json_graph.node_link_data(G)\n",[76,71067,71068],{},[79,71069,71070],{},"Add a group number to each node element cooresponding to the repository it belongs to.",[459,71072,71074],{"className":13136,"code":71073,"language":12886,"meta":464,"style":464},"for n in data['nodes']:\n    n['group'] = int(df.loc[(df.package_name == n[\"id\"]), \"repo_number\"].iloc[0])\n",[30,71075,71076,71091],{"__ignoreMap":464},[151,71077,71078,71080,71082,71084,71086,71089],{"class":469,"line":470},[151,71079,16732],{"class":1869},[151,71081,16735],{"class":503},[151,71083,16417],{"class":1869},[151,71085,17021],{"class":503},[151,71087,71088],{"class":481},"'nodes'",[151,71090,17073],{"class":503},[151,71092,71093,71096,71099,71101,71103,71105,71108,71110,71113,71115,71118,71121,71124,71126],{"class":469,"line":488},[151,71094,71095],{"class":503},"    n[",[151,71097,71098],{"class":481},"'group'",[151,71100,16654],{"class":503},[151,71102,1876],{"class":1869},[151,71104,16673],{"class":6205},[151,71106,71107],{"class":503},"(df.loc[(df.package_name ",[151,71109,17223],{"class":1869},[151,71111,71112],{"class":503}," n[",[151,71114,9739],{"class":481},[151,71116,71117],{"class":503},"]), ",[151,71119,71120],{"class":481},"\"repo_number\"",[151,71122,71123],{"class":503},"].iloc[",[151,71125,9181],{"class":477},[151,71127,38820],{"class":503},[76,71129,71130],{},[79,71131,71132],{},"Save the JSON to a file:",[459,71134,71136],{"className":13136,"code":71135,"language":12886,"meta":464,"style":464},"with open('/home/brian/Documents/github/briancaffey.github.io/aur/data.json', 'w') as outfile:\n    json.dump(data, outfile)\n",[30,71137,71138,71161],{"__ignoreMap":464},[151,71139,71140,71142,71144,71146,71149,71151,71154,71156,71158],{"class":469,"line":470},[151,71141,24959],{"class":1869},[151,71143,16970],{"class":2226},[151,71145,12386],{"class":503},[151,71147,71148],{"class":481},"'/home/brian/Documents/github/briancaffey.github.io/aur/data.json'",[151,71150,106],{"class":503},[151,71152,71153],{"class":481},"'w'",[151,71155,16995],{"class":503},[151,71157,16998],{"class":1869},[151,71159,71160],{"class":503}," outfile:\n",[151,71162,71163],{"class":469,"line":488},[151,71164,71165],{"class":503},"    json.dump(data, outfile)\n",[76,71167,71168,71177],{},[79,71169,71170,71171,71176],{},"Feed the JSON file into ",[20,71172,71175],{"href":71173,"rel":71174},"https://bl.ocks.org/mbostock/4062045",[24],"this template"," which renders a D3.js force-directed graph.",[79,71178,71179,71180,71185],{},"To save the graph as a SVG file, I ran the ",[20,71181,71184],{"href":71182,"rel":71183},"https://graphicdesign.stackexchange.com/questions/55123/how-do-i-save-an-svg-thats-on-a-website-to-my-computer",[24],"NYT crowbar script"," in the browser console:",[459,71187,71189],{"className":19459,"code":71188,"language":19461,"meta":464,"style":464},"var e = document.createElement('script')\ne.setAttribute('src', 'https://nytimes.github.io/svg-crowbar/svg-crowbar.js')\ne.setAttribute('class', 'svg-crowbar')\ndocument.body.appendChild(e)\n",[30,71190,71191,71212,71232,71250],{"__ignoreMap":464},[151,71192,71193,71195,71198,71200,71202,71205,71207,71210],{"class":469,"line":470},[151,71194,29289],{"class":12347},[151,71196,71197],{"class":503}," e ",[151,71199,1876],{"class":1869},[151,71201,40821],{"class":503},[151,71203,71204],{"class":473},"createElement",[151,71206,12386],{"class":503},[151,71208,71209],{"class":481},"'script'",[151,71211,3640],{"class":503},[151,71213,71214,71217,71220,71222,71225,71227,71230],{"class":469,"line":488},[151,71215,71216],{"class":503},"e.",[151,71218,71219],{"class":473},"setAttribute",[151,71221,12386],{"class":503},[151,71223,71224],{"class":481},"'src'",[151,71226,106],{"class":503},[151,71228,71229],{"class":481},"'https://nytimes.github.io/svg-crowbar/svg-crowbar.js'",[151,71231,3640],{"class":503},[151,71233,71234,71236,71238,71240,71243,71245,71248],{"class":469,"line":500},[151,71235,71216],{"class":503},[151,71237,71219],{"class":473},[151,71239,12386],{"class":503},[151,71241,71242],{"class":481},"'class'",[151,71244,106],{"class":503},[151,71246,71247],{"class":481},"'svg-crowbar'",[151,71249,3640],{"class":503},[151,71251,71252,71255,71258],{"class":469,"line":509},[151,71253,71254],{"class":503},"document.body.",[151,71256,71257],{"class":473},"appendChild",[151,71259,18358],{"class":503},[11,71261,71262],{},"Let's take one more look at how tightly",[56,71264,71266],{"id":71265},"official-repository-package-sizes","Official Repository Package Sizes",[11,71268,71269,71270,187,71273,71276,71277,71279,71280,208],{},"Official Packages include both a ",[30,71271,71272],{},"Package Size",[30,71274,71275],{},"Installed Size",". Here is a Bokeh plot showing ",[30,71278,71272],{}," vs. ",[30,71281,71275],{},[11,71283,71284,71287,71288,71291],{},[15,71285,71286],{},"Warning",": Don't hover directly over the cluster of plotted points near the origin of the graph. ",[15,71289,71290],{},"DOING SO WILL CRASH YOUR BROWSER",". This is because the hover tool will attempt to display all packages that you are hovered over and it may be far too many for the browser to handle. Carefully zoom in using the scroll tool and you can find some interesting trends in the types of packages and how much they are able to be compressed.",[11,71293,71294],{},"{% include package_sizes.html %}",[11,71296,71297],{},"Here's the setup for this bokeh graph:",[459,71299,71301],{"className":13136,"code":71300,"language":12886,"meta":464,"style":464},"from bokeh.plotting import figure, output_file, show, ColumnDataSource\nfrom bokeh.models import HoverTool\nfrom bokeh.io import output_notebook\noutput_notebook()\n\noutput_file(\"/home/brian/Documents/github/briancaffey.github.io/_includes/package_sizes.html\")\n\nsource = ColumnDataSource(\n        data=dict(\n            x=df.package_size,\n            y=df.installed_size,\n            desc=df.Description,\n            name=df.package_name\n        )\n    )\n\nhover = HoverTool(\n        tooltips=[\n            (\"Name\", \"@name\"),\n            (\"Package Size\", \"@x MB\"),\n            (\"Installed Size\", \"@y MB\"),\n            (\"Description\", \"@desc\"),\n        ]\n    )\n\nTOOLS = 'box_zoom,box_select,reset,pan,wheel_zoom'\n\np = figure(plot_width=400, plot_height=400, tools=[TOOLS, hover],\n           title=\"Packages Size vs. Installed Size\", sizing_mode='scale_width')\n\np.circle('x', 'y', size=5, source=source, alpha=0.2)\np.toolbar.logo = None\nshow(p)\n",[30,71302,71303,71315,71327,71339,71344,71348,71358,71362,71372,71383,71393,71403,71413,71423,71427,71431,71435,71445,71454,71469,71483,71497,71511,71515,71519,71523,71533,71537,71578,71600,71604,71644,71653],{"__ignoreMap":464},[151,71304,71305,71307,71310,71312],{"class":469,"line":470},[151,71306,16853],{"class":1869},[151,71308,71309],{"class":503}," bokeh.plotting ",[151,71311,16859],{"class":1869},[151,71313,71314],{"class":503}," figure, output_file, show, ColumnDataSource\n",[151,71316,71317,71319,71322,71324],{"class":469,"line":488},[151,71318,16853],{"class":1869},[151,71320,71321],{"class":503}," bokeh.models ",[151,71323,16859],{"class":1869},[151,71325,71326],{"class":503}," HoverTool\n",[151,71328,71329,71331,71334,71336],{"class":469,"line":500},[151,71330,16853],{"class":1869},[151,71332,71333],{"class":503}," bokeh.io ",[151,71335,16859],{"class":1869},[151,71337,71338],{"class":503}," output_notebook\n",[151,71340,71341],{"class":469,"line":509},[151,71342,71343],{"class":503},"output_notebook()\n",[151,71345,71346],{"class":469,"line":517},[151,71347,1090],{"emptyLinePlaceholder":609},[151,71349,71350,71353,71356],{"class":469,"line":534},[151,71351,71352],{"class":503},"output_file(",[151,71354,71355],{"class":481},"\"/home/brian/Documents/github/briancaffey.github.io/_includes/package_sizes.html\"",[151,71357,3640],{"class":503},[151,71359,71360],{"class":469,"line":1413},[151,71361,1090],{"emptyLinePlaceholder":609},[151,71363,71364,71367,71369],{"class":469,"line":1418},[151,71365,71366],{"class":503},"source ",[151,71368,1876],{"class":1869},[151,71370,71371],{"class":503}," ColumnDataSource(\n",[151,71373,71374,71376,71378,71381],{"class":469,"line":2462},[151,71375,57102],{"class":15210},[151,71377,1876],{"class":1869},[151,71379,71380],{"class":6205},"dict",[151,71382,15410],{"class":503},[151,71384,71385,71388,71390],{"class":469,"line":2471},[151,71386,71387],{"class":15210},"            x",[151,71389,1876],{"class":1869},[151,71391,71392],{"class":503},"df.package_size,\n",[151,71394,71395,71398,71400],{"class":469,"line":2480},[151,71396,71397],{"class":15210},"            y",[151,71399,1876],{"class":1869},[151,71401,71402],{"class":503},"df.installed_size,\n",[151,71404,71405,71408,71410],{"class":469,"line":2489},[151,71406,71407],{"class":15210},"            desc",[151,71409,1876],{"class":1869},[151,71411,71412],{"class":503},"df.Description,\n",[151,71414,71415,71418,71420],{"class":469,"line":2497},[151,71416,71417],{"class":15210},"            name",[151,71419,1876],{"class":1869},[151,71421,71422],{"class":503},"df.package_name\n",[151,71424,71425],{"class":469,"line":3140},[151,71426,16824],{"class":503},[151,71428,71429],{"class":469,"line":3149},[151,71430,39567],{"class":503},[151,71432,71433],{"class":469,"line":3158},[151,71434,1090],{"emptyLinePlaceholder":609},[151,71436,71437,71440,71442],{"class":469,"line":3167},[151,71438,71439],{"class":503},"hover ",[151,71441,1876],{"class":1869},[151,71443,71444],{"class":503}," HoverTool(\n",[151,71446,71447,71450,71452],{"class":469,"line":3175},[151,71448,71449],{"class":15210},"        tooltips",[151,71451,1876],{"class":1869},[151,71453,37620],{"class":503},[151,71455,71456,71459,71462,71464,71467],{"class":469,"line":3184},[151,71457,71458],{"class":503},"            (",[151,71460,71461],{"class":481},"\"Name\"",[151,71463,106],{"class":503},[151,71465,71466],{"class":481},"\"@name\"",[151,71468,37985],{"class":503},[151,71470,71471,71473,71476,71478,71481],{"class":469,"line":3193},[151,71472,71458],{"class":503},[151,71474,71475],{"class":481},"\"Package Size\"",[151,71477,106],{"class":503},[151,71479,71480],{"class":481},"\"@x MB\"",[151,71482,37985],{"class":503},[151,71484,71485,71487,71490,71492,71495],{"class":469,"line":3720},[151,71486,71458],{"class":503},[151,71488,71489],{"class":481},"\"Installed Size\"",[151,71491,106],{"class":503},[151,71493,71494],{"class":481},"\"@y MB\"",[151,71496,37985],{"class":503},[151,71498,71499,71501,71504,71506,71509],{"class":469,"line":3729},[151,71500,71458],{"class":503},[151,71502,71503],{"class":481},"\"Description\"",[151,71505,106],{"class":503},[151,71507,71508],{"class":481},"\"@desc\"",[151,71510,37985],{"class":503},[151,71512,71513],{"class":469,"line":3735},[151,71514,41397],{"class":503},[151,71516,71517],{"class":469,"line":3745},[151,71518,39567],{"class":503},[151,71520,71521],{"class":469,"line":3754},[151,71522,1090],{"emptyLinePlaceholder":609},[151,71524,71525,71528,71530],{"class":469,"line":3760},[151,71526,71527],{"class":477},"TOOLS",[151,71529,19865],{"class":1869},[151,71531,71532],{"class":481}," 'box_zoom,box_select,reset,pan,wheel_zoom'\n",[151,71534,71535],{"class":469,"line":3773},[151,71536,1090],{"emptyLinePlaceholder":609},[151,71538,71539,71542,71544,71547,71550,71552,71555,71557,71560,71562,71564,71566,71569,71571,71573,71575],{"class":469,"line":3782},[151,71540,71541],{"class":503},"p ",[151,71543,1876],{"class":1869},[151,71545,71546],{"class":503}," figure(",[151,71548,71549],{"class":15210},"plot_width",[151,71551,1876],{"class":1869},[151,71553,71554],{"class":477},"400",[151,71556,106],{"class":503},[151,71558,71559],{"class":15210},"plot_height",[151,71561,1876],{"class":1869},[151,71563,71554],{"class":477},[151,71565,106],{"class":503},[151,71567,71568],{"class":15210},"tools",[151,71570,1876],{"class":1869},[151,71572,6698],{"class":503},[151,71574,71527],{"class":477},[151,71576,71577],{"class":503},", hover],\n",[151,71579,71580,71583,71585,71588,71590,71593,71595,71598],{"class":469,"line":3791},[151,71581,71582],{"class":15210},"           title",[151,71584,1876],{"class":1869},[151,71586,71587],{"class":481},"\"Packages Size vs. Installed Size\"",[151,71589,106],{"class":503},[151,71591,71592],{"class":15210},"sizing_mode",[151,71594,1876],{"class":1869},[151,71596,71597],{"class":481},"'scale_width'",[151,71599,3640],{"class":503},[151,71601,71602],{"class":469,"line":3803},[151,71603,1090],{"emptyLinePlaceholder":609},[151,71605,71606,71609,71612,71614,71617,71619,71622,71624,71626,71628,71630,71632,71635,71637,71639,71642],{"class":469,"line":3811},[151,71607,71608],{"class":503},"p.circle(",[151,71610,71611],{"class":481},"'x'",[151,71613,106],{"class":503},[151,71615,71616],{"class":481},"'y'",[151,71618,106],{"class":503},[151,71620,71621],{"class":15210},"size",[151,71623,1876],{"class":1869},[151,71625,24380],{"class":477},[151,71627,106],{"class":503},[151,71629,23905],{"class":15210},[151,71631,1876],{"class":1869},[151,71633,71634],{"class":503},"source, ",[151,71636,26256],{"class":15210},[151,71638,1876],{"class":1869},[151,71640,71641],{"class":477},"0.2",[151,71643,3640],{"class":503},[151,71645,71646,71649,71651],{"class":469,"line":3820},[151,71647,71648],{"class":503},"p.toolbar.logo ",[151,71650,1876],{"class":1869},[151,71652,53115],{"class":477},[151,71654,71655],{"class":469,"line":7084},[151,71656,71657],{"class":503},"show(p)\n",[56,71659,71661],{"id":71660},"aur-word-cloud","AUR Word Cloud",[11,71663,71664],{},"Let's make a word cloud out of text descriptions for packages in the AUR.",[11,71666,71667],{},[2718,71668],{"alt":20386,"src":71669},"/static/aur/word_cloud.png",[11,71671,71672],{},"We can use a popular python package for making word clouds. Here's the code:",[459,71674,71676],{"className":13136,"code":71675,"language":12886,"meta":464,"style":464},"import numpy as np\nfrom PIL import Image\nfrom os import path\nimport matplotlib.pyplot as plt\nimport random\n\nfrom wordcloud import WordCloud, STOPWORDS\n\ndef grey_color_func(word, font_size, position, orientation, random_state=None,\n                    **kwargs):\n    return \"hsl(0, 0%%, %d%%)\" % random.randint(60, 100)\n\nmask = np.array(Image.open(\"/home/brian/Documents/aur/images/arch_logo.png\"))\n\ntext = open(\"/home/brian/Documents/aur/ipynb/package_descriptions.txt\").read()\n\nwc = WordCloud(max_words=1000, mask=mask, stopwords=stopwords, margin=10,\n               random_state=1).generate(text)\n\ndefault_colors = wc.to_array()\nplt.figure(figsize=(20, 20))\nplt.imshow(wc.recolor(color_func=grey_color_func, random_state=3),\n           interpolation=\"bilinear\")\nwc.to_file(\"arch_word_cloud.png\")\nplt.axis(\"off\")\nplt.show()\n",[30,71677,71678,71688,71698,71708,71718,71724,71728,71740,71744,71782,71791,71824,71828,71843,71847,71863,71867,71906,71917,71921,71929,71947,71968,71979,71989,71997],{"__ignoreMap":464},[151,71679,71680,71682,71684,71686],{"class":469,"line":470},[151,71681,16859],{"class":1869},[151,71683,24412],{"class":503},[151,71685,16998],{"class":1869},[151,71687,24417],{"class":503},[151,71689,71690,71692,71694,71696],{"class":469,"line":488},[151,71691,16853],{"class":1869},[151,71693,44099],{"class":477},[151,71695,44102],{"class":1869},[151,71697,44105],{"class":503},[151,71699,71700,71702,71704,71706],{"class":469,"line":500},[151,71701,16853],{"class":1869},[151,71703,44057],{"class":503},[151,71705,16859],{"class":1869},[151,71707,44062],{"class":503},[151,71709,71710,71712,71714,71716],{"class":469,"line":509},[151,71711,16859],{"class":1869},[151,71713,44073],{"class":503},[151,71715,16998],{"class":1869},[151,71717,44078],{"class":503},[151,71719,71720,71722],{"class":469,"line":517},[151,71721,16859],{"class":1869},[151,71723,44034],{"class":503},[151,71725,71726],{"class":469,"line":534},[151,71727,1090],{"emptyLinePlaceholder":609},[151,71729,71730,71732,71734,71736,71738],{"class":469,"line":1413},[151,71731,16853],{"class":1869},[151,71733,44112],{"class":503},[151,71735,16859],{"class":1869},[151,71737,44117],{"class":503},[151,71739,44120],{"class":477},[151,71741,71742],{"class":469,"line":1418},[151,71743,1090],{"emptyLinePlaceholder":609},[151,71745,71746,71748,71751,71753,71756,71758,71761,71763,71766,71768,71771,71773,71776,71778,71780],{"class":469,"line":2462},[151,71747,16925],{"class":12347},[151,71749,71750],{"class":473}," grey_color_func",[151,71752,12386],{"class":503},[151,71754,71755],{"class":15232},"word",[151,71757,106],{"class":503},[151,71759,71760],{"class":15232},"font_size",[151,71762,106],{"class":503},[151,71764,71765],{"class":15232},"position",[151,71767,106],{"class":503},[151,71769,71770],{"class":15232},"orientation",[151,71772,106],{"class":503},[151,71774,71775],{"class":15232},"random_state",[151,71777,1876],{"class":1869},[151,71779,15437],{"class":477},[151,71781,9417],{"class":503},[151,71783,71784,71787,71789],{"class":469,"line":2471},[151,71785,71786],{"class":1869},"                    **",[151,71788,37866],{"class":15232},[151,71790,15264],{"class":503},[151,71792,71793,71795,71798,71801,71803,71806,71809,71812,71815,71817,71819,71822],{"class":469,"line":2480},[151,71794,17496],{"class":1869},[151,71796,71797],{"class":481}," \"hsl(0, 0",[151,71799,71800],{"class":477},"%%",[151,71802,106],{"class":481},[151,71804,71805],{"class":477},"%d%%",[151,71807,71808],{"class":481},")\"",[151,71810,71811],{"class":1869}," %",[151,71813,71814],{"class":503}," random.randint(",[151,71816,39825],{"class":477},[151,71818,106],{"class":503},[151,71820,71821],{"class":477},"100",[151,71823,3640],{"class":503},[151,71825,71826],{"class":469,"line":2489},[151,71827,1090],{"emptyLinePlaceholder":609},[151,71829,71830,71833,71835,71838,71841],{"class":469,"line":2497},[151,71831,71832],{"class":503},"mask ",[151,71834,1876],{"class":1869},[151,71836,71837],{"class":503}," np.array(Image.open(",[151,71839,71840],{"class":481},"\"/home/brian/Documents/aur/images/arch_logo.png\"",[151,71842,12451],{"class":503},[151,71844,71845],{"class":469,"line":3140},[151,71846,1090],{"emptyLinePlaceholder":609},[151,71848,71849,71852,71854,71856,71858,71861],{"class":469,"line":3149},[151,71850,71851],{"class":503},"text ",[151,71853,1876],{"class":1869},[151,71855,16970],{"class":2226},[151,71857,12386],{"class":503},[151,71859,71860],{"class":481},"\"/home/brian/Documents/aur/ipynb/package_descriptions.txt\"",[151,71862,64791],{"class":503},[151,71864,71865],{"class":469,"line":3158},[151,71866,1090],{"emptyLinePlaceholder":609},[151,71868,71869,71871,71873,71875,71877,71879,71881,71883,71886,71888,71891,71893,71895,71898,71900,71902,71904],{"class":469,"line":3167},[151,71870,44257],{"class":503},[151,71872,1876],{"class":1869},[151,71874,44262],{"class":503},[151,71876,44295],{"class":15210},[151,71878,1876],{"class":1869},[151,71880,45779],{"class":477},[151,71882,106],{"class":503},[151,71884,71885],{"class":15210},"mask",[151,71887,1876],{"class":1869},[151,71889,71890],{"class":503},"mask, ",[151,71892,44304],{"class":15210},[151,71894,1876],{"class":1869},[151,71896,71897],{"class":503},"stopwords, ",[151,71899,44314],{"class":15210},[151,71901,1876],{"class":1869},[151,71903,12423],{"class":477},[151,71905,9417],{"class":503},[151,71907,71908,71910,71912,71914],{"class":469,"line":3175},[151,71909,44325],{"class":15210},[151,71911,1876],{"class":1869},[151,71913,6760],{"class":477},[151,71915,71916],{"class":503},").generate(text)\n",[151,71918,71919],{"class":469,"line":3184},[151,71920,1090],{"emptyLinePlaceholder":609},[151,71922,71923,71925,71927],{"class":469,"line":3193},[151,71924,44341],{"class":503},[151,71926,1876],{"class":1869},[151,71928,44346],{"class":503},[151,71930,71931,71933,71935,71937,71939,71941,71943,71945],{"class":469,"line":3720},[151,71932,44355],{"class":503},[151,71934,44358],{"class":15210},[151,71936,1876],{"class":1869},[151,71938,12386],{"class":503},[151,71940,9097],{"class":477},[151,71942,106],{"class":503},[151,71944,9097],{"class":477},[151,71946,12451],{"class":503},[151,71948,71949,71952,71955,71957,71960,71962,71964,71966],{"class":469,"line":3729},[151,71950,71951],{"class":503},"plt.imshow(wc.recolor(",[151,71953,71954],{"class":15210},"color_func",[151,71956,1876],{"class":1869},[151,71958,71959],{"class":503},"grey_color_func, ",[151,71961,71775],{"class":15210},[151,71963,1876],{"class":1869},[151,71965,6557],{"class":477},[151,71967,37985],{"class":503},[151,71969,71970,71973,71975,71977],{"class":469,"line":3735},[151,71971,71972],{"class":15210},"           interpolation",[151,71974,1876],{"class":1869},[151,71976,44384],{"class":481},[151,71978,3640],{"class":503},[151,71980,71981,71984,71987],{"class":469,"line":3745},[151,71982,71983],{"class":503},"wc.to_file(",[151,71985,71986],{"class":481},"\"arch_word_cloud.png\"",[151,71988,3640],{"class":503},[151,71990,71991,71993,71995],{"class":469,"line":3754},[151,71992,44395],{"class":503},[151,71994,44398],{"class":481},[151,71996,3640],{"class":503},[151,71998,71999],{"class":469,"line":3760},[151,72000,44415],{"class":503},[56,72002,72004],{"id":72003},"arch-wiki-members","Arch Wiki Members",[11,72006,72007],{},"The Arch wiki is the first place I go for troubleshooting any issue with Arch. Users of other Linux distributions have also said how useful it can be even if you don't user Arch Linux. Here's a look at the number of registered users on the Arch Wiki over time:",[11,72009,72010],{},[2718,72011],{"alt":20386,"src":72012},"/static/aur/wiki_users.png",[459,72014,72016],{"className":13136,"code":72015,"language":12886,"meta":464,"style":464},"sns.set()\ndf = df[df['registered'].notnull()]\ndf[\"registered\"] = pd.to_datetime(df['registered'])\nlist_of_dates = df[\"registered\"].sort_values()\ncounts = np.arange(0, len(list_of_dates))\nplt.figure(figsize=(10, 5))\n_ = plt.plot(list_of_dates, counts)\n_ = plt.title('Registered Arch Wiki members over time')\n_ = plt.xlabel('Date')\n_ = plt.ylabel('Members')\nplt.show()\nplt.savefig('/home/brian/Documents/github/briancaffey.github.io/aur/wiki_users.png')\n",[30,72017,72018,72022,72035,72052,72064,72080,72098,72106,72119,72131,72144,72148],{"__ignoreMap":464},[151,72019,72020],{"class":469,"line":470},[151,72021,70715],{"class":503},[151,72023,72024,72026,72028,72030,72033],{"class":469,"line":488},[151,72025,70720],{"class":503},[151,72027,1876],{"class":1869},[151,72029,70725],{"class":503},[151,72031,72032],{"class":481},"'registered'",[151,72034,70731],{"class":503},[151,72036,72037,72039,72042,72044,72046,72048,72050],{"class":469,"line":500},[151,72038,70736],{"class":503},[151,72040,72041],{"class":481},"\"registered\"",[151,72043,16654],{"class":503},[151,72045,1876],{"class":1869},[151,72047,70746],{"class":503},[151,72049,72032],{"class":481},[151,72051,38820],{"class":503},[151,72053,72054,72056,72058,72060,72062],{"class":469,"line":509},[151,72055,70755],{"class":503},[151,72057,1876],{"class":1869},[151,72059,70760],{"class":503},[151,72061,72041],{"class":481},[151,72063,70765],{"class":503},[151,72065,72066,72068,72070,72072,72074,72076,72078],{"class":469,"line":517},[151,72067,70770],{"class":503},[151,72069,1876],{"class":1869},[151,72071,70775],{"class":503},[151,72073,9181],{"class":477},[151,72075,106],{"class":503},[151,72077,65875],{"class":2226},[151,72079,70784],{"class":503},[151,72081,72082,72084,72086,72088,72090,72092,72094,72096],{"class":469,"line":534},[151,72083,44355],{"class":503},[151,72085,44358],{"class":15210},[151,72087,1876],{"class":1869},[151,72089,12386],{"class":503},[151,72091,12423],{"class":477},[151,72093,106],{"class":503},[151,72095,24380],{"class":477},[151,72097,12451],{"class":503},[151,72099,72100,72102,72104],{"class":469,"line":1413},[151,72101,70807],{"class":503},[151,72103,1876],{"class":1869},[151,72105,70812],{"class":503},[151,72107,72108,72110,72112,72114,72117],{"class":469,"line":1418},[151,72109,70807],{"class":503},[151,72111,1876],{"class":1869},[151,72113,70821],{"class":503},[151,72115,72116],{"class":481},"'Registered Arch Wiki members over time'",[151,72118,3640],{"class":503},[151,72120,72121,72123,72125,72127,72129],{"class":469,"line":2462},[151,72122,70807],{"class":503},[151,72124,1876],{"class":1869},[151,72126,70835],{"class":503},[151,72128,70838],{"class":481},[151,72130,3640],{"class":503},[151,72132,72133,72135,72137,72139,72142],{"class":469,"line":2471},[151,72134,70807],{"class":503},[151,72136,1876],{"class":1869},[151,72138,70849],{"class":503},[151,72140,72141],{"class":481},"'Members'",[151,72143,3640],{"class":503},[151,72145,72146],{"class":469,"line":2480},[151,72147,44415],{"class":503},[151,72149,72150,72152,72155],{"class":469,"line":2489},[151,72151,44405],{"class":503},[151,72153,72154],{"class":481},"'/home/brian/Documents/github/briancaffey.github.io/aur/wiki_users.png'",[151,72156,3640],{"class":503},[11,72158,72159,72160,643],{},"There is a massive amount of data in the Wiki that I haven't obtained for this article. You can also find some interesting statistics on the Arch Wiki site ",[20,72161,13074],{"href":72162,"rel":72163},"https://wiki.archlinux.org/index.php/ArchWiki:Statistics#Histograms",[24],[589,72165,72166],{},"html pre.shiki code .sMOD_, html code.shiki .sMOD_{--shiki-default:#24292E;--shiki-dark:#E1E4E8;--shiki-sepia:#F8F8F2}html pre.shiki code .sC2Qs, html code.shiki .sC2Qs{--shiki-default:#D73A49;--shiki-dark:#F97583;--shiki-sepia:#F92672}html pre.shiki code .sstjo, html code.shiki .sstjo{--shiki-default:#032F62;--shiki-dark:#9ECBFF;--shiki-sepia:#E6DB74}html pre.shiki code .s7F3e, html code.shiki .s7F3e{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#AE81FF}html pre.shiki code .sTrkL, html code.shiki .sTrkL{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#66D9EF}html pre.shiki code .sTHNf, html code.shiki .sTHNf{--shiki-default:#E36209;--shiki-default-font-style:inherit;--shiki-dark:#FFAB70;--shiki-dark-font-style:inherit;--shiki-sepia:#FD971F;--shiki-sepia-font-style:italic}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html .sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html.sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html pre.shiki code .s-m8C, html code.shiki .s-m8C{--shiki-default:#005CC5;--shiki-default-font-style:inherit;--shiki-dark:#79B8FF;--shiki-dark-font-style:inherit;--shiki-sepia:#66D9EF;--shiki-sepia-font-style:italic}html pre.shiki code .sq6CD, html code.shiki .sq6CD{--shiki-default:#D73A49;--shiki-default-font-style:inherit;--shiki-dark:#F97583;--shiki-dark-font-style:inherit;--shiki-sepia:#66D9EF;--shiki-sepia-font-style:italic}html pre.shiki code .srTi1, html code.shiki .srTi1{--shiki-default:#6F42C1;--shiki-dark:#B392F0;--shiki-sepia:#A6E22E}html pre.shiki code .so59x, html code.shiki .so59x{--shiki-default:#24292E;--shiki-default-font-style:inherit;--shiki-dark:#E1E4E8;--shiki-dark-font-style:inherit;--shiki-sepia:#FD971F;--shiki-sepia-font-style:italic}",{"title":464,"searchDepth":488,"depth":488,"links":72168},[72169,72170,72171,72172,72173],{"id":70692,"depth":488,"text":70693},{"id":70860,"depth":488,"text":70861},{"id":71265,"depth":488,"text":71266},{"id":71660,"depth":488,"text":71661},{"id":72003,"depth":488,"text":72004},"2017-12-02","Arch Linux provides packages through the official Arch Linux repositories and the Arch User Repository (AUR). I recently gathered data on ~50,000 packages from these repositories on archlinux.org to better understand the makeup of the packages. In this article I will share some visualizations I made as well as some key takeaways about the data set I gathered.",{"layout":48045},"/2017/12/02/arch-linux-package-data-analysis",{"title":70671,"description":72175},"2017/12/02/arch-linux-package-data-analysis",[72181,12355,12886],"arch-linux","dN_Y426m-pwCzBqV-bbP7MCJR2JChPiT8jl2dA5QUB4",{"id":72184,"title":72185,"body":72186,"comments":609,"date":72365,"description":72190,"draft":602,"extension":605,"external":606,"image":72292,"meta":72366,"navigation":609,"path":72367,"seo":72368,"stem":72369,"tags":72370,"__hash__":72372},"blog/2017/11/28/remove-root-partition-bloat-from-docker.md","Removing root partition bloat caused by docker",{"type":8,"value":72187,"toc":72363},[72188,72191,72202,72209,72214,72225,72231,72234,72240,72243,72248,72256,72259,72267,72270,72273,72279,72282,72288,72293,72296,72302,72308,72314,72317,72332,72340,72357],[11,72189,72190],{},"Recently I've been having storage issues in the root partitions of both my desktop and laptop computers. These issues came up soon after I started playing around with docker. In this article I'll talk briefly about how I fixed this problem, the resources and tools I picked up along the way, and anything else I have learned along the way.",[11,72192,72193,72194,72197,72198,72201],{},"I first learned how bad this issue was when I went to install anaconda on my desktop PC. I quiclky ran ",[30,72195,72196],{},"df -h"," and saw that my 20G root partition had less than 1G of available space. To look further into this I ran ",[30,72199,72200],{},"baobab",". In the baobab home screen the root partition had slightly more space, but it was still close to being full. The expanded view was only showing me informatoin for around 8G of storage, leaving almost 11G of space not accounted for.",[11,72203,72204,72205,72208],{},"I started reaching for different tools and packages to slim down disk usage. ",[30,72206,72207],{},"pacgraph"," is a pretty neat way to visualize the relative size of packages. Here's an example:",[11,72210,72211],{},[2718,72212],{"alt":20386,"src":72213},"/static/pacgraph.png",[11,72215,72216,72217,187,72219,72221,72222,72224],{},"This helps you quickly find packages that you can do without. After removing some large packages like Libre Office I realized that this was barely moving the needle on my storage problem. Running ",[30,72218,72196],{},[30,72220,72200],{}," again with root priviledges gave me slightly different results. At this point I turned to docker and deleted all of the images with `docker rmi ",[2718,72223],{"id":464}," -f. This didn't help either. Here are the images that I removed from my desktop:",[459,72226,72229],{"className":72227,"code":72228,"language":997},[995],"[brian@a1arch ~]$ docker images\nREPOSITORY                              TAG                 IMAGE ID            CREATED             SIZE\nflaskmicroservicesusers_users-service   latest              1e59fa4d2af5        5 days ago          739MB\n\u003Cnone>                                  \u003Cnone>              18f9191b4d9a        5 days ago          739MB\nflaskmicroservicesusers_users-db        latest              f1de1c3ef3f2        5 days ago          287MB\n\u003Cnone>                                  \u003Cnone>              11188ac6f36a        5 days ago          712MB\npostgres                                latest              599272bf538f        12 days ago         287MB\ntensorflow/tensorflow                   latest-gpu          2f243a16ff63        3 weeks ago         3.36GB\npython                                  3.6.2               26acbad26a2c        2 months ago        690MB\n",[30,72230,72228],{"__ignoreMap":464},[11,72232,72233],{},"Here's the storage profile before I started remove docker-related files:",[459,72235,72238],{"className":72236,"code":72237,"language":997},[995]," $ df -h | grep /dev/sda1\n/dev/sda1        20G   18G  737M  97% /\n",[30,72239,72237],{"__ignoreMap":464},[11,72241,72242],{},"After I removed the docker images, here is the same command:",[459,72244,72246],{"className":72245,"code":72237,"language":997},[995],[30,72247,72237],{"__ignoreMap":464},[11,72249,72250,72251,643],{},"I found a helpful serverfault question from 6 years ago that address the issue I was having titled ",[20,72252,72255],{"href":72253,"rel":72254},"https://serverfault.com/questions/275206/disk-full-du-tells-different-how-to-further-investigate",[24],"Disk full, du tells different. How to further investigate?",[11,72257,72258],{},"I saw a helpful comment related to docker:",[210,72260,72261],{},[11,72262,72263,72264],{},"Thanks - this showed that docker was filling up my hard drive with diffs in ",[30,72265,72266],{},"/var/lib/docker/aufs/diff/",[11,72268,72269],{},"Could this be my issue?",[11,72271,72272],{},"Here's the folder in question on my laptop:",[459,72274,72277],{"className":72275,"code":72276,"language":997},[995]," $ cd /var/lib/docker\n $ sudo du -s -h .\n2.6G    .\n",[30,72278,72276],{"__ignoreMap":464},[11,72280,72281],{},"On my desktop this was taking up about 10G!",[11,72283,72284,72285,208],{},"Wow! I didn't even see this when I ran ",[30,72286,72287],{},"sudo baobab",[11,72289,72290],{},[2718,72291],{"alt":20386,"src":72292},"/static/baobab.png",[11,72294,72295],{},"I stopped the docker service and deleted the overlay2 file:",[459,72297,72300],{"className":72298,"code":72299,"language":997},[995]," $ sudo systemctl stop docker\n $ cd /var/lib/docker\n $ sudo rm -rf layover2\n",[30,72301,72299],{"__ignoreMap":464},[11,72303,72304,72305,72307],{},"With ",[30,72306,72287],{}," I was also able to delete 3.6G of trash with this command:",[459,72309,72312],{"className":72310,"code":72311,"language":997},[995]," $ sudo -i\n # rm -rf /root/.local/share/Trash\n",[30,72313,72311],{"__ignoreMap":464},[11,72315,72316],{},"I think this may be related to having previously emptied the Trash in nautilus file browser with files that I might not have owned.",[11,72318,72319,72320,72325,72326,72331],{},"I think it would be a good idea to change the docker image installation directory. ",[20,72321,72324],{"href":72322,"rel":72323},"https://forums.docker.com/t/how-do-i-change-the-docker-image-installation-directory/1169",[24],"Here is a link"," from a docker forum talking about how to do that. ",[20,72327,72330],{"href":72328,"rel":72329},"https://forums.docker.com/t/some-way-to-clean-up-identify-contents-of-var-lib-docker-overlay/30604",[24],"Here is another docker forum post"," that talks about the overlay and storage issues that docker has.",[11,72333,72334,72335,208],{},"Here is a helpful snippet from the ",[20,72336,72339],{"href":72337,"rel":72338},"https://wiki.archlinux.org/index.php/Docker",[24],"Arch Wiki Docker article",[210,72341,72342,72345,72348,72351],{},[11,72343,72344],{},"Images location\nBy default, docker images are located at /var/lib/docker. They can be moved to other partitions. First, stop the docker.service.",[11,72346,72347],{},"If you have run the docker images, you need to make sure the images are unmounted totally. Once that is completed, you may move the images from /var/lib/docker to the target destination.",[11,72349,72350],{},"Then add a Drop-in snippet for the docker.service, adding the --data-root parameter to the ExecStart:",[459,72352,72355],{"className":72353,"code":72354,"language":997},[995],"/etc/systemd/system/docker.service.d/docker-storage.conf\n[Service]\nExecStart=\nExecStart=/usr/bin/dockerd --data-root=/path/to/new/location/docker -H fd://\n",[30,72356,72354],{"__ignoreMap":464},[459,72358,72361],{"className":72359,"code":72360,"language":997},[995],"\nUpdate: I did this on my desktop with a `--data-rogettingot` path in my home folder.\n\nI followed the directions form [this article](https://linuxconfig.org/how-to-move-docker-s-default-var-lib-docker-to-another-directory-on-ubuntu-debian-linux) and was able to set up docker on my home partition.\n",[30,72362,72360],{"__ignoreMap":464},{"title":464,"searchDepth":488,"depth":488,"links":72364},[],"2017-11-28",{"layout":48045},"/2017/11/28/remove-root-partition-bloat-from-docker",{"title":72185,"description":72190},"2017/11/28/remove-root-partition-bloat-from-docker",[30129,72371],"linux","ybCRaGwh7aK22BZ7xY38Z4Unh5nTJMR90l6y7AYTwxg",{"id":72374,"title":72375,"body":72376,"comments":609,"date":74433,"description":72380,"draft":602,"extension":605,"external":606,"image":74424,"meta":74434,"navigation":609,"path":74435,"seo":74436,"stem":74437,"tags":74438,"__hash__":74439},"blog/2017/11/20/using-tensorflow-and-tensor-board-with-docker.md","Using Tensorflow and Tensorboard with Docker",{"type":8,"value":72377,"toc":74431},[72378,72381,72384,72390,72400,72406,72418,72421,72427,72430,74404,74407,74413,74420,74425,74428],[11,72379,72380],{},"In my last article we set up Tensorflow with Docker. Next I want to try to get Tensorboard running.",[11,72382,72383],{},"When we opened the Jupyter notebook, our command included port mapping. Here is that command:",[459,72385,72388],{"className":72386,"code":72387,"language":997},[995],"$ sudo nvidia-docker run -it -p 8888:8888 tensorflow/tensorflow:latest-gpu\n",[30,72389,72387],{"__ignoreMap":464},[11,72391,72392,72393,72396,72397,33226],{},"Tensorboard will be served in our browser on port ",[30,72394,72395],{},"6006",", so we will want to do that port mapping in our ",[30,72398,72399],{},"nvidia-docker",[459,72401,72404],{"className":72402,"code":72403,"language":997},[995],"sudo nvidia-docker run -p 0.0.0.0:6006:6006 -it tensorflow/tensorflow:latest-gpu bash\n",[30,72405,72403],{"__ignoreMap":464},[11,72407,72408,72409,72414,72415,643],{},"I want to run ",[20,72410,72413],{"href":72411,"rel":72412},"https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/tutorials/mnist/mnist_with_summaries.py",[24],"this script"," from the Tensorflow github repo. It is an example of MNIST with summaries. Summaries are logs that are captured from script and they provide the data that runs Tensorboard. In this case they are recorded in ",[30,72416,72417],{},"/tmp/tensorflow/mnist/logs/",[11,72419,72420],{},"To start with this script let's just copy and paste it into a file. We will need to add vim to our docker container for that:",[459,72422,72425],{"className":72423,"code":72424,"language":997},[995],"# apt-get update\n# apt-get install vim\n",[30,72426,72424],{"__ignoreMap":464},[11,72428,72429],{},"Now we can copy and paste the script and run it:",[459,72431,72433],{"className":13136,"code":72432,"language":12886,"meta":464,"style":464},"root@eb9e069064d7:~# python tb.py\nSuccessfully downloaded train-images-idx3-ubyte.gz 9912422 bytes.\nExtracting /tmp/tensorflow/mnist/input_data/train-images-idx3-ubyte.gz\nSuccessfully downloaded train-labels-idx1-ubyte.gz 28881 bytes.\nExtracting /tmp/tensorflow/mnist/input_data/train-labels-idx1-ubyte.gz\nSuccessfully downloaded t10k-images-idx3-ubyte.gz 1648877 bytes.\nExtracting /tmp/tensorflow/mnist/input_data/t10k-images-idx3-ubyte.gz\nSuccessfully downloaded t10k-labels-idx1-ubyte.gz 4542 bytes.\nExtracting /tmp/tensorflow/mnist/input_data/t10k-labels-idx1-ubyte.gz\n2017-11-20 03:52:53.792141: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA\n2017-11-20 03:52:53.878640: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:892] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n2017-11-20 03:52:53.878892: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties:\nname: GeForce GTX 1080 major: 6 minor: 1 memoryClockRate(GHz): 1.7335\npciBusID: 0000:01:00.0\ntotalMemory: 7.92GiB freeMemory: 7.43GiB\n2017-11-20 03:52:53.878904: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GTX 1080, pci bus id: 0000:01:00.0, compute capability: 6.1)\nAccuracy at step 0: 0.1235\nAccuracy at step 10: 0.7297\nAccuracy at step 20: 0.8414\nAccuracy at step 30: 0.8717\nAccuracy at step 40: 0.886\nAccuracy at step 50: 0.896\nAccuracy at step 60: 0.9027\nAccuracy at step 70: 0.9068\nAccuracy at step 80: 0.9101\nAccuracy at step 90: 0.9121\n2017-11-20 03:52:57.583676: I tensorflow/stream_executor/dso_loader.cc:139] successfully opened CUDA library libcupti.so.8.0 locally\nAdding run metadata for 99\nAccuracy at step 100: 0.9124\nAccuracy at step 110: 0.9164\nAccuracy at step 120: 0.9198\nAccuracy at step 130: 0.9205\nAccuracy at step 140: 0.9142\nAccuracy at step 150: 0.9224\nAccuracy at step 160: 0.9294\nAccuracy at step 170: 0.928\nAccuracy at step 180: 0.9312\nAccuracy at step 190: 0.9301\nAdding run metadata for 199\nAccuracy at step 200: 0.9346\nAccuracy at step 210: 0.9381\nAccuracy at step 220: 0.9396\nAccuracy at step 230: 0.9406\nAccuracy at step 240: 0.9273\nAccuracy at step 250: 0.941\nAccuracy at step 260: 0.9369\nAccuracy at step 270: 0.9329\nAccuracy at step 280: 0.9404\nAccuracy at step 290: 0.9444\nAdding run metadata for 299\nAccuracy at step 300: 0.9438\nAccuracy at step 310: 0.9426\nAccuracy at step 320: 0.9462\nAccuracy at step 330: 0.9449\nAccuracy at step 340: 0.9478\nAccuracy at step 350: 0.9458\nAccuracy at step 360: 0.9464\nAccuracy at step 370: 0.9474\nAccuracy at step 380: 0.9528\nAccuracy at step 390: 0.9499\nAdding run metadata for 399\nAccuracy at step 400: 0.9507\nAccuracy at step 410: 0.9501\nAccuracy at step 420: 0.9513\nAccuracy at step 430: 0.9483\nAccuracy at step 440: 0.9518\nAccuracy at step 450: 0.949\nAccuracy at step 460: 0.9543\nAccuracy at step 470: 0.9552\nAccuracy at step 480: 0.9515\nAccuracy at step 490: 0.9544\nAdding run metadata for 499\nAccuracy at step 500: 0.9586\nAccuracy at step 510: 0.9567\nAccuracy at step 520: 0.9572\nAccuracy at step 530: 0.9574\nAccuracy at step 540: 0.9584\nAccuracy at step 550: 0.9593\nAccuracy at step 560: 0.958\nAccuracy at step 570: 0.9575\nAccuracy at step 580: 0.9582\nAccuracy at step 590: 0.9609\nAdding run metadata for 599\nAccuracy at step 600: 0.9618\nAccuracy at step 610: 0.9605\nAccuracy at step 620: 0.9606\nAccuracy at step 630: 0.961\nAccuracy at step 640: 0.963\nAccuracy at step 650: 0.9614\nAccuracy at step 660: 0.9622\nAccuracy at step 670: 0.9634\nAccuracy at step 680: 0.9641\nAccuracy at step 690: 0.9627\nAdding run metadata for 699\nAccuracy at step 700: 0.9623\nAccuracy at step 710: 0.9612\nAccuracy at step 720: 0.9628\nAccuracy at step 730: 0.965\nAccuracy at step 740: 0.9635\nAccuracy at step 750: 0.9635\nAccuracy at step 760: 0.9648\nAccuracy at step 770: 0.9637\nAccuracy at step 780: 0.9658\nAccuracy at step 790: 0.9649\nAdding run metadata for 799\nAccuracy at step 800: 0.9681\nAccuracy at step 810: 0.9661\nAccuracy at step 820: 0.9657\nAccuracy at step 830: 0.9646\nAccuracy at step 840: 0.9647\nAccuracy at step 850: 0.965\nAccuracy at step 860: 0.9677\nAccuracy at step 870: 0.9649\nAccuracy at step 880: 0.9675\nAccuracy at step 890: 0.969\nAdding run metadata for 899\nAccuracy at step 900: 0.9689\nAccuracy at step 910: 0.967\nAccuracy at step 920: 0.9672\nAccuracy at step 930: 0.9645\nAccuracy at step 940: 0.9657\nAccuracy at step 950: 0.9699\nAccuracy at step 960: 0.968\nAccuracy at step 970: 0.9679\nAccuracy at step 980: 0.9651\nAccuracy at step 990: 0.9683\nAdding run metadata for 999\n",[30,72434,72435,72451,72478,72521,72545,72581,72605,72642,72665,72701,72781,72856,72914,72941,72960,72979,73084,73096,73107,73118,73129,73140,73152,73163,73175,73186,73197,73245,73255,73266,73278,73290,73302,73314,73325,73337,73349,73361,73373,73382,73393,73405,73417,73429,73441,73453,73465,73477,73489,73501,73510,73521,73533,73545,73556,73568,73580,73592,73604,73616,73628,73637,73648,73660,73672,73684,73696,73708,73720,73732,73744,73756,73765,73776,73788,73800,73812,73824,73836,73848,73860,73872,73884,73893,73904,73916,73928,73940,73952,73964,73976,73988,74000,74012,74021,74033,74045,74057,74069,74081,74092,74104,74116,74128,74140,74149,74161,74173,74185,74197,74209,74220,74232,74243,74255,74267,74276,74288,74300,74312,74324,74335,74347,74359,74371,74383,74395],{"__ignoreMap":464},[151,72436,72437,72440,72443,72446,72448],{"class":469,"line":470},[151,72438,72439],{"class":503},"root",[151,72441,72442],{"class":1869},"@",[151,72444,72445],{"class":503},"eb9e069064d7:",[151,72447,7879],{"class":1869},[151,72449,72450],{"class":1527},"# python tb.py\n",[151,72452,72453,72456,72458,72460,72462,72465,72467,72470,72473,72476],{"class":469,"line":488},[151,72454,72455],{"class":503},"Successfully downloaded train",[151,72457,12445],{"class":1869},[151,72459,35383],{"class":503},[151,72461,12445],{"class":1869},[151,72463,72464],{"class":503},"idx3",[151,72466,12445],{"class":1869},[151,72468,72469],{"class":503},"ubyte.gz ",[151,72471,72472],{"class":477},"9912422",[151,72474,72475],{"class":6205}," bytes",[151,72477,6565],{"class":503},[151,72479,72480,72483,72485,72488,72490,72493,72495,72498,72500,72503,72505,72508,72510,72512,72514,72516,72518],{"class":469,"line":500},[151,72481,72482],{"class":503},"Extracting ",[151,72484,19883],{"class":1869},[151,72486,72487],{"class":503},"tmp",[151,72489,19883],{"class":1869},[151,72491,72492],{"class":503},"tensorflow",[151,72494,19883],{"class":1869},[151,72496,72497],{"class":503},"mnist",[151,72499,19883],{"class":1869},[151,72501,72502],{"class":503},"input_data",[151,72504,19883],{"class":1869},[151,72506,72507],{"class":503},"train",[151,72509,12445],{"class":1869},[151,72511,35383],{"class":503},[151,72513,12445],{"class":1869},[151,72515,72464],{"class":503},[151,72517,12445],{"class":1869},[151,72519,72520],{"class":503},"ubyte.gz\n",[151,72522,72523,72525,72527,72529,72531,72534,72536,72538,72541,72543],{"class":469,"line":509},[151,72524,72455],{"class":503},[151,72526,12445],{"class":1869},[151,72528,51160],{"class":503},[151,72530,12445],{"class":1869},[151,72532,72533],{"class":503},"idx1",[151,72535,12445],{"class":1869},[151,72537,72469],{"class":503},[151,72539,72540],{"class":477},"28881",[151,72542,72475],{"class":6205},[151,72544,6565],{"class":503},[151,72546,72547,72549,72551,72553,72555,72557,72559,72561,72563,72565,72567,72569,72571,72573,72575,72577,72579],{"class":469,"line":517},[151,72548,72482],{"class":503},[151,72550,19883],{"class":1869},[151,72552,72487],{"class":503},[151,72554,19883],{"class":1869},[151,72556,72492],{"class":503},[151,72558,19883],{"class":1869},[151,72560,72497],{"class":503},[151,72562,19883],{"class":1869},[151,72564,72502],{"class":503},[151,72566,19883],{"class":1869},[151,72568,72507],{"class":503},[151,72570,12445],{"class":1869},[151,72572,51160],{"class":503},[151,72574,12445],{"class":1869},[151,72576,72533],{"class":503},[151,72578,12445],{"class":1869},[151,72580,72520],{"class":503},[151,72582,72583,72586,72588,72590,72592,72594,72596,72598,72601,72603],{"class":469,"line":534},[151,72584,72585],{"class":503},"Successfully downloaded t10k",[151,72587,12445],{"class":1869},[151,72589,35383],{"class":503},[151,72591,12445],{"class":1869},[151,72593,72464],{"class":503},[151,72595,12445],{"class":1869},[151,72597,72469],{"class":503},[151,72599,72600],{"class":477},"1648877",[151,72602,72475],{"class":6205},[151,72604,6565],{"class":503},[151,72606,72607,72609,72611,72613,72615,72617,72619,72621,72623,72625,72627,72630,72632,72634,72636,72638,72640],{"class":469,"line":1413},[151,72608,72482],{"class":503},[151,72610,19883],{"class":1869},[151,72612,72487],{"class":503},[151,72614,19883],{"class":1869},[151,72616,72492],{"class":503},[151,72618,19883],{"class":1869},[151,72620,72497],{"class":503},[151,72622,19883],{"class":1869},[151,72624,72502],{"class":503},[151,72626,19883],{"class":1869},[151,72628,72629],{"class":503},"t10k",[151,72631,12445],{"class":1869},[151,72633,35383],{"class":503},[151,72635,12445],{"class":1869},[151,72637,72464],{"class":503},[151,72639,12445],{"class":1869},[151,72641,72520],{"class":503},[151,72643,72644,72646,72648,72650,72652,72654,72656,72658,72661,72663],{"class":469,"line":1418},[151,72645,72585],{"class":503},[151,72647,12445],{"class":1869},[151,72649,51160],{"class":503},[151,72651,12445],{"class":1869},[151,72653,72533],{"class":503},[151,72655,12445],{"class":1869},[151,72657,72469],{"class":503},[151,72659,72660],{"class":477},"4542",[151,72662,72475],{"class":6205},[151,72664,6565],{"class":503},[151,72666,72667,72669,72671,72673,72675,72677,72679,72681,72683,72685,72687,72689,72691,72693,72695,72697,72699],{"class":469,"line":2462},[151,72668,72482],{"class":503},[151,72670,19883],{"class":1869},[151,72672,72487],{"class":503},[151,72674,19883],{"class":1869},[151,72676,72492],{"class":503},[151,72678,19883],{"class":1869},[151,72680,72497],{"class":503},[151,72682,19883],{"class":1869},[151,72684,72502],{"class":503},[151,72686,19883],{"class":1869},[151,72688,72629],{"class":503},[151,72690,12445],{"class":1869},[151,72692,51160],{"class":503},[151,72694,12445],{"class":1869},[151,72696,72533],{"class":503},[151,72698,12445],{"class":1869},[151,72700,72520],{"class":503},[151,72702,72703,72706,72708,72710,72712,72714,72716,72718,72720,72722,72724,72727,72730,72732,72734,72736,72739,72741,72744,72747,72750,72753,72756,72758,72761,72764,72767,72769,72772,72775,72778],{"class":469,"line":2471},[151,72704,72705],{"class":477},"2017",[151,72707,12445],{"class":1869},[151,72709,42377],{"class":477},[151,72711,12445],{"class":1869},[151,72713,9097],{"class":477},[151,72715,57890],{"class":477},[151,72717,6557],{"class":6607},[151,72719,208],{"class":503},[151,72721,45428],{"class":477},[151,72723,208],{"class":503},[151,72725,72726],{"class":477},"53.792141",[151,72728,72729],{"class":503},": I tensorflow",[151,72731,19883],{"class":1869},[151,72733,70879],{"class":503},[151,72735,19883],{"class":1869},[151,72737,72738],{"class":503},"platform",[151,72740,19883],{"class":1869},[151,72742,72743],{"class":503},"cpu_feature_guard.cc:",[151,72745,72746],{"class":477},"137",[151,72748,72749],{"class":503},"] Your ",[151,72751,72752],{"class":477},"CPU",[151,72754,72755],{"class":503}," supports instructions that this TensorFlow binary was ",[151,72757,241],{"class":1869},[151,72759,72760],{"class":503}," compiled to use: ",[151,72762,72763],{"class":477},"SSE4",[151,72765,72766],{"class":503},".1 ",[151,72768,72763],{"class":477},[151,72770,72771],{"class":503},".2 ",[151,72773,72774],{"class":477},"AVX",[151,72776,72777],{"class":477}," AVX2",[151,72779,72780],{"class":477}," FMA\n",[151,72782,72783,72785,72787,72789,72791,72793,72795,72797,72799,72801,72803,72806,72808,72810,72813,72815,72817,72819,72822,72825,72828,72831,72834,72836,72839,72841,72843,72846,72848,72851,72853],{"class":469,"line":2480},[151,72784,72705],{"class":477},[151,72786,12445],{"class":1869},[151,72788,42377],{"class":477},[151,72790,12445],{"class":1869},[151,72792,9097],{"class":477},[151,72794,57890],{"class":477},[151,72796,6557],{"class":6607},[151,72798,208],{"class":503},[151,72800,45428],{"class":477},[151,72802,208],{"class":503},[151,72804,72805],{"class":477},"53.878640",[151,72807,72729],{"class":503},[151,72809,19883],{"class":1869},[151,72811,72812],{"class":503},"stream_executor",[151,72814,19883],{"class":1869},[151,72816,25737],{"class":503},[151,72818,19883],{"class":1869},[151,72820,72821],{"class":503},"cuda_gpu_executor.cc:",[151,72823,72824],{"class":477},"892",[151,72826,72827],{"class":503},"] successful ",[151,72829,72830],{"class":477},"NUMA",[151,72832,72833],{"class":503}," node read ",[151,72835,16853],{"class":1869},[151,72837,72838],{"class":503}," SysFS had negative value (",[151,72840,12445],{"class":1869},[151,72842,6760],{"class":477},[151,72844,72845],{"class":503},"), but there must be at least one ",[151,72847,72830],{"class":477},[151,72849,72850],{"class":503}," node, so returning ",[151,72852,72830],{"class":477},[151,72854,72855],{"class":503}," node zero\n",[151,72857,72858,72860,72862,72864,72866,72868,72870,72872,72874,72876,72878,72881,72883,72885,72887,72889,72892,72894,72896,72898,72901,72904,72907,72909,72911],{"class":469,"line":2489},[151,72859,72705],{"class":477},[151,72861,12445],{"class":1869},[151,72863,42377],{"class":477},[151,72865,12445],{"class":1869},[151,72867,9097],{"class":477},[151,72869,57890],{"class":477},[151,72871,6557],{"class":6607},[151,72873,208],{"class":503},[151,72875,45428],{"class":477},[151,72877,208],{"class":503},[151,72879,72880],{"class":477},"53.878892",[151,72882,72729],{"class":503},[151,72884,19883],{"class":1869},[151,72886,70879],{"class":503},[151,72888,19883],{"class":1869},[151,72890,72891],{"class":503},"common_runtime",[151,72893,19883],{"class":1869},[151,72895,21573],{"class":503},[151,72897,19883],{"class":1869},[151,72899,72900],{"class":503},"gpu_device.cc:",[151,72902,72903],{"class":477},"1030",[151,72905,72906],{"class":503},"] Found device ",[151,72908,9181],{"class":477},[151,72910,2173],{"class":1869},[151,72912,72913],{"class":503}," properties:\n",[151,72915,72916,72919,72922,72925,72928,72930,72933,72935,72938],{"class":469,"line":2497},[151,72917,72918],{"class":503},"name: GeForce ",[151,72920,72921],{"class":477},"GTX",[151,72923,72924],{"class":477}," 1080",[151,72926,72927],{"class":503}," major: ",[151,72929,25038],{"class":477},[151,72931,72932],{"class":503}," minor: ",[151,72934,6760],{"class":477},[151,72936,72937],{"class":503}," memoryClockRate(GHz): ",[151,72939,72940],{"class":477},"1.7335\n",[151,72942,72943,72946,72949,72951,72953,72955,72957],{"class":469,"line":3140},[151,72944,72945],{"class":503},"pciBusID: ",[151,72947,72948],{"class":477},"0000",[151,72950,208],{"class":503},[151,72952,9181],{"class":477},[151,72954,6760],{"class":6607},[151,72956,208],{"class":503},[151,72958,72959],{"class":477},"00.0\n",[151,72961,72962,72965,72968,72971,72974,72976],{"class":469,"line":3149},[151,72963,72964],{"class":503},"totalMemory: ",[151,72966,72967],{"class":477},"7.",[151,72969,72970],{"class":6607},"92GiB",[151,72972,72973],{"class":503}," freeMemory: ",[151,72975,72967],{"class":477},[151,72977,72978],{"class":6607},"43GiB\n",[151,72980,72981,72983,72985,72987,72989,72991,72993,72995,72997,72999,73001,73004,73006,73008,73010,73012,73014,73016,73018,73020,73022,73025,73028,73030,73033,73036,73038,73040,73042,73044,73047,73049,73052,73054,73056,73059,73061,73063,73065,73067,73069,73071,73073,73076,73079,73082],{"class":469,"line":3158},[151,72982,72705],{"class":477},[151,72984,12445],{"class":1869},[151,72986,42377],{"class":477},[151,72988,12445],{"class":1869},[151,72990,9097],{"class":477},[151,72992,57890],{"class":477},[151,72994,6557],{"class":6607},[151,72996,208],{"class":503},[151,72998,45428],{"class":477},[151,73000,208],{"class":503},[151,73002,73003],{"class":477},"53.878904",[151,73005,72729],{"class":503},[151,73007,19883],{"class":1869},[151,73009,70879],{"class":503},[151,73011,19883],{"class":1869},[151,73013,72891],{"class":503},[151,73015,19883],{"class":1869},[151,73017,21573],{"class":503},[151,73019,19883],{"class":1869},[151,73021,72900],{"class":503},[151,73023,73024],{"class":477},"1120",[151,73026,73027],{"class":503},"] Creating TensorFlow device (",[151,73029,19883],{"class":1869},[151,73031,73032],{"class":503},"device:",[151,73034,73035],{"class":477},"GPU",[151,73037,208],{"class":503},[151,73039,9181],{"class":477},[151,73041,16995],{"class":503},[151,73043,70192],{"class":6607},[151,73045,73046],{"class":503}," (device: ",[151,73048,9181],{"class":477},[151,73050,73051],{"class":503},", name: GeForce ",[151,73053,72921],{"class":477},[151,73055,72924],{"class":477},[151,73057,73058],{"class":503},", pci bus ",[151,73060,47409],{"class":2226},[151,73062,6208],{"class":503},[151,73064,72948],{"class":477},[151,73066,208],{"class":503},[151,73068,9181],{"class":477},[151,73070,6760],{"class":6607},[151,73072,208],{"class":503},[151,73074,73075],{"class":477},"00.0",[151,73077,73078],{"class":503},", compute capability: ",[151,73080,73081],{"class":477},"6.1",[151,73083,3640],{"class":503},[151,73085,73086,73089,73091,73093],{"class":469,"line":3167},[151,73087,73088],{"class":503},"Accuracy at step ",[151,73090,9181],{"class":477},[151,73092,6208],{"class":503},[151,73094,73095],{"class":477},"0.1235\n",[151,73097,73098,73100,73102,73104],{"class":469,"line":3175},[151,73099,73088],{"class":503},[151,73101,12423],{"class":477},[151,73103,6208],{"class":503},[151,73105,73106],{"class":477},"0.7297\n",[151,73108,73109,73111,73113,73115],{"class":469,"line":3184},[151,73110,73088],{"class":503},[151,73112,9097],{"class":477},[151,73114,6208],{"class":503},[151,73116,73117],{"class":477},"0.8414\n",[151,73119,73120,73122,73124,73126],{"class":469,"line":3193},[151,73121,73088],{"class":503},[151,73123,42017],{"class":477},[151,73125,6208],{"class":503},[151,73127,73128],{"class":477},"0.8717\n",[151,73130,73131,73133,73135,73137],{"class":469,"line":3720},[151,73132,73088],{"class":503},[151,73134,44365],{"class":477},[151,73136,6208],{"class":503},[151,73138,73139],{"class":477},"0.886\n",[151,73141,73142,73144,73147,73149],{"class":469,"line":3729},[151,73143,73088],{"class":503},[151,73145,73146],{"class":477},"50",[151,73148,6208],{"class":503},[151,73150,73151],{"class":477},"0.896\n",[151,73153,73154,73156,73158,73160],{"class":469,"line":3735},[151,73155,73088],{"class":503},[151,73157,39825],{"class":477},[151,73159,6208],{"class":503},[151,73161,73162],{"class":477},"0.9027\n",[151,73164,73165,73167,73170,73172],{"class":469,"line":3745},[151,73166,73088],{"class":503},[151,73168,73169],{"class":477},"70",[151,73171,6208],{"class":503},[151,73173,73174],{"class":477},"0.9068\n",[151,73176,73177,73179,73181,73183],{"class":469,"line":3754},[151,73178,73088],{"class":503},[151,73180,27033],{"class":477},[151,73182,6208],{"class":503},[151,73184,73185],{"class":477},"0.9101\n",[151,73187,73188,73190,73192,73194],{"class":469,"line":3760},[151,73189,73088],{"class":503},[151,73191,65941],{"class":477},[151,73193,6208],{"class":503},[151,73195,73196],{"class":477},"0.9121\n",[151,73198,73199,73201,73203,73205,73207,73209,73211,73213,73215,73217,73219,73222,73224,73226,73228,73230,73233,73236,73239,73242],{"class":469,"line":3773},[151,73200,72705],{"class":477},[151,73202,12445],{"class":1869},[151,73204,42377],{"class":477},[151,73206,12445],{"class":1869},[151,73208,9097],{"class":477},[151,73210,57890],{"class":477},[151,73212,6557],{"class":6607},[151,73214,208],{"class":503},[151,73216,45428],{"class":477},[151,73218,208],{"class":503},[151,73220,73221],{"class":477},"57.583676",[151,73223,72729],{"class":503},[151,73225,19883],{"class":1869},[151,73227,72812],{"class":503},[151,73229,19883],{"class":1869},[151,73231,73232],{"class":503},"dso_loader.cc:",[151,73234,73235],{"class":477},"139",[151,73237,73238],{"class":503},"] successfully opened ",[151,73240,73241],{"class":477},"CUDA",[151,73243,73244],{"class":503}," library libcupti.so.8.0 locally\n",[151,73246,73247,73250,73252],{"class":469,"line":3782},[151,73248,73249],{"class":503},"Adding run metadata ",[151,73251,16732],{"class":1869},[151,73253,73254],{"class":477}," 99\n",[151,73256,73257,73259,73261,73263],{"class":469,"line":3791},[151,73258,73088],{"class":503},[151,73260,71821],{"class":477},[151,73262,6208],{"class":503},[151,73264,73265],{"class":477},"0.9124\n",[151,73267,73268,73270,73273,73275],{"class":469,"line":3803},[151,73269,73088],{"class":503},[151,73271,73272],{"class":477},"110",[151,73274,6208],{"class":503},[151,73276,73277],{"class":477},"0.9164\n",[151,73279,73280,73282,73285,73287],{"class":469,"line":3811},[151,73281,73088],{"class":503},[151,73283,73284],{"class":477},"120",[151,73286,6208],{"class":503},[151,73288,73289],{"class":477},"0.9198\n",[151,73291,73292,73294,73297,73299],{"class":469,"line":3820},[151,73293,73088],{"class":503},[151,73295,73296],{"class":477},"130",[151,73298,6208],{"class":503},[151,73300,73301],{"class":477},"0.9205\n",[151,73303,73304,73306,73309,73311],{"class":469,"line":7084},[151,73305,73088],{"class":503},[151,73307,73308],{"class":477},"140",[151,73310,6208],{"class":503},[151,73312,73313],{"class":477},"0.9142\n",[151,73315,73316,73318,73320,73322],{"class":469,"line":7148},[151,73317,73088],{"class":503},[151,73319,45949],{"class":477},[151,73321,6208],{"class":503},[151,73323,73324],{"class":477},"0.9224\n",[151,73326,73327,73329,73332,73334],{"class":469,"line":7211},[151,73328,73088],{"class":503},[151,73330,73331],{"class":477},"160",[151,73333,6208],{"class":503},[151,73335,73336],{"class":477},"0.9294\n",[151,73338,73339,73341,73344,73346],{"class":469,"line":7273},[151,73340,73088],{"class":503},[151,73342,73343],{"class":477},"170",[151,73345,6208],{"class":503},[151,73347,73348],{"class":477},"0.928\n",[151,73350,73351,73353,73356,73358],{"class":469,"line":7335},[151,73352,73088],{"class":503},[151,73354,73355],{"class":477},"180",[151,73357,6208],{"class":503},[151,73359,73360],{"class":477},"0.9312\n",[151,73362,73363,73365,73368,73370],{"class":469,"line":7398},[151,73364,73088],{"class":503},[151,73366,73367],{"class":477},"190",[151,73369,6208],{"class":503},[151,73371,73372],{"class":477},"0.9301\n",[151,73374,73375,73377,73379],{"class":469,"line":7462},[151,73376,73249],{"class":503},[151,73378,16732],{"class":1869},[151,73380,73381],{"class":477}," 199\n",[151,73383,73384,73386,73388,73390],{"class":469,"line":7467},[151,73385,73088],{"class":503},[151,73387,41624],{"class":477},[151,73389,6208],{"class":503},[151,73391,73392],{"class":477},"0.9346\n",[151,73394,73395,73397,73400,73402],{"class":469,"line":7532},[151,73396,73088],{"class":503},[151,73398,73399],{"class":477},"210",[151,73401,6208],{"class":503},[151,73403,73404],{"class":477},"0.9381\n",[151,73406,73407,73409,73412,73414],{"class":469,"line":7537},[151,73408,73088],{"class":503},[151,73410,73411],{"class":477},"220",[151,73413,6208],{"class":503},[151,73415,73416],{"class":477},"0.9396\n",[151,73418,73419,73421,73424,73426],{"class":469,"line":7603},[151,73420,73088],{"class":503},[151,73422,73423],{"class":477},"230",[151,73425,6208],{"class":503},[151,73427,73428],{"class":477},"0.9406\n",[151,73430,73431,73433,73436,73438],{"class":469,"line":7608},[151,73432,73088],{"class":503},[151,73434,73435],{"class":477},"240",[151,73437,6208],{"class":503},[151,73439,73440],{"class":477},"0.9273\n",[151,73442,73443,73445,73448,73450],{"class":469,"line":7673},[151,73444,73088],{"class":503},[151,73446,73447],{"class":477},"250",[151,73449,6208],{"class":503},[151,73451,73452],{"class":477},"0.941\n",[151,73454,73455,73457,73460,73462],{"class":469,"line":7678},[151,73456,73088],{"class":503},[151,73458,73459],{"class":477},"260",[151,73461,6208],{"class":503},[151,73463,73464],{"class":477},"0.9369\n",[151,73466,73467,73469,73472,73474],{"class":469,"line":7708},[151,73468,73088],{"class":503},[151,73470,73471],{"class":477},"270",[151,73473,6208],{"class":503},[151,73475,73476],{"class":477},"0.9329\n",[151,73478,73479,73481,73484,73486],{"class":469,"line":7713},[151,73480,73088],{"class":503},[151,73482,73483],{"class":477},"280",[151,73485,6208],{"class":503},[151,73487,73488],{"class":477},"0.9404\n",[151,73490,73491,73493,73496,73498],{"class":469,"line":7746},[151,73492,73088],{"class":503},[151,73494,73495],{"class":477},"290",[151,73497,6208],{"class":503},[151,73499,73500],{"class":477},"0.9444\n",[151,73502,73503,73505,73507],{"class":469,"line":7751},[151,73504,73249],{"class":503},[151,73506,16732],{"class":1869},[151,73508,73509],{"class":477}," 299\n",[151,73511,73512,73514,73516,73518],{"class":469,"line":7816},[151,73513,73088],{"class":503},[151,73515,59584],{"class":477},[151,73517,6208],{"class":503},[151,73519,73520],{"class":477},"0.9438\n",[151,73522,73523,73525,73528,73530],{"class":469,"line":7821},[151,73524,73088],{"class":503},[151,73526,73527],{"class":477},"310",[151,73529,6208],{"class":503},[151,73531,73532],{"class":477},"0.9426\n",[151,73534,73535,73537,73540,73542],{"class":469,"line":7847},[151,73536,73088],{"class":503},[151,73538,73539],{"class":477},"320",[151,73541,6208],{"class":503},[151,73543,73544],{"class":477},"0.9462\n",[151,73546,73547,73549,73551,73553],{"class":469,"line":7852},[151,73548,73088],{"class":503},[151,73550,41573],{"class":477},[151,73552,6208],{"class":503},[151,73554,73555],{"class":477},"0.9449\n",[151,73557,73558,73560,73563,73565],{"class":469,"line":7887},[151,73559,73088],{"class":503},[151,73561,73562],{"class":477},"340",[151,73564,6208],{"class":503},[151,73566,73567],{"class":477},"0.9478\n",[151,73569,73570,73572,73575,73577],{"class":469,"line":7892},[151,73571,73088],{"class":503},[151,73573,73574],{"class":477},"350",[151,73576,6208],{"class":503},[151,73578,73579],{"class":477},"0.9458\n",[151,73581,73582,73584,73587,73589],{"class":469,"line":7924},[151,73583,73088],{"class":503},[151,73585,73586],{"class":477},"360",[151,73588,6208],{"class":503},[151,73590,73591],{"class":477},"0.9464\n",[151,73593,73594,73596,73599,73601],{"class":469,"line":7929},[151,73595,73088],{"class":503},[151,73597,73598],{"class":477},"370",[151,73600,6208],{"class":503},[151,73602,73603],{"class":477},"0.9474\n",[151,73605,73606,73608,73611,73613],{"class":469,"line":7991},[151,73607,73088],{"class":503},[151,73609,73610],{"class":477},"380",[151,73612,6208],{"class":503},[151,73614,73615],{"class":477},"0.9528\n",[151,73617,73618,73620,73623,73625],{"class":469,"line":7996},[151,73619,73088],{"class":503},[151,73621,73622],{"class":477},"390",[151,73624,6208],{"class":503},[151,73626,73627],{"class":477},"0.9499\n",[151,73629,73630,73632,73634],{"class":469,"line":8078},[151,73631,73249],{"class":503},[151,73633,16732],{"class":1869},[151,73635,73636],{"class":477}," 399\n",[151,73638,73639,73641,73643,73645],{"class":469,"line":8140},[151,73640,73088],{"class":503},[151,73642,71554],{"class":477},[151,73644,6208],{"class":503},[151,73646,73647],{"class":477},"0.9507\n",[151,73649,73650,73652,73655,73657],{"class":469,"line":8145},[151,73651,73088],{"class":503},[151,73653,73654],{"class":477},"410",[151,73656,6208],{"class":503},[151,73658,73659],{"class":477},"0.9501\n",[151,73661,73662,73664,73667,73669],{"class":469,"line":8259},[151,73663,73088],{"class":503},[151,73665,73666],{"class":477},"420",[151,73668,6208],{"class":503},[151,73670,73671],{"class":477},"0.9513\n",[151,73673,73674,73676,73679,73681],{"class":469,"line":8264},[151,73675,73088],{"class":503},[151,73677,73678],{"class":477},"430",[151,73680,6208],{"class":503},[151,73682,73683],{"class":477},"0.9483\n",[151,73685,73686,73688,73691,73693],{"class":469,"line":8613},[151,73687,73088],{"class":503},[151,73689,73690],{"class":477},"440",[151,73692,6208],{"class":503},[151,73694,73695],{"class":477},"0.9518\n",[151,73697,73698,73700,73703,73705],{"class":469,"line":8678},[151,73699,73088],{"class":503},[151,73701,73702],{"class":477},"450",[151,73704,6208],{"class":503},[151,73706,73707],{"class":477},"0.949\n",[151,73709,73710,73712,73715,73717],{"class":469,"line":8742},[151,73711,73088],{"class":503},[151,73713,73714],{"class":477},"460",[151,73716,6208],{"class":503},[151,73718,73719],{"class":477},"0.9543\n",[151,73721,73722,73724,73727,73729],{"class":469,"line":8806},[151,73723,73088],{"class":503},[151,73725,73726],{"class":477},"470",[151,73728,6208],{"class":503},[151,73730,73731],{"class":477},"0.9552\n",[151,73733,73734,73736,73739,73741],{"class":469,"line":8870},[151,73735,73088],{"class":503},[151,73737,73738],{"class":477},"480",[151,73740,6208],{"class":503},[151,73742,73743],{"class":477},"0.9515\n",[151,73745,73746,73748,73751,73753],{"class":469,"line":8875},[151,73747,73088],{"class":503},[151,73749,73750],{"class":477},"490",[151,73752,6208],{"class":503},[151,73754,73755],{"class":477},"0.9544\n",[151,73757,73758,73760,73762],{"class":469,"line":8881},[151,73759,73249],{"class":503},[151,73761,16732],{"class":1869},[151,73763,73764],{"class":477}," 499\n",[151,73766,73767,73769,73771,73773],{"class":469,"line":8886},[151,73768,73088],{"class":503},[151,73770,12208],{"class":477},[151,73772,6208],{"class":503},[151,73774,73775],{"class":477},"0.9586\n",[151,73777,73778,73780,73783,73785],{"class":469,"line":8892},[151,73779,73088],{"class":503},[151,73781,73782],{"class":477},"510",[151,73784,6208],{"class":503},[151,73786,73787],{"class":477},"0.9567\n",[151,73789,73790,73792,73795,73797],{"class":469,"line":8963},[151,73791,73088],{"class":503},[151,73793,73794],{"class":477},"520",[151,73796,6208],{"class":503},[151,73798,73799],{"class":477},"0.9572\n",[151,73801,73802,73804,73807,73809],{"class":469,"line":8969},[151,73803,73088],{"class":503},[151,73805,73806],{"class":477},"530",[151,73808,6208],{"class":503},[151,73810,73811],{"class":477},"0.9574\n",[151,73813,73814,73816,73819,73821],{"class":469,"line":15001},[151,73815,73088],{"class":503},[151,73817,73818],{"class":477},"540",[151,73820,6208],{"class":503},[151,73822,73823],{"class":477},"0.9584\n",[151,73825,73826,73828,73831,73833],{"class":469,"line":15009},[151,73827,73088],{"class":503},[151,73829,73830],{"class":477},"550",[151,73832,6208],{"class":503},[151,73834,73835],{"class":477},"0.9593\n",[151,73837,73838,73840,73843,73845],{"class":469,"line":15019},[151,73839,73088],{"class":503},[151,73841,73842],{"class":477},"560",[151,73844,6208],{"class":503},[151,73846,73847],{"class":477},"0.958\n",[151,73849,73850,73852,73855,73857],{"class":469,"line":15027},[151,73851,73088],{"class":503},[151,73853,73854],{"class":477},"570",[151,73856,6208],{"class":503},[151,73858,73859],{"class":477},"0.9575\n",[151,73861,73862,73864,73867,73869],{"class":469,"line":15037},[151,73863,73088],{"class":503},[151,73865,73866],{"class":477},"580",[151,73868,6208],{"class":503},[151,73870,73871],{"class":477},"0.9582\n",[151,73873,73874,73876,73879,73881],{"class":469,"line":15045},[151,73875,73088],{"class":503},[151,73877,73878],{"class":477},"590",[151,73880,6208],{"class":503},[151,73882,73883],{"class":477},"0.9609\n",[151,73885,73886,73888,73890],{"class":469,"line":15055},[151,73887,73249],{"class":503},[151,73889,16732],{"class":1869},[151,73891,73892],{"class":477}," 599\n",[151,73894,73895,73897,73899,73901],{"class":469,"line":15060},[151,73896,73088],{"class":503},[151,73898,44836],{"class":477},[151,73900,6208],{"class":503},[151,73902,73903],{"class":477},"0.9618\n",[151,73905,73906,73908,73911,73913],{"class":469,"line":15068},[151,73907,73088],{"class":503},[151,73909,73910],{"class":477},"610",[151,73912,6208],{"class":503},[151,73914,73915],{"class":477},"0.9605\n",[151,73917,73918,73920,73923,73925],{"class":469,"line":15076},[151,73919,73088],{"class":503},[151,73921,73922],{"class":477},"620",[151,73924,6208],{"class":503},[151,73926,73927],{"class":477},"0.9606\n",[151,73929,73930,73932,73935,73937],{"class":469,"line":15085},[151,73931,73088],{"class":503},[151,73933,73934],{"class":477},"630",[151,73936,6208],{"class":503},[151,73938,73939],{"class":477},"0.961\n",[151,73941,73942,73944,73947,73949],{"class":469,"line":15095},[151,73943,73088],{"class":503},[151,73945,73946],{"class":477},"640",[151,73948,6208],{"class":503},[151,73950,73951],{"class":477},"0.963\n",[151,73953,73954,73956,73959,73961],{"class":469,"line":15105},[151,73955,73088],{"class":503},[151,73957,73958],{"class":477},"650",[151,73960,6208],{"class":503},[151,73962,73963],{"class":477},"0.9614\n",[151,73965,73966,73968,73971,73973],{"class":469,"line":15110},[151,73967,73088],{"class":503},[151,73969,73970],{"class":477},"660",[151,73972,6208],{"class":503},[151,73974,73975],{"class":477},"0.9622\n",[151,73977,73978,73980,73983,73985],{"class":469,"line":15118},[151,73979,73088],{"class":503},[151,73981,73982],{"class":477},"670",[151,73984,6208],{"class":503},[151,73986,73987],{"class":477},"0.9634\n",[151,73989,73990,73992,73995,73997],{"class":469,"line":15128},[151,73991,73088],{"class":503},[151,73993,73994],{"class":477},"680",[151,73996,6208],{"class":503},[151,73998,73999],{"class":477},"0.9641\n",[151,74001,74002,74004,74007,74009],{"class":469,"line":15139},[151,74003,73088],{"class":503},[151,74005,74006],{"class":477},"690",[151,74008,6208],{"class":503},[151,74010,74011],{"class":477},"0.9627\n",[151,74013,74014,74016,74018],{"class":469,"line":31954},[151,74015,73249],{"class":503},[151,74017,16732],{"class":1869},[151,74019,74020],{"class":477}," 699\n",[151,74022,74023,74025,74028,74030],{"class":469,"line":31960},[151,74024,73088],{"class":503},[151,74026,74027],{"class":477},"700",[151,74029,6208],{"class":503},[151,74031,74032],{"class":477},"0.9623\n",[151,74034,74035,74037,74040,74042],{"class":469,"line":31965},[151,74036,73088],{"class":503},[151,74038,74039],{"class":477},"710",[151,74041,6208],{"class":503},[151,74043,74044],{"class":477},"0.9612\n",[151,74046,74047,74049,74052,74054],{"class":469,"line":31971},[151,74048,73088],{"class":503},[151,74050,74051],{"class":477},"720",[151,74053,6208],{"class":503},[151,74055,74056],{"class":477},"0.9628\n",[151,74058,74059,74061,74064,74066],{"class":469,"line":31983},[151,74060,73088],{"class":503},[151,74062,74063],{"class":477},"730",[151,74065,6208],{"class":503},[151,74067,74068],{"class":477},"0.965\n",[151,74070,74071,74073,74076,74078],{"class":469,"line":31994},[151,74072,73088],{"class":503},[151,74074,74075],{"class":477},"740",[151,74077,6208],{"class":503},[151,74079,74080],{"class":477},"0.9635\n",[151,74082,74083,74085,74088,74090],{"class":469,"line":32007},[151,74084,73088],{"class":503},[151,74086,74087],{"class":477},"750",[151,74089,6208],{"class":503},[151,74091,74080],{"class":477},[151,74093,74094,74096,74099,74101],{"class":469,"line":32018},[151,74095,73088],{"class":503},[151,74097,74098],{"class":477},"760",[151,74100,6208],{"class":503},[151,74102,74103],{"class":477},"0.9648\n",[151,74105,74106,74108,74111,74113],{"class":469,"line":32026},[151,74107,73088],{"class":503},[151,74109,74110],{"class":477},"770",[151,74112,6208],{"class":503},[151,74114,74115],{"class":477},"0.9637\n",[151,74117,74118,74120,74123,74125],{"class":469,"line":32031},[151,74119,73088],{"class":503},[151,74121,74122],{"class":477},"780",[151,74124,6208],{"class":503},[151,74126,74127],{"class":477},"0.9658\n",[151,74129,74130,74132,74135,74137],{"class":469,"line":32036},[151,74131,73088],{"class":503},[151,74133,74134],{"class":477},"790",[151,74136,6208],{"class":503},[151,74138,74139],{"class":477},"0.9649\n",[151,74141,74142,74144,74146],{"class":469,"line":32042},[151,74143,73249],{"class":503},[151,74145,16732],{"class":1869},[151,74147,74148],{"class":477}," 799\n",[151,74150,74151,74153,74156,74158],{"class":469,"line":32054},[151,74152,73088],{"class":503},[151,74154,74155],{"class":477},"800",[151,74157,6208],{"class":503},[151,74159,74160],{"class":477},"0.9681\n",[151,74162,74163,74165,74168,74170],{"class":469,"line":32067},[151,74164,73088],{"class":503},[151,74166,74167],{"class":477},"810",[151,74169,6208],{"class":503},[151,74171,74172],{"class":477},"0.9661\n",[151,74174,74175,74177,74180,74182],{"class":469,"line":32086},[151,74176,73088],{"class":503},[151,74178,74179],{"class":477},"820",[151,74181,6208],{"class":503},[151,74183,74184],{"class":477},"0.9657\n",[151,74186,74187,74189,74192,74194],{"class":469,"line":32097},[151,74188,73088],{"class":503},[151,74190,74191],{"class":477},"830",[151,74193,6208],{"class":503},[151,74195,74196],{"class":477},"0.9646\n",[151,74198,74199,74201,74204,74206],{"class":469,"line":25585},[151,74200,73088],{"class":503},[151,74202,74203],{"class":477},"840",[151,74205,6208],{"class":503},[151,74207,74208],{"class":477},"0.9647\n",[151,74210,74211,74213,74216,74218],{"class":469,"line":32112},[151,74212,73088],{"class":503},[151,74214,74215],{"class":477},"850",[151,74217,6208],{"class":503},[151,74219,74068],{"class":477},[151,74221,74222,74224,74227,74229],{"class":469,"line":32117},[151,74223,73088],{"class":503},[151,74225,74226],{"class":477},"860",[151,74228,6208],{"class":503},[151,74230,74231],{"class":477},"0.9677\n",[151,74233,74234,74236,74239,74241],{"class":469,"line":32123},[151,74235,73088],{"class":503},[151,74237,74238],{"class":477},"870",[151,74240,6208],{"class":503},[151,74242,74139],{"class":477},[151,74244,74245,74247,74250,74252],{"class":469,"line":32151},[151,74246,73088],{"class":503},[151,74248,74249],{"class":477},"880",[151,74251,6208],{"class":503},[151,74253,74254],{"class":477},"0.9675\n",[151,74256,74257,74259,74262,74264],{"class":469,"line":32156},[151,74258,73088],{"class":503},[151,74260,74261],{"class":477},"890",[151,74263,6208],{"class":503},[151,74265,74266],{"class":477},"0.969\n",[151,74268,74269,74271,74273],{"class":469,"line":32162},[151,74270,73249],{"class":503},[151,74272,16732],{"class":1869},[151,74274,74275],{"class":477}," 899\n",[151,74277,74278,74280,74283,74285],{"class":469,"line":32168},[151,74279,73088],{"class":503},[151,74281,74282],{"class":477},"900",[151,74284,6208],{"class":503},[151,74286,74287],{"class":477},"0.9689\n",[151,74289,74290,74292,74295,74297],{"class":469,"line":32180},[151,74291,73088],{"class":503},[151,74293,74294],{"class":477},"910",[151,74296,6208],{"class":503},[151,74298,74299],{"class":477},"0.967\n",[151,74301,74302,74304,74307,74309],{"class":469,"line":32192},[151,74303,73088],{"class":503},[151,74305,74306],{"class":477},"920",[151,74308,6208],{"class":503},[151,74310,74311],{"class":477},"0.9672\n",[151,74313,74314,74316,74319,74321],{"class":469,"line":32207},[151,74315,73088],{"class":503},[151,74317,74318],{"class":477},"930",[151,74320,6208],{"class":503},[151,74322,74323],{"class":477},"0.9645\n",[151,74325,74326,74328,74331,74333],{"class":469,"line":32217},[151,74327,73088],{"class":503},[151,74329,74330],{"class":477},"940",[151,74332,6208],{"class":503},[151,74334,74184],{"class":477},[151,74336,74337,74339,74342,74344],{"class":469,"line":32226},[151,74338,73088],{"class":503},[151,74340,74341],{"class":477},"950",[151,74343,6208],{"class":503},[151,74345,74346],{"class":477},"0.9699\n",[151,74348,74349,74351,74354,74356],{"class":469,"line":32231},[151,74350,73088],{"class":503},[151,74352,74353],{"class":477},"960",[151,74355,6208],{"class":503},[151,74357,74358],{"class":477},"0.968\n",[151,74360,74361,74363,74366,74368],{"class":469,"line":32236},[151,74362,73088],{"class":503},[151,74364,74365],{"class":477},"970",[151,74367,6208],{"class":503},[151,74369,74370],{"class":477},"0.9679\n",[151,74372,74373,74375,74378,74380],{"class":469,"line":32244},[151,74374,73088],{"class":503},[151,74376,74377],{"class":477},"980",[151,74379,6208],{"class":503},[151,74381,74382],{"class":477},"0.9651\n",[151,74384,74385,74387,74390,74392],{"class":469,"line":32249},[151,74386,73088],{"class":503},[151,74388,74389],{"class":477},"990",[151,74391,6208],{"class":503},[151,74393,74394],{"class":477},"0.9683\n",[151,74396,74397,74399,74401],{"class":469,"line":32255},[151,74398,73249],{"class":503},[151,74400,16732],{"class":1869},[151,74402,74403],{"class":477}," 999\n",[11,74405,74406],{},"The script completed successfully! Now we can can take a look at what happened during the training. Launch Tensorboard with the following command:",[459,74408,74411],{"className":74409,"code":74410,"language":997},[995],"root@eb9e069064d7:~# tensorboard --logdir=/tmp/tensorflow/mnist/logs/\nTensorBoard 0.4.0rc2 at http://eb9e069064d7:6006 (Press CTRL+C to quit)\n",[30,74412,74410],{"__ignoreMap":464},[11,74414,74415,74416,74419],{},"Now we can simply navigate to ",[30,74417,74418],{},"localhost:6006"," in our browser to start using Tensorboard. Here's a screenshot of Tensorboard showing accuracy:",[11,74421,74422],{},[2718,74423],{"alt":20386,"src":74424},"/static/tf.png",[11,74426,74427],{},"This wasn't too bad. The MNIST example included a very nice script with everything set up properly. My next big challenge is to implement some type of learning model with a data set of my own and visualize it with TensorBoard, but I'll have to go through several examples before then.",[589,74429,74430],{},"html pre.shiki code .sMOD_, html code.shiki .sMOD_{--shiki-default:#24292E;--shiki-dark:#E1E4E8;--shiki-sepia:#F8F8F2}html pre.shiki code .sC2Qs, html code.shiki .sC2Qs{--shiki-default:#D73A49;--shiki-dark:#F97583;--shiki-sepia:#F92672}html pre.shiki code .s8-w5, html code.shiki .s8-w5{--shiki-default:#6A737D;--shiki-dark:#6A737D;--shiki-sepia:#88846F}html pre.shiki code .s7F3e, html code.shiki .s7F3e{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#AE81FF}html pre.shiki code .s-m8C, html code.shiki .s-m8C{--shiki-default:#005CC5;--shiki-default-font-style:inherit;--shiki-dark:#79B8FF;--shiki-dark-font-style:inherit;--shiki-sepia:#66D9EF;--shiki-sepia-font-style:italic}html pre.shiki code .st05x, html code.shiki .st05x{--shiki-default:#B31D28;--shiki-default-font-style:italic;--shiki-dark:#FDAEB7;--shiki-dark-font-style:italic;--shiki-sepia:#F44747;--shiki-sepia-font-style:inherit}html pre.shiki code .sTrkL, html code.shiki .sTrkL{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#66D9EF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html .sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html.sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}",{"title":464,"searchDepth":488,"depth":488,"links":74432},[],"2017-11-20",{"layout":48045},"/2017/11/20/using-tensorflow-and-tensor-board-with-docker",{"title":72375,"description":72380},"2017/11/20/using-tensorflow-and-tensor-board-with-docker",[72492,30129],"R44VFya4N8HAuMcwfUzeye9Qnfm-tkrE8zBLy-zeYX0",{"id":74441,"title":74442,"body":74443,"comments":609,"date":74433,"description":74505,"draft":602,"extension":605,"external":606,"image":46120,"meta":74506,"navigation":609,"path":74507,"seo":74508,"stem":74509,"tags":74510,"__hash__":74512},"blog/2017/11/24/how-to-get-color-emoji-in-arch-linux.md","How to enable color emoji on Arch Linux with Emoji One Font",{"type":8,"value":74444,"toc":74503},[74445,74454,74461,74468,74471,74477,74480,74500],[11,74446,74447,74448,74453],{},"This is a short article about how to enable color emoji on Arch Linux. I have searched for a working solution for this a few times but never found something that worked. I stumbled upon ",[20,74449,74452],{"href":74450,"rel":74451},"https://gist.github.com/himalay/5c404a5f6653cb35154ceb3a6c606211",[24],"this Gist"," that I used to hack together a solution that enables color emoji from Emoji One on Arch Linux.",[11,74455,74456,74457,74460],{},"This setup doesn't show all emoji in ",[30,74458,74459],{},"urxvt",", and I have heard that this is simply not possible, but most other programs and UIs display the emoji just fine. Here's what I did:",[11,74462,74463,74464,74467],{},"First, install ",[30,74465,74466],{},"tff-emojione",", a package that seems to have been added to the AUR just a week ago.",[11,74469,74470],{},"Next, run the script from the gist mentioned above:",[459,74472,74475],{"className":74473,"code":74474,"language":997},[995],"# create folders if does not exist\nmkdir -p ~/.fonts\nmkdir -p ~/.config/fontconfig/\n\n# download noto color emoji font from https://www.google.com/get/noto/#emoji-zsye-color\n# extract NotoColorEmoji.ttf file into ~/.fonts/\n\n# create font config file\ncat \u003C\u003C 'EOF' > ~/.config/fontconfig/fonts.conf\n\u003C?xml version=\"1.0\" encoding=\"UTF-8\"?>\u003C!DOCTYPE fontconfig SYSTEM \"fonts.dtd\">\n\u003Cmatch>\n \u003Ctest name=\"family\">\u003Cstring>sans-serif\u003C/string>\u003C/test>\n \u003Cedit name=\"family\" mode=\"prepend\" binding=\"strong\">\n \u003Cstring>Noto Color Emoji\u003C/string>\n \u003C/edit>\n \u003C/match>\n\u003Cmatch>\n \u003Ctest name=\"family\">\u003Cstring>serif\u003C/string>\u003C/test>\n \u003Cedit name=\"family\" mode=\"prepend\" binding=\"strong\">\n \u003Cstring>Noto Color Emoji\u003C/string>\n \u003C/edit>\n \u003C/match>\n\u003Cmatch>\n \u003Ctest name=\"family\">\u003Cstring>Apple Color Emoji\u003C/string>\u003C/test>\n \u003Cedit name=\"family\" mode=\"prepend\" binding=\"strong\">\n \u003Cstring>Noto Color Emoji\u003C/string>\n \u003C/edit>\n \u003C/match>\nEOF\n# build font information cache files\nfc-cache -f -v\n",[30,74476,74474],{"__ignoreMap":464},[11,74478,74479],{},"As recommended in the comments of the gist, you will notice that numbers in most applications are represented with emoji numbers.",[11,74481,74482,74483,52562,74486,74489,74490,74493,74494,74496,74497,74499],{},"To fix this, remove all three instances of ",[30,74484,74485],{},"mode=\"prepend\" binding=\"strong\"",[30,74487,74488],{},"~/.config/fontconfig/fonts.conf"," and then run ",[30,74491,74492],{},"fc-cache -f -v ",". This should fix some issues, but you may notice that spaces between words are not displayed in some applications and some of the instances of numbers displayed as emoji should be have fixed, but not all. Once I removed all of the text from ",[30,74495,74488],{}," and ran ",[30,74498,74492],{}," one more time, I seemed to get the space and number issues to go away while the emoji still work!",[11,74501,74502],{},"I'm really not sure how or why this works, but it has solved the issue I've been having of emojis not displaying. Hopefully this helps if you are trying to add color emoji to your Arch Linux installation. Apparently the next version of Ubuntu will include support for color emoji out of the box, but for now you will have to hack together your own solution for Arch Linux.",{"title":464,"searchDepth":488,"depth":488,"links":74504},[],"This is a short article about how to enable color emoji on Arch Linux. I have searched for a working solution for this a few times but never found something that worked. I stumbled upon this Gist that I used to hack together a solution that enables color emoji from Emoji One on Arch Linux.",{"layout":48045},"/2017/11/24/how-to-get-color-emoji-in-arch-linux",{"title":74442,"description":74505},"2017/11/24/how-to-get-color-emoji-in-arch-linux",[72181,74511],"emoji","WeGruEu1CuX5gLYYbZyzgtW37DWVTYo8LW2xL71jp9k",{"id":74514,"title":74515,"body":74516,"comments":609,"date":74895,"description":74896,"draft":602,"extension":605,"external":606,"image":74877,"meta":74897,"navigation":609,"path":74898,"seo":74899,"stem":74900,"tags":74901,"__hash__":74902},"blog/2017/11/19/tensorflow-gpu-setup-with-docker-on-arch-linux.md","Installing the GPU version of Tensorflow with Docker on Arch Linux",{"type":8,"value":74517,"toc":74889},[74518,74521,74535,74539,74542,74546,74553,74556,74562,74565,74571,74574,74580,74594,74600,74603,74606,74612,74616,74625,74628,74633,74639,74646,74653,74659,74662,74668,74671,74677,74682,74688,74691,74697,74700,74706,74709,74715,74718,74805,74816,74826,74829,74835,74838,74844,74850,74853,74856,74862,74865,74873,74878,74881,74886],[11,74519,74520],{},"I've tried installing the GPU version of Tensorflow a few times before and failed. There seems to be lots of confusion about the build process, of which there are many. Also, over the last few years there have been many new versions of the software needed to support the GPU version of Tensorflow as well as the first official release of Tensorflow itself (which is now on version 1.4), such as CUDA and cudnn, and different version of python. This is one more attempt at installing the GPU version of Tensor Flow on my Desktop PC that is currently dual booting with Arch Linux and Windows 10. I've decided to try going the docker route because it should eliminate some of the headache of missing depedencies. Here are the specs for my computer:",[76,74522,74523,74526,74529,74532],{},[79,74524,74525],{},"i7-6700K",[79,74527,74528],{},"NVIDIA GTX 1080",[79,74530,74531],{},"Asus Hero VIII motherboard",[79,74533,74534],{},"Arch Linux on a 128 GB SSD (Windows 10 is installed on a separate SSD)",[56,74536,74538],{"id":74537},"installing-cuda-and-cudnn","Installing CUDA and cudnn",[11,74540,74541],{},"We don't need to install these when installing Tensorflow with Docker. Read to the bottom for more info.",[56,74543,74545],{"id":74544},"installing-docker","Installing Docker",[11,74547,74548,74549,643],{},"To install docker on our machine, let's start with the ",[20,74550,74552],{"href":72337,"rel":74551},[24],"Arch Wiki article on docker",[11,74554,74555],{},"We need to add the Loopback module to the Linux Kernel, so we run:",[459,74557,74560],{"className":74558,"code":74559,"language":997},[995],"# tee /etc/modules-load.d/loop.conf \u003C\u003C\u003C \"loop\"\n# modprobe loop\n$ reboot\n",[30,74561,74559],{"__ignoreMap":464},[11,74563,74564],{},"Ater rebooting we can install docker:",[459,74566,74569],{"className":74567,"code":74568,"language":997},[995],"yaourt -S docker\n",[30,74570,74568],{"__ignoreMap":464},[11,74572,74573],{},"Now we want to add ourself to the docker group with the following command:",[459,74575,74578],{"className":74576,"code":74577,"language":997},[995],"$ sudo gpasswd -a brian docker\n[sudo] password for brian:\nAdding user brian to group docker\n",[30,74579,74577],{"__ignoreMap":464},[11,74581,74582,74583,74586,74587,74590,74591,74593],{},"If you run ",[30,74584,74585],{},"groups",", you won't see docker listed in the groups you (brian) belong to. Run ",[30,74588,74589],{},"newgrp docker"," and then re-run docker and you should see ",[30,74592,30129],{}," listed with any other groups you belong to:",[459,74595,74598],{"className":74596,"code":74597,"language":997},[995],"[brian@a1arch ~]$ groups\nwheel storage power users\n[brian@a1arch ~]$ newgrp docker\n                   -`                    brian@a1arch\n                  .o+`                   ------------\n                 `ooo/                   OS: Arch Linux x86_64\n                `+oooo:                  Kernel: 4.12.8-2-ARCH\n               `+oooooo:                 Uptime: 6 mins\n               -+oooooo+:                Packages: 1127\n             `/:-:++oooo+:               Shell: bash 4.4.12\n            `/++++/+++++++:              Resolution: 1920x1080\n           `/++++++++++++++:             WM: i3\n          `/+++ooooooooooooo/`           Theme: Adwaita [GTK2]\n         ./ooosssso++osssssso+`          Icons: Adwaita [GTK2]\n        .oossssso-````/ossssss+`         Terminal: urxvt\n       -osssssso.      :ssssssso.        Terminal Font: Inconsolata-12\n      :osssssss/        osssso+++.       CPU: Intel i7-6700K (8) @ 4.200GHz\n     /ossssssss/        +ssssooo/-       GPU: NVIDIA GeForce GTX 1080\n   `/ossssso+/:-        -:/+osssso+-     Memory: 3289MiB / 15975MiB\n  `+sso+:-`                 `.-/+oso:\n `++:.                           `-/+/\n .`                                 `/\n\n[brian@a1arch ~]$ groups\ndocker wheel storage power users\n",[30,74599,74597],{"__ignoreMap":464},[11,74601,74602],{},"Doing this prevents us from having to write sudo each time we run docker.",[11,74604,74605],{},"Next we need to start the docker daemon.",[459,74607,74610],{"className":74608,"code":74609,"language":997},[995],"$ systemctl start docker\n==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ====\nAuthentication is required to start 'docker.service'.\nAuthenticating as: brian\nPassword:\n==== AUTHENTICATION COMPLETE ====\n$\n",[30,74611,74609],{"__ignoreMap":464},[736,74613,74615],{"id":74614},"side-note","Side note",[11,74617,74618,74619,74624],{},"There seems to be an ",[20,74620,74623],{"href":74621,"rel":74622},"https://github.com/moby/moby/issues/23289",[24],"Arch Linux-specific bug"," which prevents us from enabling docker (and nvidia-docker which we will get next). There is a solution to downgrade to an older version of docker, or you can just start the docker service and the nvidia-docker service when you want to use them. I have found it faster to first start nvidia-docker and then start docker services.",[11,74626,74627],{},"So far so good. Next let's look at the Tensorflow documentation for installing Tensorflow with docker.",[11,74629,74630,74631,208],{},"We need to install ",[30,74632,72399],{},[459,74634,74637],{"className":74635,"code":74636,"language":997},[995],"$ yaourt -S nvidia-docker\n[...]\n[sudo] password for brian:\nloading packages...\nresolving dependencies...\nlooking for conflicting packages...\n\nPackages (1) nvidia-docker-1.0.1-1\n\nTotal Installed Size:  13.34 MiB\n\n:: Proceed with installation? [Y/n]\n(1/1) checking keys in keyring                                 [##################################] 100%\n(1/1) checking package integrity                               [##################################] 100%\n(1/1) loading package files                                    [##################################] 100%\n(1/1) checking for file conflicts                              [##################################] 100%\n(1/1) checking available disk space                            [##################################] 100%\n:: Processing package changes...\n(1/1) installing nvidia-docker                                 [##################################] 100%\n=> Prior to running 'CUDA'-containers, ensure that the nvidia-docker-plugin\n   is loaded. -> https://github.com/NVIDIA/nvidia-docker#other-distributions\n\n*) manually; sudo -b nohup nvidia-docker-plugin > /tmp/nvidia-docker.log\n\n*) automatically at startup; systemctl enable nvidia-docker.service\nOptional dependencies for nvidia-docker\n    cuda [installed]\n    nvidia [installed]\n    opencl-nvidia [installed]\n:: Running post-transaction hooks...\n(1/1) Arming ConditionNeedsUpdate...\n",[30,74638,74636],{"__ignoreMap":464},[11,74640,74641,74642,643],{},"Next it says: Launch a Docker container that contains one of the TensorFlow binary images. Those images are available ",[20,74643,13074],{"href":74644,"rel":74645},"https://hub.docker.com/r/tensorflow/tensorflow/tags/",[24],[11,74647,74648,74649,74652],{},"Next I pulled the container with the ",[30,74650,74651],{},"gpu-latest"," tag and it started to download the container:",[459,74654,74657],{"className":74655,"code":74656,"language":997},[995],"$ docker pull tensorflow/tensorflow:gpu-latest\n[sudo] password for brian:\nlatest-gpu: Pulling from tensorflow/tensorflow\nae79f2514705: Pull complete\nc59d01a7e4ca: Pull complete\n41ba73a9054d: Pull complete\nf1bbfd495cc1: Pull complete\n0c346f7223e2: Pull complete\n5dcd01667896: Pull complete\nca677f607487: Downloading  180.7MB/453MB\nb4637619a887: Download complete\n8c644ff287da: Downloading    224MB/465.6MB\n119c5f576e79: Download complete\n009f82e71a7c: Download complete\ndbc0fb5872c7: Downloading  17.83MB/66.54MB\n5ef01389c5b2: Waiting\n04f824004b76: Waiting\n5861b82f52e5: Waiting\na495a3b4e6e1: Waiting\n3a0a25b1bbaf: Pulling fs layer\nb76a0afeb1e1: Waiting\n",[30,74658,74656],{"__ignoreMap":464},[11,74660,74661],{},"It finished after several minutes:",[459,74663,74666],{"className":74664,"code":74665,"language":997},[995],"ca677f607487: Pull complete\nb4637619a887: Pull complete\n8c644ff287da: Pull complete\n119c5f576e79: Pull complete\n009f82e71a7c: Pull complete\ndbc0fb5872c7: Pull complete\n5ef01389c5b2: Pull complete\n04f824004b76: Pull complete\n5861b82f52e5: Pull complete\na495a3b4e6e1: Pull complete\n3a0a25b1bbaf: Pull complete\nb76a0afeb1e1: Pull complete\nDigest: sha256:90e27448121b321c5ec66069fb2c718301df2ddaf25ba916b6f53719141572b0\nStatus: Downloaded newer image for tensorflow/tensorflow:latest-gpu\n$\n",[30,74667,74665],{"__ignoreMap":464},[11,74669,74670],{},"Let's verify that it has the image:",[459,74672,74675],{"className":74673,"code":74674,"language":997},[995],"$ docker images\nREPOSITORY              TAG                 IMAGE ID            CREATED             SIZE\ntensorflow/tensorflow   latest-gpu          2f243a16ff63        13 days ago         3.36GB\n",[30,74676,74674],{"__ignoreMap":464},[11,74678,74679,74680,50307],{},"Next let's start the ",[30,74681,72399],{},[459,74683,74686],{"className":74684,"code":74685,"language":997},[995],"$ systemctl start nvidia-docker\n==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ====\nAuthentication is required to start 'nvidia-docker.service'.\nAuthenticating as: brian\nPassword:\n==== AUTHENTICATION COMPLETE ====\n$\n",[30,74687,74685],{"__ignoreMap":464},[11,74689,74690],{},"OK, we should be ready to launch the image:",[459,74692,74695],{"className":74693,"code":74694,"language":997},[995],"$ nvidia-docker run -it tensorflow/tensorflow:latest-gpu bash\nroot@761a62c1cff1:/notebooks#\n",[30,74696,74694],{"__ignoreMap":464},[11,74698,74699],{},"This is looking good. Let's try to start python:",[459,74701,74704],{"className":74702,"code":74703,"language":997},[995],"root@761a62c1cff1:/notebooks# python\nPython 2.7.12 (default, Nov 19 2016, 06:48:10)\n[GCC 5.4.0 20160609] on linux2\nType \"help\", \"copyright\", \"credits\" or \"license\" for more information.\n>>> import tensorflow as tf\n>>>\n",[30,74705,74703],{"__ignoreMap":464},[11,74707,74708],{},"That works! Let's try out the classic MNIST hand-written digit classification problem that comes packaged as a notebook with the container image:",[459,74710,74713],{"className":74711,"code":74712,"language":997},[995],"$ nvidia-docker run -it -p 8888:8888 tensorflow/tensorflow:latest-gpu\n[sudo] password for brian:\n[I 21:54:26.671 NotebookApp] Writing notebook server cookie secret to /root/.local/share/jupyter/runtime/notebook_cookie_secret\n[W 21:54:26.689 NotebookApp] WARNING: The notebook server is listening on all IP addresses and not using encryption. This is not recommended.\n[I 21:54:26.693 NotebookApp] Serving notebooks from local directory: /notebooks\n[I 21:54:26.693 NotebookApp] 0 active kernels\n[I 21:54:26.693 NotebookApp] The Jupyter Notebook is running at:\n[I 21:54:26.693 NotebookApp] http://[all ip addresses on your system]:8888/?token=cda89aff96a3d4a9741cc755aac07f65f3aa372f60a198bd\n[I 21:54:26.693 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).\n[C 21:54:26.693 NotebookApp]\n\n    Copy/paste this URL into your browser when you connect for the first time,\n    to login with a token:\n        http://localhost:8888/?token=cda89aff96a3d4a9741cc755aac07f65f3aa372f60a198bd\n[I 21:54:34.489 NotebookApp] 302 GET /?token=cda89aff96a3d4a9741cc755aac07f65f3aa372f60a198bd (172.17.0.1) 0.32ms\n[I 21:54:59.019 NotebookApp] Writing notebook-signing key to /root/.local/share/jupyter/notebook_secret\n[W 21:54:59.023 NotebookApp] Notebook 3_mnist_from_scratch.ipynb is not trusted\n[W 21:54:59.049 NotebookApp] 404 GET /nbextensions/widgets/notebook/js/extension.js?v=20171119215426 (172.17.0.1) 4.38ms referer=http://localhost:8888/notebooks/3_mnist_from_scratch.ipynb\n[I 21:54:59.813 NotebookApp] Kernel started: 00027a3e-59ae-47ce-90a5-752a9d1fe075\n[I 21:55:00.199 NotebookApp] Adapting to protocol v5.1 for kernel 00027a3e-59ae-47ce-90a5-752a9d1fe075\n[I 21:56:59.815 NotebookApp] Saving file at /3_mnist_from_scratch.ipynb\n[W 21:56:59.816 NotebookApp] Notebook 3_mnist_from_scratch.ipynb is not trusted\n2017-11-19 21:57:03.988627: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA\n2017-11-19 21:57:04.070873: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:892] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n2017-11-19 21:57:04.071129: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties:\nname: GeForce GTX 1080 major: 6 minor: 1 memoryClockRate(GHz): 1.7335\npciBusID: 0000:01:00.0\ntotalMemory: 7.92GiB freeMemory: 7.44GiB\n2017-11-19 21:57:04.071143: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GTX 1080, pci bus id: 0000:01:00.0, compute capability: 6.1)\n",[30,74714,74712],{"__ignoreMap":464},[11,74716,74717],{},"I was only able to get the entire notebook to run after making a few small configuration tweaks to the tensorflow Interactive Session to fix some memory issues:",[459,74719,74721],{"className":13136,"code":74720,"language":12886,"meta":464,"style":464},"gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=0.75)\n\ns = tf.InteractiveSession(config=tf.ConfigProto(gpu_options=gpu_options))\n\n# Use our newly created session as the default for\n# subsequent operations.\ns.as_default()\n\n# Initialize all the variables we defined above.\ntf.global_variables_initializer().run()\n",[30,74722,74723,74743,74747,74772,74776,74781,74786,74791,74795,74800],{"__ignoreMap":464},[151,74724,74725,74728,74730,74733,74736,74738,74741],{"class":469,"line":470},[151,74726,74727],{"class":503},"gpu_options ",[151,74729,1876],{"class":1869},[151,74731,74732],{"class":503}," tf.GPUOptions(",[151,74734,74735],{"class":15210},"per_process_gpu_memory_fraction",[151,74737,1876],{"class":1869},[151,74739,74740],{"class":477},"0.75",[151,74742,3640],{"class":503},[151,74744,74745],{"class":469,"line":488},[151,74746,1090],{"emptyLinePlaceholder":609},[151,74748,74749,74752,74754,74757,74759,74761,74764,74767,74769],{"class":469,"line":500},[151,74750,74751],{"class":503},"s ",[151,74753,1876],{"class":1869},[151,74755,74756],{"class":503}," tf.InteractiveSession(",[151,74758,15233],{"class":15210},[151,74760,1876],{"class":1869},[151,74762,74763],{"class":503},"tf.ConfigProto(",[151,74765,74766],{"class":15210},"gpu_options",[151,74768,1876],{"class":1869},[151,74770,74771],{"class":503},"gpu_options))\n",[151,74773,74774],{"class":469,"line":509},[151,74775,1090],{"emptyLinePlaceholder":609},[151,74777,74778],{"class":469,"line":517},[151,74779,74780],{"class":1527},"# Use our newly created session as the default for\n",[151,74782,74783],{"class":469,"line":534},[151,74784,74785],{"class":1527},"# subsequent operations.\n",[151,74787,74788],{"class":469,"line":1413},[151,74789,74790],{"class":503},"s.as_default()\n",[151,74792,74793],{"class":469,"line":1418},[151,74794,1090],{"emptyLinePlaceholder":609},[151,74796,74797],{"class":469,"line":2462},[151,74798,74799],{"class":1527},"# Initialize all the variables we defined above.\n",[151,74801,74802],{"class":469,"line":2471},[151,74803,74804],{"class":503},"tf.global_variables_initializer().run()\n",[11,74806,74807,74808,74810,74811,13576],{},"Without setting ",[30,74809,74766],{},", Tensorflow allocates 95% of available GPU memory (according to ",[20,74812,74815],{"href":74813,"rel":74814},"https://stackoverflow.com/questions/34514324/error-using-tensorflow-with-gpu",[24],"this SO question",[11,74817,74818,74819,74822,74823,74825],{},"Setting it to ",[30,74820,74821],{},"0.333"," was too low and didn't allow for training to complete, but setting it to ",[30,74824,74740],{}," seemed to work just fine.",[11,74827,74828],{},"You can monitor GPU memory usage on NVIDIA cards with the following command:",[459,74830,74833],{"className":74831,"code":74832,"language":997},[995],"$ nvidia-smi\nSun Nov 19 17:03:03 2017\n+-----------------------------------------------------------------------------+\n| NVIDIA-SMI 384.59                 Driver Version: 384.59                    |\n|-------------------------------+----------------------+----------------------+\n| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n|===============================+======================+======================|\n|   0  GeForce GTX 1080    Off  | 00000000:01:00.0  On |                  N/A |\n| 27%   32C    P8    10W / 180W |   6707MiB /  8105MiB |      0%      Default |\n+-------------------------------+----------------------+----------------------+\n\n+-----------------------------------------------------------------------------+\n| Processes:                                                       GPU Memory |\n|  GPU       PID  Type  Process name                               Usage      |\n|=============================================================================|\n|    0       350    C   /usr/bin/python                               6365MiB |\n|    0       554    G   /usr/lib/xorg-server/Xorg                       19MiB |\n|    0       588    G   /usr/bin/gnome-shell                            28MiB |\n|    0       853    G   /usr/lib/xorg-server/Xorg                      186MiB |\n|    0       873    G   compton                                          2MiB |\n|    0      1114    G   ...el-token=A50C2F183DB4F79482A2D8768ED1B285    64MiB |\n|    0      2190    G   ...el-token=1AC796A35DBDCDBE07AEC2FC1E8026C4    35MiB |\n+-----------------------------------------------------------------------------+\n",[30,74834,74832],{"__ignoreMap":464},[11,74836,74837],{},"I think this was a success! I'm fairly certain that we were leveraging the GPU to run the MNIST hand-written digit notebook. I didn't see messages that CUDNN loaded, but I can find versions of both CUDNN and CUDA in the docker image:",[459,74839,74842],{"className":74840,"code":74841,"language":997},[995],"root@80f65a971e9a:/# ls /usr/include/x86_64-linux-gnu/\na.out.h  bits  cudnn_v6.h      fpu_control.h  gnu        python2.7\nasm      c++   expat_config.h  freetype2      ieee754.h  sys\n",[30,74843,74841],{"__ignoreMap":464},[459,74845,74848],{"className":74846,"code":74847,"language":997},[995],"root@80f65a971e9a:/# nvcc --version\nnvcc: NVIDIA (R) Cuda compiler driver\nCopyright (c) 2005-2016 NVIDIA Corporation\nBuilt on Tue_Jan_10_13:22:03_CST_2017\nCuda compilation tools, release 8.0, V8.0.61\n",[30,74849,74847],{"__ignoreMap":464},[11,74851,74852],{},"In previous attempts I had to register for an NVIDIA developer account and install these packages, but they seem to be packaged with the container.",[11,74854,74855],{},"Finally, we can check the installed python packages:",[459,74857,74860],{"className":74858,"code":74859,"language":997},[995],"root@80f65a971e9a:~# pip freeze | grep tensorflow\ntensorflow-gpu==1.4.0\ntensorflow-tensorboard==0.4.0rc2\nroot@80f65a971e9a:~#\n",[30,74861,74859],{"__ignoreMap":464},[11,74863,74864],{},"This looks good, but I'm still not 100% sure that everything was done properly. I would like to learn more about Tensorflow and also play around with some examples using Tensorboard. Let me know if you have any questions or comments about this setup, I'm still learning! Thanks for reading.",[11,74866,74867,74868,208],{},"Just for fun, here's a DeepDream rendering of a famous Donald Trump picture using Google's pre-trained ",[20,74869,74872],{"href":74870,"rel":74871},"https://github.com/google/inception",[24],"Inception model",[11,74874,74875],{},[2718,74876],{"alt":20386,"src":74877},"/static/trump.png",[11,74879,74880],{},"For comparison, here is the original image:",[11,74882,74883],{},[2718,74884],{"alt":20386,"src":74885},"/static/trump_original.jpg",[589,74887,74888],{},"html pre.shiki code .sMOD_, html code.shiki .sMOD_{--shiki-default:#24292E;--shiki-dark:#E1E4E8;--shiki-sepia:#F8F8F2}html pre.shiki code .sC2Qs, html code.shiki .sC2Qs{--shiki-default:#D73A49;--shiki-dark:#F97583;--shiki-sepia:#F92672}html pre.shiki code .sTHNf, html code.shiki .sTHNf{--shiki-default:#E36209;--shiki-default-font-style:inherit;--shiki-dark:#FFAB70;--shiki-dark-font-style:inherit;--shiki-sepia:#FD971F;--shiki-sepia-font-style:italic}html pre.shiki code .s7F3e, html code.shiki .s7F3e{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#AE81FF}html pre.shiki code .s8-w5, html code.shiki .s8-w5{--shiki-default:#6A737D;--shiki-dark:#6A737D;--shiki-sepia:#88846F}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html .sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html.sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}",{"title":464,"searchDepth":488,"depth":488,"links":74890},[74891,74892],{"id":74537,"depth":488,"text":74538},{"id":74544,"depth":488,"text":74545,"children":74893},[74894],{"id":74614,"depth":500,"text":74615},"2017-11-19","A walkthrough of Tensorflow setup and usage on Arch Linux with docker",{"layout":48045},"/2017/11/19/tensorflow-gpu-setup-with-docker-on-arch-linux",{"title":74515,"description":74896},"2017/11/19/tensorflow-gpu-setup-with-docker-on-arch-linux",[72181,72492,30129,11133,12886],"mKANSfaJEuaA7GncUWmftde8jQY_Ivj2DCI9HpJk69c",{"id":74904,"title":74905,"body":74906,"comments":609,"date":75858,"description":75859,"draft":602,"extension":605,"external":606,"image":74912,"meta":75860,"navigation":609,"path":75861,"seo":75862,"stem":75863,"tags":75864,"__hash__":75865},"blog/2017/10/31/a-binary-clock-written-in-bash.md","A binary clock written in bash",{"type":8,"value":74907,"toc":75856},[74908,74913,74922,74931,74937,74940,74943,75389,75392,75397,75422,75459,75468,75473,75625,75628,75695,75704,75716,75725,75731,75758,75766,75800,75830,75833,75837,75840,75846,75853],[11,74909,74910],{},[2718,74911],{"alt":20386,"src":74912},"/static/binaryclock.png",[11,74914,74915,74916,74921],{},"Configuring the i3 window manager on my laptop has got me interested in learning more about bash scripting. As an exercise for getting more familiar with bash, I set out to write a simple ",[20,74917,74920],{"href":74918,"rel":74919},"https://en.wikipedia.org/wiki/Binary_clock",[24],"binary clock"," application that runs in the terminal.",[11,74923,74924,74925,74930],{},"To simplifiy my clock, I decided to display Unix time as a binary number with ones and zeros represented as the unicode symbols ● and ○, respectively. ",[20,74926,74929],{"href":74927,"rel":74928},"https://en.wikipedia.org/wiki/Unix_time",[24],"Unix time"," is the number of second that have passed since January 1, 1970. Here's what I had in mind when I started out:",[459,74932,74935],{"className":74933,"code":74934,"language":997},[995]," ○ ○ ● ● ○\n ○ ● ● ● ●\n ○ ● ● ○ ●\n ● ● ○ ● ○\n ● ○ ○ ● ●\n ● ● ○ ● ○\n",[30,74936,74934],{"__ignoreMap":464},[11,74938,74939],{},"In this representation, the lower right cell represents the one's place, the next cell to the left represents the two's place, the next over the four's place, the next the eight's, and so on.",[11,74941,74942],{},"Here's the code that I ended up using for my clock program:",[459,74944,74946],{"className":461,"code":74945,"language":463,"meta":464,"style":464},"#!/bin/bash\nresize -s 8 19\n\nfunction decToBin { echo \"ibase=10; obase=2; $1\" | bc; };\n\ndraw() {\n  binstring=$(decToBin {$(date '+%s')})\n\n  for i in {31..6..-5}\n    do\n      echo $binstring | tail -c $i | head -c 5\n      printf \"\\n\"\n    done\n}\n\nprintf '\\e[?25l'\nclear\n\nwhile true ; do\n  printf '\\033[;H'\n  offset_v=$(( $(( $(tput lines)  / 2  ))  - 3  ))\n  v=$(( $offset_v > 0 ? $offset_v : 0 ));\n  for i in `seq 1 $v`;\n    do\n        printf \"\\n\"\n    done\n  offset_h=$(( $(( $(tput cols)  / 2  ))  - 7  ))\n  h=$(( $offset_h > 0 ? $offset_h : 0 ));\n  $(echo draw) | sed \"s/1/ $(tput setaf 6)● /g\" |\n                 sed \"s/0/ $(tput setaf 6)○ /g\" |\n                 sed \"s/^/$(head -c $h \u003C /dev/zero | tr '\\0' '\\ ';)/\"\n  sleep 1\ndone\n",[30,74947,74948,74952,74966,74970,74997,75001,75008,75035,75039,75052,75057,75086,75094,75099,75103,75107,75115,75120,75124,75135,75143,75181,75208,75230,75234,75241,75245,75277,75302,75332,75351,75378,75385],{"__ignoreMap":464},[151,74949,74950],{"class":469,"line":470},[151,74951,31244],{"class":1527},[151,74953,74954,74957,74960,74963],{"class":469,"line":488},[151,74955,74956],{"class":473},"resize",[151,74958,74959],{"class":477}," -s",[151,74961,74962],{"class":477}," 8",[151,74964,74965],{"class":477}," 19\n",[151,74967,74968],{"class":469,"line":500},[151,74969,1090],{"emptyLinePlaceholder":609},[151,74971,74972,74974,74977,74979,74981,74984,74987,74989,74991,74994],{"class":469,"line":509},[151,74973,59958],{"class":12347},[151,74975,74976],{"class":473}," decToBin",[151,74978,12351],{"class":503},[151,74980,412],{"class":2226},[151,74982,74983],{"class":481}," \"ibase=10; obase=2; ",[151,74985,74986],{"class":27724},"$1",[151,74988,8592],{"class":481},[151,74990,3959],{"class":1869},[151,74992,74993],{"class":473}," bc",[151,74995,74996],{"class":503},"; };\n",[151,74998,74999],{"class":469,"line":517},[151,75000,1090],{"emptyLinePlaceholder":609},[151,75002,75003,75006],{"class":469,"line":534},[151,75004,75005],{"class":473},"draw",[151,75007,70284],{"class":503},[151,75009,75010,75013,75015,75017,75020,75022,75024,75026,75029,75031,75033],{"class":469,"line":1413},[151,75011,75012],{"class":503},"  binstring",[151,75014,1876],{"class":1869},[151,75016,31456],{"class":503},[151,75018,75019],{"class":473},"decToBin",[151,75021,52023],{"class":481},[151,75023,31456],{"class":503},[151,75025,19646],{"class":473},[151,75027,75028],{"class":481}," '+%s'",[151,75030,748],{"class":503},[151,75032,2001],{"class":481},[151,75034,3640],{"class":503},[151,75036,75037],{"class":469,"line":1418},[151,75038,1090],{"emptyLinePlaceholder":609},[151,75040,75041,75043,75045,75047,75049],{"class":469,"line":2462},[151,75042,59972],{"class":1869},[151,75044,67225],{"class":503},[151,75046,16417],{"class":1869},[151,75048,52023],{"class":503},[151,75050,75051],{"class":473},"31..6..-5}\n",[151,75053,75054],{"class":469,"line":2471},[151,75055,75056],{"class":1869},"    do\n",[151,75058,75059,75062,75065,75067,75070,75073,75076,75078,75081,75083],{"class":469,"line":2480},[151,75060,75061],{"class":2226},"      echo",[151,75063,75064],{"class":503}," $binstring ",[151,75066,3947],{"class":1869},[151,75068,75069],{"class":473}," tail",[151,75071,75072],{"class":477}," -c",[151,75074,75075],{"class":503}," $i ",[151,75077,3947],{"class":1869},[151,75079,75080],{"class":473}," head",[151,75082,75072],{"class":477},[151,75084,75085],{"class":477}," 5\n",[151,75087,75088,75091],{"class":469,"line":2489},[151,75089,75090],{"class":2226},"      printf",[151,75092,75093],{"class":481}," \"\\n\"\n",[151,75095,75096],{"class":469,"line":2497},[151,75097,75098],{"class":1869},"    done\n",[151,75100,75101],{"class":469,"line":3140},[151,75102,6274],{"class":503},[151,75104,75105],{"class":469,"line":3149},[151,75106,1090],{"emptyLinePlaceholder":609},[151,75108,75109,75112],{"class":469,"line":3158},[151,75110,75111],{"class":2226},"printf",[151,75113,75114],{"class":481}," '\\e[?25l'\n",[151,75116,75117],{"class":469,"line":3167},[151,75118,75119],{"class":473},"clear\n",[151,75121,75122],{"class":469,"line":3175},[151,75123,1090],{"emptyLinePlaceholder":609},[151,75125,75126,75128,75130,75133],{"class":469,"line":3184},[151,75127,68561],{"class":1869},[151,75129,529],{"class":2226},[151,75131,75132],{"class":503}," ; ",[151,75134,31389],{"class":1869},[151,75136,75137,75140],{"class":469,"line":3193},[151,75138,75139],{"class":2226},"  printf",[151,75141,75142],{"class":481}," '\\033[;H'\n",[151,75144,75145,75148,75150,75153,75156,75159,75162,75165,75167,75169,75171,75174,75176,75178],{"class":469,"line":3720},[151,75146,75147],{"class":503},"  offset_v",[151,75149,1876],{"class":1869},[151,75151,75152],{"class":503},"$(( ",[151,75154,75155],{"class":473},"$((",[151,75157,75158],{"class":503}," $(",[151,75160,75161],{"class":473},"tput",[151,75163,75164],{"class":481}," lines",[151,75166,57949],{"class":503},[151,75168,19883],{"class":481},[151,75170,59070],{"class":477},[151,75172,75173],{"class":503},"  ))  ",[151,75175,12445],{"class":473},[151,75177,3650],{"class":477},[151,75179,75180],{"class":503},"  ))\n",[151,75182,75183,75186,75188,75191,75193,75195,75198,75201,75203,75205],{"class":469,"line":3729},[151,75184,75185],{"class":503},"  v",[151,75187,1876],{"class":1869},[151,75189,75190],{"class":503},"$(( $offset_v ",[151,75192,3663],{"class":1869},[151,75194,57890],{"class":477},[151,75196,75197],{"class":481}," ?",[151,75199,75200],{"class":503}," $offset_v ",[151,75202,208],{"class":481},[151,75204,57890],{"class":477},[151,75206,75207],{"class":503}," ));\n",[151,75209,75210,75212,75214,75216,75218,75221,75223,75226,75228],{"class":469,"line":3735},[151,75211,59972],{"class":1869},[151,75213,67225],{"class":503},[151,75215,16417],{"class":1869},[151,75217,2218],{"class":481},[151,75219,75220],{"class":473},"seq",[151,75222,12448],{"class":477},[151,75224,75225],{"class":503}," $v",[151,75227,2798],{"class":481},[151,75229,20086],{"class":503},[151,75231,75232],{"class":469,"line":3745},[151,75233,75056],{"class":1869},[151,75235,75236,75239],{"class":469,"line":3754},[151,75237,75238],{"class":2226},"        printf",[151,75240,75093],{"class":481},[151,75242,75243],{"class":469,"line":3760},[151,75244,75098],{"class":1869},[151,75246,75247,75250,75252,75254,75256,75258,75260,75263,75265,75267,75269,75271,75273,75275],{"class":469,"line":3773},[151,75248,75249],{"class":503},"  offset_h",[151,75251,1876],{"class":1869},[151,75253,75152],{"class":503},[151,75255,75155],{"class":473},[151,75257,75158],{"class":503},[151,75259,75161],{"class":473},[151,75261,75262],{"class":481}," cols",[151,75264,57949],{"class":503},[151,75266,19883],{"class":481},[151,75268,59070],{"class":477},[151,75270,75173],{"class":503},[151,75272,12445],{"class":473},[151,75274,3768],{"class":477},[151,75276,75180],{"class":503},[151,75278,75279,75282,75284,75287,75289,75291,75293,75296,75298,75300],{"class":469,"line":3782},[151,75280,75281],{"class":503},"  h",[151,75283,1876],{"class":1869},[151,75285,75286],{"class":503},"$(( $offset_h ",[151,75288,3663],{"class":1869},[151,75290,57890],{"class":477},[151,75292,75197],{"class":481},[151,75294,75295],{"class":503}," $offset_h ",[151,75297,208],{"class":481},[151,75299,57890],{"class":477},[151,75301,75207],{"class":503},[151,75303,75304,75307,75310,75312,75314,75317,75320,75322,75325,75327,75330],{"class":469,"line":3791},[151,75305,75306],{"class":473},"  $(echo",[151,75308,75309],{"class":481}," draw",[151,75311,16995],{"class":503},[151,75313,3947],{"class":1869},[151,75315,75316],{"class":473}," sed",[151,75318,75319],{"class":481}," \"s/1/ $(",[151,75321,75161],{"class":473},[151,75323,75324],{"class":481}," setaf ",[151,75326,25038],{"class":477},[151,75328,75329],{"class":481},")● /g\"",[151,75331,3979],{"class":1869},[151,75333,75334,75337,75340,75342,75344,75346,75349],{"class":469,"line":3803},[151,75335,75336],{"class":473},"                 sed",[151,75338,75339],{"class":481}," \"s/0/ $(",[151,75341,75161],{"class":473},[151,75343,75324],{"class":481},[151,75345,25038],{"class":477},[151,75347,75348],{"class":481},")○ /g\"",[151,75350,3979],{"class":1869},[151,75352,75353,75355,75358,75360,75362,75365,75368,75371,75373,75375],{"class":469,"line":3811},[151,75354,75336],{"class":473},[151,75356,75357],{"class":481}," \"s/^/$(",[151,75359,20975],{"class":473},[151,75361,75072],{"class":477},[151,75363,75364],{"class":503}," $h",[151,75366,75367],{"class":1869}," \u003C",[151,75369,75370],{"class":481}," /dev/zero ",[151,75372,3947],{"class":1869},[151,75374,32140],{"class":473},[151,75376,75377],{"class":481}," '\\0' '\\ ';)/\"\n",[151,75379,75380,75383],{"class":469,"line":3820},[151,75381,75382],{"class":473},"  sleep",[151,75384,3181],{"class":477},[151,75386,75387],{"class":469,"line":7084},[151,75388,31933],{"class":1869},[11,75390,75391],{},"The program uses two function and one while loop to display the time.",[11,75393,75394,75396],{},[30,75395,75019],{}," is a simple helper function to convert decimal numbers to binary representations.",[11,75398,75399,75401,75402,187,75404,75406,75407,75409,75410,75413,75414,187,75416,75418,75419,208],{},[30,75400,75005],{}," structures the the string of ones and zeros into 6 rows and five columns of ones and zeros. This function uses ",[30,75403,20975],{},[30,75405,55700],{}," in combination with a ",[30,75408,16732],{}," loop to iterate over a string. Notice the ",[30,75411,75412],{},"-c"," flag on ",[30,75415,55700],{},[30,75417,20975],{},". The following is from the ",[30,75420,75421],{},"man head",[459,75423,75425],{"className":461,"code":75424,"language":463,"meta":464,"style":464},"       -c, --bytes=[-]NUM\n              print the first NUM bytes of each file;\n",[30,75426,75427,75435],{"__ignoreMap":464},[151,75428,75429,75432],{"class":469,"line":470},[151,75430,75431],{"class":473},"       -c,",[151,75433,75434],{"class":477}," --bytes=[-]NUM\n",[151,75436,75437,75440,75442,75445,75448,75450,75452,75455,75457],{"class":469,"line":488},[151,75438,75439],{"class":2226},"              print",[151,75441,3084],{"class":481},[151,75443,75444],{"class":481}," first",[151,75446,75447],{"class":481}," NUM",[151,75449,72475],{"class":481},[151,75451,3090],{"class":481},[151,75453,75454],{"class":481}," each",[151,75456,4231],{"class":481},[151,75458,20086],{"class":503},[11,75460,75461,75462,75465,75466,643],{},"This gets ",[30,75463,75464],{},"NUM"," number of ones and zeros (each being one byte) from the string of ones and zeros that results from ",[30,75467,75019],{},[11,75469,27190,75470,75472],{},[30,75471,68561],{}," loop, I measure the length and width of the terminal window to center the position of the clock in case it has been changed with the following lines of code:",[459,75474,75476],{"className":461,"code":75475,"language":463,"meta":464,"style":464},"[...]\n  offset_v=$(( $(( $(tput lines)  / 2  ))  - 3  ))\n  v=$(( $offset_v > 0 ? $offset_v : 0 ));\n  for i in `seq 1 $v`;\n    do\n        printf \"\\n\"\n    done\n  offset_h=$(( $(( $(tput cols)  / 2  ))  - 7  ))\n  h=$(( $offset_h > 0 ? $offset_h : 0 ));\n[...]\n",[30,75477,75478,75483,75513,75535,75555,75559,75565,75569,75599,75621],{"__ignoreMap":464},[151,75479,75480],{"class":469,"line":470},[151,75481,75482],{"class":503},"[...]\n",[151,75484,75485,75487,75489,75491,75493,75495,75497,75499,75501,75503,75505,75507,75509,75511],{"class":469,"line":488},[151,75486,75147],{"class":503},[151,75488,1876],{"class":1869},[151,75490,75152],{"class":503},[151,75492,75155],{"class":473},[151,75494,75158],{"class":503},[151,75496,75161],{"class":473},[151,75498,75164],{"class":481},[151,75500,57949],{"class":503},[151,75502,19883],{"class":481},[151,75504,59070],{"class":477},[151,75506,75173],{"class":503},[151,75508,12445],{"class":473},[151,75510,3650],{"class":477},[151,75512,75180],{"class":503},[151,75514,75515,75517,75519,75521,75523,75525,75527,75529,75531,75533],{"class":469,"line":500},[151,75516,75185],{"class":503},[151,75518,1876],{"class":1869},[151,75520,75190],{"class":503},[151,75522,3663],{"class":1869},[151,75524,57890],{"class":477},[151,75526,75197],{"class":481},[151,75528,75200],{"class":503},[151,75530,208],{"class":481},[151,75532,57890],{"class":477},[151,75534,75207],{"class":503},[151,75536,75537,75539,75541,75543,75545,75547,75549,75551,75553],{"class":469,"line":509},[151,75538,59972],{"class":1869},[151,75540,67225],{"class":503},[151,75542,16417],{"class":1869},[151,75544,2218],{"class":481},[151,75546,75220],{"class":473},[151,75548,12448],{"class":477},[151,75550,75225],{"class":503},[151,75552,2798],{"class":481},[151,75554,20086],{"class":503},[151,75556,75557],{"class":469,"line":517},[151,75558,75056],{"class":1869},[151,75560,75561,75563],{"class":469,"line":534},[151,75562,75238],{"class":2226},[151,75564,75093],{"class":481},[151,75566,75567],{"class":469,"line":1413},[151,75568,75098],{"class":1869},[151,75570,75571,75573,75575,75577,75579,75581,75583,75585,75587,75589,75591,75593,75595,75597],{"class":469,"line":1418},[151,75572,75249],{"class":503},[151,75574,1876],{"class":1869},[151,75576,75152],{"class":503},[151,75578,75155],{"class":473},[151,75580,75158],{"class":503},[151,75582,75161],{"class":473},[151,75584,75262],{"class":481},[151,75586,57949],{"class":503},[151,75588,19883],{"class":481},[151,75590,59070],{"class":477},[151,75592,75173],{"class":503},[151,75594,12445],{"class":473},[151,75596,3768],{"class":477},[151,75598,75180],{"class":503},[151,75600,75601,75603,75605,75607,75609,75611,75613,75615,75617,75619],{"class":469,"line":2462},[151,75602,75281],{"class":503},[151,75604,1876],{"class":1869},[151,75606,75286],{"class":503},[151,75608,3663],{"class":1869},[151,75610,57890],{"class":477},[151,75612,75197],{"class":481},[151,75614,75295],{"class":503},[151,75616,208],{"class":481},[151,75618,57890],{"class":477},[151,75620,75207],{"class":503},[151,75622,75623],{"class":469,"line":2471},[151,75624,75482],{"class":503},[11,75626,75627],{},"Finally, I convert the ones and zeros to the colored unicode circles with the following lines of code:",[459,75629,75631],{"className":461,"code":75630,"language":463,"meta":464,"style":464},"  $(echo draw) | sed \"s/1/ $(tput setaf 6)● /g\" |\n                 sed \"s/0/ $(tput setaf 6)○ /g\" |\n                 sed \"s/^/$(head -c $h \u003C /dev/zero | tr '\\0' '\\ ';)/\"\n",[30,75632,75633,75657,75673],{"__ignoreMap":464},[151,75634,75635,75637,75639,75641,75643,75645,75647,75649,75651,75653,75655],{"class":469,"line":470},[151,75636,75306],{"class":473},[151,75638,75309],{"class":481},[151,75640,16995],{"class":503},[151,75642,3947],{"class":1869},[151,75644,75316],{"class":473},[151,75646,75319],{"class":481},[151,75648,75161],{"class":473},[151,75650,75324],{"class":481},[151,75652,25038],{"class":477},[151,75654,75329],{"class":481},[151,75656,3979],{"class":1869},[151,75658,75659,75661,75663,75665,75667,75669,75671],{"class":469,"line":488},[151,75660,75336],{"class":473},[151,75662,75339],{"class":481},[151,75664,75161],{"class":473},[151,75666,75324],{"class":481},[151,75668,25038],{"class":477},[151,75670,75348],{"class":481},[151,75672,3979],{"class":1869},[151,75674,75675,75677,75679,75681,75683,75685,75687,75689,75691,75693],{"class":469,"line":500},[151,75676,75336],{"class":473},[151,75678,75357],{"class":481},[151,75680,20975],{"class":473},[151,75682,75072],{"class":477},[151,75684,75364],{"class":503},[151,75686,75367],{"class":1869},[151,75688,75370],{"class":481},[151,75690,3947],{"class":1869},[151,75692,32140],{"class":473},[151,75694,75377],{"class":481},[11,75696,75697,75698,313,75700,75703],{},"Piping the output of ",[30,75699,75005],{},[30,75701,75702],{},"sed"," lets us do some simple substition using the pattern:",[459,75705,75707],{"className":461,"code":75706,"language":463,"meta":464,"style":464},"sed \"s/\u003Cwhat you want to swap out>/\u003Cwhat you want to swap in>/g\"\n",[30,75708,75709],{"__ignoreMap":464},[151,75710,75711,75713],{"class":469,"line":470},[151,75712,75702],{"class":473},[151,75714,75715],{"class":481}," \"s/\u003Cwhat you want to swap out>/\u003Cwhat you want to swap in>/g\"\n",[11,75717,19225,75718,75721,75722,75724],{},[30,75719,75720],{},"\"../g\""," at the end of the ",[30,75723,75702],{}," argument specifies that we want to make the substition globally.",[11,75726,75727,75728,75730],{},"The last ",[30,75729,75702],{}," command inserts spaces to the right of each row for the horizontal offset (in order to center the clock on our terminal window). This uses another interesting pattern that I came across on StackOverflow:",[459,75732,75734],{"className":461,"code":75733,"language":463,"meta":464,"style":464},"sed \"s/^/$(head -c $h \u003C /dev/zero | tr '\\0' '\\ ';)/\"\n",[30,75735,75736],{"__ignoreMap":464},[151,75737,75738,75740,75742,75744,75746,75748,75750,75752,75754,75756],{"class":469,"line":470},[151,75739,75702],{"class":473},[151,75741,75357],{"class":481},[151,75743,20975],{"class":473},[151,75745,75072],{"class":477},[151,75747,75364],{"class":503},[151,75749,75367],{"class":1869},[151,75751,75370],{"class":481},[151,75753,3947],{"class":1869},[151,75755,32140],{"class":473},[151,75757,75377],{"class":481},[11,75759,19225,75760,75762,75763,75765],{},[30,75761,57705],{}," is a regular expression that represents the beginning of a line. So with this ",[30,75764,75702],{}," substitution we will be adding to the beginning of each line. What we are adding is the following:",[459,75767,75769],{"className":461,"code":75768,"language":463,"meta":464,"style":464},"$(head -c $h \u003C /dev/zero | tr '\\0' '\\ ';)/\n",[30,75770,75771],{"__ignoreMap":464},[151,75772,75773,75775,75777,75779,75782,75784,75787,75789,75791,75794,75797],{"class":469,"line":470},[151,75774,31456],{"class":503},[151,75776,20975],{"class":473},[151,75778,75072],{"class":477},[151,75780,75781],{"class":503}," $h ",[151,75783,3613],{"class":1869},[151,75785,75786],{"class":481}," /dev/zero",[151,75788,3959],{"class":1869},[151,75790,32140],{"class":473},[151,75792,75793],{"class":481}," '\\0'",[151,75795,75796],{"class":481}," '\\ '",[151,75798,75799],{"class":503},";)/\n",[11,75801,75802,75803,75806,75807,75809,75810,18952,75813,75815,75816,75818,75819,75822,75823,75825,75826,75829],{},"This takes the number of columns that we want to shift our clock as ",[30,75804,75805],{},"$h"," and reads the first ",[30,75808,75805],{}," bytes from ",[30,75811,75812],{},"/dev/zero",[30,75814,75812],{}," produces a continuous stream of NULL (zero value) bytes, so the first ",[30,75817,75805],{}," bytes will be something like ",[30,75820,75821],{},"\\0, \\0, \\0, \\0, \\0",". We then pipe this output to ",[30,75824,1137],{}," which translates the null bytes into spaces (",[30,75827,75828],{},"'\\ '",") which help us pad our clock.",[11,75831,75832],{},"Here's a screenshot of the clock in action:",[11,75834,75835],{},[2718,75836],{"alt":20386,"src":74912},[11,75838,75839],{},"Here's the script on my github account:",[11,75841,75842],{},[20,75843,75844],{"href":75844,"rel":75845},"https://github.com/briancaffey/binaryclock/blob/master/binaryclock",[24],[11,75847,75848,75849,75852],{},"The clock works well on ",[30,75850,75851],{},"rxvt-unicode",", but I need to make some small changes to make it work on other terminal emulators.",[589,75854,75855],{},"html pre.shiki code .s8-w5, html code.shiki .s8-w5{--shiki-default:#6A737D;--shiki-dark:#6A737D;--shiki-sepia:#88846F}html pre.shiki code .srTi1, html code.shiki .srTi1{--shiki-default:#6F42C1;--shiki-dark:#B392F0;--shiki-sepia:#A6E22E}html pre.shiki code .s7F3e, html code.shiki .s7F3e{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#AE81FF}html pre.shiki code .sq6CD, html code.shiki .sq6CD{--shiki-default:#D73A49;--shiki-default-font-style:inherit;--shiki-dark:#F97583;--shiki-dark-font-style:inherit;--shiki-sepia:#66D9EF;--shiki-sepia-font-style:italic}html pre.shiki code .sMOD_, html code.shiki .sMOD_{--shiki-default:#24292E;--shiki-dark:#E1E4E8;--shiki-sepia:#F8F8F2}html pre.shiki code .sTrkL, html code.shiki .sTrkL{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#66D9EF}html pre.shiki code .sstjo, html code.shiki .sstjo{--shiki-default:#032F62;--shiki-dark:#9ECBFF;--shiki-sepia:#E6DB74}html pre.shiki code .sdpu8, html code.shiki .sdpu8{--shiki-default:#005CC5;--shiki-default-font-style:inherit;--shiki-dark:#79B8FF;--shiki-dark-font-style:inherit;--shiki-sepia:#FD971F;--shiki-sepia-font-style:italic}html pre.shiki code .sC2Qs, html code.shiki .sC2Qs{--shiki-default:#D73A49;--shiki-dark:#F97583;--shiki-sepia:#F92672}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html .sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html.sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}",{"title":464,"searchDepth":488,"depth":488,"links":75857},[],"2017-10-31","A program that displays the current time as a binary representation",{"layout":48045},"/2017/10/31/a-binary-clock-written-in-bash",{"title":74905,"description":75859},"2017/10/31/a-binary-clock-written-in-bash",[463],"J04foYj65xekIq7plIUBMcerIWDhDDrZw1bJefgiV7A",{"id":75867,"title":75868,"body":75869,"comments":609,"date":76748,"description":464,"draft":602,"extension":605,"external":606,"image":76749,"meta":76750,"navigation":609,"path":76751,"seo":76752,"stem":76753,"tags":76754,"__hash__":76756},"blog/2017/10/17/moving-from-gnome-to-i3-on-arch-linux.md","Moving from Gnome Desktop to i3 window manager on Arch Linux",{"type":8,"value":75870,"toc":76731},[75871,75876,75885,75888,75891,75929,75935,75941,75947,75956,75967,75984,75987,75995,75998,76004,76010,76016,76025,76028,76032,76044,76050,76058,76065,76075,76085,76091,76094,76103,76109,76115,76125,76140,76144,76155,76163,76166,76172,76177,76180,76188,76191,76203,76217,76223,76229,76232,76235,76241,76244,76250,76256,76262,76275,76281,76287,76293,76299,76304,76307,76313,76316,76322,76328,76333,76339,76342,76344,76350,76356,76362,76367,76373,76376,76380,76383,76386,76394,76400,76407,76410,76416,76419,76425,76433,76440,76446,76449,76456,76459,76465,76475,76481,76488,76491,76496,76502,76505,76509,76515,76518,76524,76528,76533,76539,76542,76548,76554,76557,76594,76600,76604,76612,76618,76621,76627,76633,76636,76640,76644,76650,76653,76659,76664,76670,76674,76677,76725],[11,75872,75873],{},[2718,75874],{"alt":20386,"src":75875},"/static/gnome.png",[11,75877,75878,75879,75884],{},"I recently tried out ",[20,75880,75883],{"href":75881,"rel":75882},"http://i3wm.org",[24],"i3"," on my laptop and I'm really liking it so far. I am going to try to recreate the same i3 configuration on my desktop installation of Arch Linux. I'll try to faithfully cover each step of the process in this article.",[11,75886,75887],{},"i3 sounded like a nice idea at first, but there were a lot of aspects of my Gnome desktop that I didn't think I could do without. I'm still new to i3, but I have found it very interesting to see how everything can be configured. If you are thinking about switching to i3, hopefully this can help you out.",[11,75889,75890],{},"Here's a list of everything I want to go through:",[76,75892,75893,75896,75899,75902,75908,75911,75914,75917,75920,75923,75926],{},[79,75894,75895],{},"Installing i3",[79,75897,75898],{},"Basic commands",[79,75900,75901],{},"Workflow",[79,75903,75904,75905],{},"Setting a background with ",[30,75906,75907],{},"feh",[79,75909,75910],{},"The i3 config file",[79,75912,75913],{},"Setting uxrvt as a default terminal",[79,75915,75916],{},"Customizing uxrvt",[79,75918,75919],{},"Customizing workspaces",[79,75921,75922],{},"Setting up blocks with i3blocks",[79,75924,75925],{},"Custom lock screen with animation",[79,75927,75928],{},"i3-gaps",[11,75930,75931,75932,75934],{},"Start out by installing ",[30,75933,75883],{}," with pacman as follows",[459,75936,75939],{"className":75937,"code":75938,"language":997},[995],"[brian@archthinkpad ~]$ sudo pacman -S i3\n[sudo] password for brian:\n:: There are 4 members in group i3:\n:: Repository community\n   1) i3-wm  2) i3blocks  3) i3lock  4) i3status\n\nEnter a selection (default=all):\n",[30,75940,75938],{"__ignoreMap":464},[11,75942,75943,75944,75946],{},"You won't find ",[30,75945,75883],{}," with a regular search in the AUR because it is a package group, containing a number of packages that will help us do things with i3.",[11,75948,75949,75950,75952,75953,75955],{},"Once you have installed ",[30,75951,75883],{},", logout of your current Gnome session, and then go back to login and select ",[30,75954,75883],{}," from the login menu.",[11,75957,75958,75959,75962,75963,75966],{},"You will be greated with a black screen and a dialogue box that says ",[30,75960,75961],{},"i3: first confuguration",". I recommend that you press ",[30,75964,75965],{},"Enter"," and have i3 generate a config file for you as it says in the prompt.",[11,75968,75969,75970,30583,75973,75976,75977,75979,75980,75983],{},"Next, choose either ",[30,75971,75972],{},"Win",[30,75974,75975],{},"Alt"," as the key that will help you launch most commands in i3. I use ",[30,75978,75975],{},", but it won't make a difference in this tutorial since we will be refering to whichever key you select as ",[30,75981,75982],{},"Mod1"," from here on out.",[11,75985,75986],{},"Once you make this selection, the prompt will go away and you are met with a black screen and and a status bar on the bottom, as well as workspace indicator on the bottom left. Welcome to i3!",[11,75988,75989,75990,75994],{},"At this point, you should have a read through the very well-written i3 User Guide linked ",[20,75991,13074],{"href":75992,"rel":75993},"https://i3wm.org/docs/userguide.html",[24],". Learn how to move windows around, close windows and stack windows.",[11,75996,75997],{},"Here's the one command you really need to get going:",[11,75999,76000,76003],{},[30,76001,76002],{},"Mod1+Enter",": open a new terminal",[11,76005,76006,76007,208],{},"You probably want to change this terminal right away. There's a handy little program launcher that we can use for now called ",[30,76008,76009],{},"dmenu",[459,76011,76014],{"className":76012,"code":76013,"language":997},[995],"[brian@archthinkpad ~]$ sudo pacman -Ss dmenu\n[sudo] password for brian:\ncommunity/dmenu 4.7-1 [installed]\n    A generic menu for X\ncommunity/pdmenu 1.3.2-2\n    simple full screen menu program\n[brian@archthinkpad ~]$\n",[30,76015,76013],{"__ignoreMap":464},[11,76017,76018,76019,76021,76022,643],{},"Once you install ",[30,76020,76009],{},", you can easily launch programs with ",[30,76023,76024],{},"Mod1+d",[11,76026,76027],{},"In a minute we will customize dmenu to look better, but for now we need to get into the meat of i3: customization.",[56,76029,76031],{"id":76030},"customization","Customization",[11,76033,76034,76035,76037,76038,313,76041,37588],{},"Most of the work you do in customizing i3 involves editing ",[30,76036,75883],{},"'s config file. To change the configuration of i3, copy ",[30,76039,76040],{},"/etc/i3/config",[30,76042,76043],{},"~/.i3/config",[459,76045,76048],{"className":76046,"code":76047,"language":997},[995],"cp /etc/i3/config ~/.i3/config\n",[30,76049,76047],{"__ignoreMap":464},[11,76051,76052,76053,30583,76055,643],{},"At this point I will reference my \"Dotfiles\" on github. Dotfiles is used to refer to hidden configuration folders and files prepended with a \".\", such as ",[30,76054,76043],{},[30,76056,76057],{},"~/.Xresources",[11,76059,76060,76061,643],{},"A public repo with my dotfiles is available ",[20,76062,13074],{"href":76063,"rel":76064},"https://github.com/briancaffey/.i3",[24],[11,76066,76067,76068,76071,76072,76074],{},"Let's start by adding a background image. Find an image you like and add it to ",[30,76069,76070],{},"~/Pictures",". Then install ",[30,76073,75907],{}," from the AUR if you don't alread have it.",[11,76076,76077,76079,76080,76082,76083,208],{},[30,76078,75907],{}," will let us set a background image from the command line, and we can do so each time we launch ",[30,76081,75883],{}," by adding the following line to ",[30,76084,76043],{},[459,76086,76089],{"className":76087,"code":76088,"language":997},[995],"exec_always feh --bg-scale ~/Pictures/image.jpg\n",[30,76090,76088],{"__ignoreMap":464},[11,76092,76093],{},"Now we can run the following command to restart i3 in place (without having to logout):",[11,76095,76096,22885,76098,22885,76101],{},[30,76097,75982],{},[30,76099,76100],{},"shift",[30,76102,58741],{},[11,76104,76105,76106,76108],{},"You will see this shortcut in ",[30,76107,76043],{}," in the following line:",[459,76110,76113],{"className":76111,"code":76112,"language":997},[995],"bindsym Mod1+Shift+r restart\n",[30,76114,76112],{"__ignoreMap":464},[11,76116,76117,76118,22885,76120,22885,76122,76124],{},"This binds ",[30,76119,75982],{},[30,76121,76100],{},[30,76123,58741],{}," to instructions that restart i3.",[11,76126,76127,76128,76131,76132,76135,76136,76139],{},"If you write ",[30,76129,76130],{},"bindsym Some+key+combo exec reboot",", your system will reboot when you press ",[30,76133,76134],{},"Some+key+combo",". You can call any command this way, allowing for a high level of customization. It is helpful to see what others have done by browsing ",[30,76137,76138],{},".dotfiles"," online. You can mix and match commands to your liking. If you make an error, i3 will warn you when you start or restart i3.",[56,76141,76143],{"id":76142},"customizing-the-terminal-urxvt","Customizing the Terminal (urxvt)",[11,76145,76146,76147,76150,76151,76154],{},"Next let's take care of our terminal. You will notice that the command to launch a new terminal actually launches another program called ",[30,76148,76149],{},"i3-sensible-terminal",". Run ",[30,76152,76153],{},"man i3-sensible-terminal"," to see how this works in detail. It basically picks a terminal program for you based on what you have installed on your system.",[11,76156,76157,76158,76160,76161,643],{},"For my terminal, I use a program called ",[30,76159,74459],{},". This is a popular terminal program in the i3 community because of the fact that it is highly customizable. There are a lot of options for terminals, so feel free to use whatever you like. I will go into depth here about how I customize ",[30,76162,74459],{},[11,76164,76165],{},"First, install it with:",[459,76167,76170],{"className":76168,"code":76169,"language":997},[995],"sudo pacman -S rxvt-unicode\n",[30,76171,76169],{"__ignoreMap":464},[11,76173,76174,76175],{},"And then launch it with ",[30,76176,74459],{},[11,76178,76179],{},"It probably looks equally bad to whatever default you were using, but we are about to fix it up so it looks and works great.",[11,76181,76182,76183,643],{},"First, have a look at the ",[20,76184,76187],{"href":76185,"rel":76186},"https://wiki.archlinux.org/index.php/rxvt-unicode",[24],"Arch Wikie article on rxvt-unicode",[736,76189,76057],{"id":76190},"xresources",[11,76192,76193,76194,76196,76197,76199,76200,76202],{},"Just like how the behavior of i3 is read from ",[30,76195,76043],{},", the behavior of ",[30,76198,74459],{}," is read from a file in your home directory called ",[30,76201,76057],{},". This file won't be here, so we need to create it.",[11,76204,76205,76206,76211,76212,76214,76215,208],{},"For simplicity, I recommend that you copy the contents of ",[20,76207,76210],{"href":76208,"rel":76209},"https://raw.githubusercontent.com/briancaffey/.i3/master/.Xresources",[24],"this link"," into you newly created ",[30,76213,76057],{}," file and then run the following command to refresh the settings for ",[30,76216,74459],{},[459,76218,76221],{"className":76219,"code":76220,"language":997},[995],"xrdb ~/.Xresources\n",[30,76222,76220],{"__ignoreMap":464},[11,76224,76225,76226,76228],{},"Now restart ",[30,76227,74459],{}," and you should see that it looks very different.",[11,76230,76231],{},"Look over the arch wiki article mentioned above for more information on how to customize urxvt.",[56,76233,76234],{"id":76234},"pywal",[11,76236,76237,76238,76240],{},"Next lets work on terminal colors. There is a great program called ",[30,76239,76234],{}," which reads one or several image file and then applies a color scheme to your system based on the colors found.",[11,76242,76243],{},"Install it from the AUR:",[459,76245,76248],{"className":76246,"code":76247,"language":997},[995],"sudo pacman -S python-pywal\n",[30,76249,76247],{"__ignoreMap":464},[11,76251,76252,76253,208],{},"Now we just need to add the following line to ",[30,76254,76255],{},"~/.bashrc",[459,76257,76260],{"className":76258,"code":76259,"language":997},[995],"# pywal\nsetsid wal -i ~/Pictures/image.jpg\n",[30,76261,76259],{"__ignoreMap":464},[11,76263,10635,76264,76267,76268,76270,76271,76274],{},[30,76265,76266],{},"source ~/.bashrc"," and reopen ",[30,76269,74459],{}," and it should have new color scheme that goes well with you background image. You can install another program called ",[30,76272,76273],{},"neofetch"," to print out the current colorscheme along with system information:",[459,76276,76279],{"className":76277,"code":76278,"language":997},[995],"yaourt -S neofetch\n",[30,76280,76278],{"__ignoreMap":464},[11,76282,76283,76284,76286],{},"You can add the following line to the bottom of your ",[30,76285,76255],{}," file and have neofetch run whenever you open a new terminal:",[459,76288,76291],{"className":76289,"code":76290,"language":997},[995],"setsid wal -r\n",[30,76292,76290],{"__ignoreMap":464},[459,76294,76297],{"className":76295,"code":76296,"language":997},[995],"neofetch\n",[30,76298,76296],{"__ignoreMap":464},[11,76300,76301,76302,37166],{},"Here are a few other customizations I have in my ",[30,76303,76043],{},[11,76305,76306],{},"Remove window boarders:",[459,76308,76311],{"className":76309,"code":76310,"language":997},[995],"for_window [class=\"^.*\"] border pixel 0\n",[30,76312,76310],{"__ignoreMap":464},[11,76314,76315],{},"Enable smooth transitions with compton:",[11,76317,76318,76321],{},[30,76319,76320],{},"compton"," is a package that enables for nice transitions when navigating i3. First, install the compton package:",[459,76323,76326],{"className":76324,"code":76325,"language":997},[995],"yaourt -S compton\n",[30,76327,76325],{"__ignoreMap":464},[11,76329,76330,76331,208],{},"Next we need to add the following line to ",[30,76332,76043],{},[459,76334,76337],{"className":76335,"code":76336,"language":997},[995],"exec compton -f\n",[30,76338,76336],{"__ignoreMap":464},[11,76340,76341],{},"You might need to reboot to see how the effect that compton has.",[56,76343,75928],{"id":75928},[11,76345,76346,76347,76349],{},"We can now add a neat feature to our i3 setup by installing a popular fork of i3 callde ",[30,76348,75928],{},". It adds some additional functionality to i3, including the ability to add gaps in between our windows.",[11,76351,76352,76353,76355],{},"Install ",[30,76354,75928],{}," from the AUR:",[459,76357,76360],{"className":76358,"code":76359,"language":997},[995],"yaourt -S i3-gaps\n",[30,76361,76359],{"__ignoreMap":464},[11,76363,76364,76365,208],{},"Remove the packages in conflict and then add the following lines to ",[30,76366,76043],{},[459,76368,76371],{"className":76369,"code":76370,"language":997},[995],"# i3-gaps\ngaps inner 10\ngaps outer 0\n",[30,76372,76370],{"__ignoreMap":464},[11,76374,76375],{},"Refresh i3 and you should now see gaps in between your windows.",[56,76377,76379],{"id":76378},"the-bar","The Bar",[11,76381,76382],{},"Things should be looking pretty good, but we still need to do some work on the bar at the bottom of the screen.",[11,76384,76385],{},"Let's install FontAwesome so we can use some nice icons in our bar:",[11,76387,76388,76389,76393],{},"Find the most recent release of FontAwesome ",[20,76390,13074],{"href":76391,"rel":76392},"https://github.com/FortAwesome/Font-Awesome/releases",[24],", click on the zip download link and then run:",[459,76395,76398],{"className":76396,"code":76397,"language":997},[995],"unzip ~/Downloads/Font-Awesome-4.7.0\n",[30,76399,76397],{"__ignoreMap":464},[11,76401,76402,76403,76406],{},"The release number may be different for you. Once you have unzipped the file, we want to move all of the files ending with ",[30,76404,76405],{},".ttf"," to a folder that may or may not exist on your machine.",[11,76408,76409],{},"First, run:",[459,76411,76414],{"className":76412,"code":76413,"language":997},[995],"mkdir ~/.fonts\n",[30,76415,76413],{"__ignoreMap":464},[11,76417,76418],{},"and then run:",[459,76420,76423],{"className":76421,"code":76422,"language":997},[995],"cp ~/Downloads/Font-Awesome-4.7.0/fonts/*.ttf ~/.fonts\n",[30,76424,76422],{"__ignoreMap":464},[11,76426,76427,76428,76432],{},"Now that you have these fonts installed, go over to ",[20,76429,76430],{"href":76430,"rel":76431},"http://fontawesome.io/cheatsheet/",[24]," and you should see lots of icons that you might not have been able to see before.",[11,76434,76435,76436,76439],{},"We will come back to the fonts in just a minute. First let's change the bar by replacing ",[30,76437,76438],{},"bar {...}"," with the following:",[459,76441,76444],{"className":76442,"code":76443,"language":997},[995],"bar {\n    position top\n        status_command i3blocks -c /home/brian/.i3/i3blocks.conf\n    colors {\n        background $bg-color\n            separator #757575\n        #                  border             background         text\n        focused_workspace  $bg-color          $bg-color          $text-color\n        inactive_workspace $inactive-bg-color $inactive-bg-color $inactive-text-color\n        urgent_workspace   $urgent-bg-color   $urgent-bg-color   $text-color\n    }\n}\n",[30,76445,76443],{"__ignoreMap":464},[11,76447,76448],{},"Don't refresh i3 just yet. Let's go through this block first.",[11,76450,76451,76452,76455],{},"We tell i3 that our bar will appear at the top of the screen, and that the contents of the bar come from the command: ",[30,76453,76454],{},"status_command i3blocks -c /home/brian/.i3/i3blocks.conf",". Finally, we define some background colors.",[11,76457,76458],{},"First, pull in the colors from my dotfiles before the bar block:",[459,76460,76463],{"className":76461,"code":76462,"language":997},[995],"set $bg-color            #2f343f\nset $inactive-bg-color   #2f343f\nset $text-color          #f3f4f5\nset $inactive-text-color #676E7D\nset $urgent-bg-color     #E53935\n\n# window colors\n#                       border              background         text                 indicator\nclient.focused          $bg-color           $bg-color          $text-color          #00ff00\nclient.unfocused        $inactive-bg-color $inactive-bg-color $inactive-text-color #00ff00\nclient.focused_inactive $inactive-bg-color $inactive-bg-color $inactive-text-color #00ff00\nclient.urgent           $urgent-bg-color    $urgent-bg-color   $text-color          #00ff00\n",[30,76464,76462],{"__ignoreMap":464},[11,76466,76467,76468,76471,76472,76474],{},"Next let's take care of the i3blocks command. The ",[30,76469,76470],{},"i3blocks"," package was installed when we instaled i3, so we just need to provide an absolute path to a file as the argument for this command, as well as the ",[30,76473,75412],{}," flag. In this file we simply define what shows up in the bar. Here's a simple version that shows date, time, CPU temperature and wifi:",[459,76476,76479],{"className":76477,"code":76478,"language":997},[995],"[wifi]\nlabel=\ncommand=iwgetid -r\nseparator=true\ninterval=3\n\n#[volume]\n#label=\n#interval=1\n#separator=true\n#command=amixer get Master | egrep -o \"[0-9]+%\" | sed -n '2 p'\n\n#[cpu]\n#label=\n#interval=10\n#separator=true\n\n[temperature]\ncommand=T=$(cat /sys/class/thermal/thermal_zone0/temp); echo $(( $T / 1000 ))°C\nlabel=\ninterval=10\nseparator=true\n\n[time]\ncommand= date '+%H:%M:%S'\ninterval=2\nlabel=\nseparator=true\n\n[day]\ncommand= date '+%a %b %e, %Y'\ninterval=2\nlabel=\nseparator=true\n",[30,76480,76478],{"__ignoreMap":464},[11,76482,76483,76484,76487],{},"This is why it is called i3",[51,76485,76486],{},"blocks",", because each part of the bar is defined in a block that has a command, an interval in seconds that determines how often the command is run, and a label. You can choose any text or icon for the labels.",[11,76489,76490],{},"Next we can add some labels to the workspaces on the left side of the bar. I like to divide my workspaces into groups to keep things organized.",[11,76492,76493,76494,37166],{},"Add the following to your ",[30,76495,76043],{},[459,76497,76500],{"className":76498,"code":76499,"language":997},[995],"set $workspace10 \"H0me \"\nset $workspace1 \"F1rst \"\nset $workspace4 \"Edi4or \"\nset $workspace3 \"Brows3r \"\nset $workspace2 \"2erminal \"\nset $workspace5 \"Mu5ic \"\nset $workspace8 \"O8S \"\n\n# switch to workspace\nbindsym Mod1+1 workspace $workspace1\nbindsym Mod1+2 workspace $workspace2\nbindsym Mod1+3 workspace $workspace3\nbindsym Mod1+4 workspace $workspace4\nbindsym Mod1+5 workspace $workspace5\nbindsym Mod1+6 workspace 6\nbindsym Mod1+7 workspace 7\nbindsym Mod1+8 workspace $workspace8\nbindsym Mod1+9 workspace 9\nbindsym Mod1+0 workspace $workspace10\n\n# move focused container to workspace\nbindsym Mod1+Shift+1 move container to workspace $workspace1\nbindsym Mod1+Shift+2 move container to workspace $workspace2\nbindsym Mod1+Shift+3 move container to workspace $workspace3\nbindsym Mod1+Shift+4 move container to workspace $workspace4\nbindsym Mod1+Shift+5 move container to workspace $workspace5\nbindsym Mod1+Shift+6 move container to workspace 6\nbindsym Mod1+Shift+7 move container to workspace 7\nbindsym Mod1+Shift+8 move container to workspace $workspace8\nbindsym Mod1+Shift+9 move container to workspace 9\nbindsym Mod1+Shift+0 move container to workspace $workspace10\n",[30,76501,76499],{"__ignoreMap":464},[11,76503,76504],{},"Things are looking pretty good at this point. Here are some other things that I have found to be very helpful:",[736,76506,76508],{"id":76507},"brightness-controller","Brightness Controller",[459,76510,76513],{"className":76511,"code":76512,"language":997},[995],"yaourt -S brightness-controller\n",[30,76514,76512],{"__ignoreMap":464},[11,76516,76517],{},"Next we can add a shortcut for this package:",[459,76519,76522],{"className":76520,"code":76521,"language":997},[995],"bindsym Mod1+Ctrl+b exec brightness-controller\n",[30,76523,76521],{"__ignoreMap":464},[736,76525,76527],{"id":76526},"ranger","Ranger",[11,76529,76530,76532],{},[30,76531,76526],{}," is a terminal-based file browser that is nice to use.",[459,76534,76537],{"className":76535,"code":76536,"language":997},[995],"yaourt -S ranger\n",[30,76538,76536],{"__ignoreMap":464},[11,76540,76541],{},"There is a lot we can do to customize ranger, here are some important things that I do. First, run the following command:",[459,76543,76546],{"className":76544,"code":76545,"language":997},[995],"ranger --copy-config=all\n",[30,76547,76545],{"__ignoreMap":464},[11,76549,76550,76551,643],{},"This will create a file located here: ",[30,76552,76553],{},"~/.config/ranger/rc.conf",[11,76555,76556],{},"We need to make the following adjustments:",[76,76558,76559,76568,76576,76582,76588],{},[79,76560,76561,76562,313,76565],{},"change ",[30,76563,76564],{},"set preview_images false",[30,76566,76567],{},"set preview_images true",[79,76569,76561,76570,313,76573],{},[30,76571,76572],{},"set draw_borders false",[30,76574,76575],{},"set draw_borders true",[79,76577,76578,76579],{},"install a package called ",[30,76580,76581],{},"w3m",[79,76583,76584,76585],{},"add (overwrite) this line: ",[30,76586,76587],{},"set preview_images_method w3m",[79,76589,10029,76590,76593],{},[30,76591,76592],{},"source ~/.config/ranger/rc.conf"," and then launch ranger in a new terminal",[11,76595,76596,76597,76599],{},"These changes will allow you to preview images right inside of the ranger file browser. There may be another combination of settings that get this to work, but these worked for me. Be careful that things you want to change in ",[30,76598,76553],{}," are not overwritten by other lines.",[56,76601,76603],{"id":76602},"rofi","Rofi",[11,76605,76606,76608,76609,76611],{},[30,76607,76602],{}," is a launcher similar to ",[30,76610,76009],{}," that we used in the beginnings, but it is better. Here's how to install it:",[459,76613,76616],{"className":76614,"code":76615,"language":997},[995],"sudo pacman -S rofi\n",[30,76617,76615],{"__ignoreMap":464},[11,76619,76620],{},"Next we can add a key binding so we can quickly launch any program with rofi:",[459,76622,76625],{"className":76623,"code":76624,"language":997},[995],"bindsym Mod1+space exec rofi -show run\n",[30,76626,76624],{"__ignoreMap":464},[11,76628,76629,76630,643],{},"Be careful when you add new bindings! i3 will give you an error if you assign one binding to more than one command. I had to shuffle some of the default bindings around to use ",[30,76631,76632],{},"Mod1+space",[11,76634,76635],{},"This is a good place to stop for now. There is a lot you can do with i3, so it is good to take things one step at a time and also try to do things in different ways to see what you like.",[56,76637,76639],{"id":76638},"extras","Extras",[736,76641,76643],{"id":76642},"chinese-input-support","Chinese Input Support",[11,76645,76352,76646,76649],{},[30,76647,76648],{},"ibus-pinyin"," from AUR",[11,76651,76652],{},"Run the ibus daemon:",[459,76654,76657],{"className":76655,"code":76656,"language":997},[995],"ibus-daemon -xim&\n",[30,76658,76656],{"__ignoreMap":464},[11,76660,76661,76662],{},"Install another Chinese font and set in ",[30,76663,76057],{},[11,76665,76666],{},[20,76667,76668],{"href":76668,"rel":76669},"https://askubuntu.com/questions/826577/switch-keyboard-layouts-with-i3/826578",[24],[56,76671,76673],{"id":76672},"dotfiles","Dotfiles",[11,76675,76676],{},"Here are the configuration files I use for i3:",[76,76678,76679,76684,76690,76698,76705,76712,76720],{},[79,76680,76681,76683],{},[30,76682,76043],{},": general configurations for i3",[79,76685,76686,76689],{},[30,76687,76688],{},"~/.i3/i3blocks.conf",": configuration for status bar in i3",[79,76691,76692,76695,76696],{},[30,76693,76694],{},"~/.i3/blocks/scripts/",": scripts that run for ",[30,76697,76470],{},[79,76699,76700,76702,76703],{},[30,76701,76057],{},": configuration for ",[30,76704,75851],{},[79,76706,76707,76709,76710],{},[30,76708,76553],{},": configuration files for ",[30,76711,76526],{},[79,76713,76714,76702,76717],{},[30,76715,76716],{},"~/.vimrc",[30,76718,76719],{},"vim",[79,76721,76722,76724],{},[30,76723,76255],{},": environment variables, functions and aliases",[11,76726,76727,76728,643],{},"I'm constantly changing things around in these files but I try to keep them up to date in a repo on my Github account which you can find ",[20,76729,13074],{"href":76063,"rel":76730},[24],{"title":464,"searchDepth":488,"depth":488,"links":76732},[76733,76734,76737,76738,76739,76743,76744,76747],{"id":76030,"depth":488,"text":76031},{"id":76142,"depth":488,"text":76143,"children":76735},[76736],{"id":76190,"depth":500,"text":76057},{"id":76234,"depth":488,"text":76234},{"id":75928,"depth":488,"text":75928},{"id":76378,"depth":488,"text":76379,"children":76740},[76741,76742],{"id":76507,"depth":500,"text":76508},{"id":76526,"depth":500,"text":76527},{"id":76602,"depth":488,"text":76603},{"id":76638,"depth":488,"text":76639,"children":76745},[76746],{"id":76642,"depth":500,"text":76643},{"id":76672,"depth":488,"text":76673},"2017-10-17","/static/gnome-i3.png",{"layout":48045},"/2017/10/17/moving-from-gnome-to-i3-on-arch-linux",{"title":75868,"description":464},"2017/10/17/moving-from-gnome-to-i3-on-arch-linux",[72181,75883,76755],"gnome","iihtujHj81LaR4jH3gZCG0z8XCZ1wE09c_CmI4hE7ag",{"id":76758,"title":76759,"body":76760,"comments":609,"date":76748,"description":77014,"draft":602,"extension":605,"external":606,"image":77015,"meta":77016,"navigation":609,"path":77017,"seo":77018,"stem":77019,"tags":77020,"__hash__":77022},"blog/2017/10/17/reading-rss-with-python.md","Reading RSS with Python",{"type":8,"value":76761,"toc":77012},[76762,76779,76790,76792,76801,76811,76817,76825,76921,76924,77006,77009],[11,76763,76764,76765,76768,76769,76771,76772,76774,76775,76778],{},"On my other personal website, ",[20,76766,76767],{"href":76767},"briancaffey.com",", I have a blog. The content on that blog has mostly mirrored what I put on this github pages site, ",[20,76770,662],{"href":662},". I want to display my most recent blog posts from briancaffey.github.io on briancaffey.com, and to do this I will be using the RSS feed that comes with a Jekyll site. This should be pretty simple, we are going to use the ",[30,76773,54354],{}," librrary, as well as the ",[30,76776,76777],{},"feedparser"," library.",[11,76780,76781,76782,76786,76787,76789],{},"Here are some ",[20,76783,36434],{"href":76784,"rel":76785},"https://wiki.python.org/moin/RssLibraries",[24]," on how to use ",[30,76788,76777],{},", it is very simple.",[11,76791,76165],{},[459,76793,76795],{"className":13136,"code":76794,"language":12886,"meta":464,"style":464},"pip install feedparser\n",[30,76796,76797],{"__ignoreMap":464},[151,76798,76799],{"class":469,"line":470},[151,76800,76794],{"class":503},[11,76802,76803,76804,76806,76807,76810],{},"Here's the setup that I will be using in utility a function that will be imported to ",[30,76805,61903],{}," and called in the ",[30,76808,76809],{},"home()"," function that renders the homepage for briancaffey.com:",[459,76812,76815],{"className":76813,"code":76814,"language":997},[995],"import feedparser\n\ndef get_blog_posts(number_of_posts):\n    url = \"http://briancaffey.github.io/feed\"\n    feed = feedparser.parse(url)\n    posts = feed['items'][:number_of_posts]\n    return posts\n\n",[30,76816,76814],{"__ignoreMap":464},[11,76818,76819,76820,76822,76823,208],{},"Next, in ",[30,76821,61903],{},", we just need to import the function, call it with the number of articles we want to show, save the returned value to a variable and then pass that to ",[30,76824,39524],{},[459,76826,76828],{"className":13136,"code":76827,"language":12886,"meta":464,"style":464},"from utils import get_blog_posts\ndef home(request):\n    ...\n    posts = get_blog_posts(4)\n\n    context = {\n        ...\n        'recent_posts': posts,\n        ...\n    }\n\n    return render(request, 'home.html', context)\n",[30,76829,76830,76842,76855,76860,76874,76878,76886,76890,76898,76902,76906,76910],{"__ignoreMap":464},[151,76831,76832,76834,76837,76839],{"class":469,"line":470},[151,76833,16853],{"class":1869},[151,76835,76836],{"class":503}," utils ",[151,76838,16859],{"class":1869},[151,76840,76841],{"class":503}," get_blog_posts\n",[151,76843,76844,76846,76849,76851,76853],{"class":469,"line":488},[151,76845,16925],{"class":12347},[151,76847,76848],{"class":473}," home",[151,76850,12386],{"class":503},[151,76852,59686],{"class":15232},[151,76854,15264],{"class":503},[151,76856,76857],{"class":469,"line":500},[151,76858,76859],{"class":477},"    ...\n",[151,76861,76862,76865,76867,76870,76872],{"class":469,"line":509},[151,76863,76864],{"class":503},"    posts ",[151,76866,1876],{"class":1869},[151,76868,76869],{"class":503}," get_blog_posts(",[151,76871,9187],{"class":477},[151,76873,3640],{"class":503},[151,76875,76876],{"class":469,"line":517},[151,76877,1090],{"emptyLinePlaceholder":609},[151,76879,76880,76882,76884],{"class":469,"line":534},[151,76881,59844],{"class":503},[151,76883,1876],{"class":1869},[151,76885,19833],{"class":503},[151,76887,76888],{"class":469,"line":1413},[151,76889,15617],{"class":477},[151,76891,76892,76895],{"class":469,"line":1418},[151,76893,76894],{"class":481},"        'recent_posts'",[151,76896,76897],{"class":503},": posts,\n",[151,76899,76900],{"class":469,"line":2462},[151,76901,15617],{"class":477},[151,76903,76904],{"class":469,"line":2471},[151,76905,9461],{"class":503},[151,76907,76908],{"class":469,"line":2480},[151,76909,1090],{"emptyLinePlaceholder":609},[151,76911,76912,76914,76916,76919],{"class":469,"line":2489},[151,76913,17496],{"class":1869},[151,76915,59902],{"class":503},[151,76917,76918],{"class":481},"'home.html'",[151,76920,59908],{"class":503},[11,76922,76923],{},"In the context, we can access the following data for each item:",[1131,76925,76926,76934],{},[1134,76927,76928],{},[1137,76929,76930,76932],{},[1140,76931,8398],{},[1140,76933,19656],{},[1153,76935,76936,76946,76956,76966,76976,76986,76996],{},[1137,76937,76938,76943],{},[1158,76939,11601,76940],{},[151,76941,76942],{}," \"date\"",[1158,76944,76945],{},"\"2004-02-13T22:28:23+08:00\" - ISO 8601 date",[1137,76947,76948,76953],{},[1158,76949,11601,76950],{},[151,76951,76952],{}," \"date_parsed\"",[1158,76954,76955],{},"(2004,02,13,14,28,23,4,44,0)",[1137,76957,76958,76963],{},[1158,76959,11601,76960],{},[151,76961,76962],{}," \"title\"",[1158,76964,76965],{},"title for item",[1137,76967,76968,76973],{},[1158,76969,11601,76970],{},[151,76971,76972],{}," \"summary\"",[1158,76974,76975],{},"change summary",[1137,76977,76978,76983],{},[1158,76979,11601,76980],{},[151,76981,76982],{}," \"link\"",[1158,76984,76985],{},"URL to the page",[1137,76987,76988,76993],{},[1158,76989,11601,76990],{},[151,76991,76992],{}," \"wiki_diff\"",[1158,76994,76995],{},"for wiki, a link to the diff for the page",[1137,76997,76998,77003],{},[1158,76999,11601,77000],{},[151,77001,77002],{}," \"wiki_history\"",[1158,77004,77005],{},"for wiki, a link to the page history",[11,77007,77008],{},"That's it!",[589,77010,77011],{},"html pre.shiki code .sMOD_, html code.shiki .sMOD_{--shiki-default:#24292E;--shiki-dark:#E1E4E8;--shiki-sepia:#F8F8F2}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html .sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html.sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html pre.shiki code .sC2Qs, html code.shiki .sC2Qs{--shiki-default:#D73A49;--shiki-dark:#F97583;--shiki-sepia:#F92672}html pre.shiki code .sq6CD, html code.shiki .sq6CD{--shiki-default:#D73A49;--shiki-default-font-style:inherit;--shiki-dark:#F97583;--shiki-dark-font-style:inherit;--shiki-sepia:#66D9EF;--shiki-sepia-font-style:italic}html pre.shiki code .srTi1, html code.shiki .srTi1{--shiki-default:#6F42C1;--shiki-dark:#B392F0;--shiki-sepia:#A6E22E}html pre.shiki code .so59x, html code.shiki .so59x{--shiki-default:#24292E;--shiki-default-font-style:inherit;--shiki-dark:#E1E4E8;--shiki-dark-font-style:inherit;--shiki-sepia:#FD971F;--shiki-sepia-font-style:italic}html pre.shiki code .s7F3e, html code.shiki .s7F3e{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#AE81FF}html pre.shiki code .sstjo, html code.shiki .sstjo{--shiki-default:#032F62;--shiki-dark:#9ECBFF;--shiki-sepia:#E6DB74}",{"title":464,"searchDepth":488,"depth":488,"links":77013},[],"On my other personal website, briancaffey.com, I have a blog. The content on that blog has mostly mirrored what I put on this github pages site, briancaffey.github.io. I want to display my most recent blog posts from briancaffey.github.io on briancaffey.com, and to do this I will be using the RSS feed that comes with a Jekyll site. This should be pretty simple, we are going to use the requests librrary, as well as the feedparser library.","/static/python_rss.png",{"layout":48045},"/2017/10/17/reading-rss-with-python",{"title":76759,"description":77014},"2017/10/17/reading-rss-with-python",[12886,77021],"rss","WWdj5QDzDqVacz0cIkDp4dbC7PSdsO7qYlp-_7SX964",{"id":77024,"title":77025,"body":77026,"comments":609,"date":80994,"description":80995,"draft":602,"extension":605,"external":606,"image":78884,"meta":80996,"navigation":609,"path":80997,"seo":80998,"stem":80999,"tags":81000,"__hash__":81002},"blog/2017/10/03/simple-games-in-react.md","Simple Board Games in ReactJS",{"type":8,"value":77027,"toc":80992},[77028,77037,77040,77045,77051,77054,78874,78877,78880,78885,78890,78893,80690,80693,80980,80983,80986,80989],[11,77029,77030,77031,77036],{},"To ease into learning ReactJS, I took a shot at implementing a simple tic-tac-toe game with React. This is covered in the official ",[20,77032,77035],{"href":77033,"rel":77034},"https://reactjs.org/tutorial/tutorial.html",[24],"Facebook React tutorial",", but I haven't actually looked at how they did this yet. Instead, I wanted to see how far I could get on my own, and then fallback to the tutorial if I needed help. I heard that there are many different ways that React components can be organized and structured in a React project, so I wanted to see how my results compared to what the official tutorial recommends.",[11,77038,77039],{},"Here's the final result:",[11,77041,77042],{},[2718,77043],{"alt":20386,"src":77044},"/static/react/ttt.png",[11,77046,77047,77048],{},"You can play this game ",[20,77049,13074],{"href":77050,"target":59345},"/static/react/tic-tac-react/tic-tac-react.html",[11,77052,77053],{},"I wanted to give the game some additional features, so I let the player set the dimensions of the board to be any integer greater than 1. Here's a look at the main component called \"Board\" which contains most of the business logic:",[459,77055,77057],{"className":19459,"code":77056,"language":19461,"meta":464,"style":464},"{% raw %}import React from 'react';\nimport { grid } from '../data/grid.js';\nimport { Square } from './Square';\n\nexport class Board extends React.Component {\n  constructor(props){\n    super(props);\n    this.state = {\n      dim:3,\n      grid:Array(3).fill(0).map(x=>Array(3).fill(\"+\")),\n      player:'X',\n      winner:null,\n      active:true,\n    };\n    this.handleOnClick = this.handleOnClick.bind(this);\n    this.checkWins = this.checkWins.bind(this);\n    this.handleReset = this.handleReset.bind(this);\n    this.dims = [parseFloat(500/this.state.grid.length), parseFloat(500/this.state.grid[0].length)]\n  }\n\n  handleReset(){\n    const newGrid = Array(this.state.dim).fill(0).map(x=>Array(this.state.dim).fill(\"+\"))\n    this.setState({'grid':newGrid, 'player':'X'});\n  }\n\n  checkWins(x, y){\n    const g = this.state.grid\n\n    function checkDiagonal1(){\n      if (x == y){\n        const result = new Set(g.map((_, i)=>g[i][i]));\n        announceWin(result);\n      }\n    }\n\n    function checkDiagonal2(){\n      if (x+y+1 == g.length){\n        const result = new Set(g.map((_, i)=>g[i][g.length-1-i]))\n        announceWin(result);\n      }\n    }\n\n    function checkHorizontal(x){\n      const result = new Set(g[x]);\n      announceWin(result);\n    }\n\n    function checkVertical(y){\n      const result = new Set(g.map((x)=>x[y]));\n      announceWin(result);\n    }\n\n    function announceWin(l){\n      if (l.size == 1){\n        if (l.has(\"X\")){\n          setTimeout(()=>{alert(\"X wins\")}, 10);\n          return;\n        } else {\n          setTimeout(()=>{alert(\"O wins\")}, 10);\n          return;\n        }\n      }\n    }\n\n    checkDiagonal1();\n    checkDiagonal2();\n    checkHorizontal(x);\n    checkVertical(y);\n  }\n\n  handleOnClick(x, y){\n    const g = this.state.grid\n    if (this.state.active){\n      if (g[x][y] == '+'){\n        g[x][y] = this.state.player;\n        this.setState({'grid':g});\n        this.state.player = this.state.player == 'X' ? 'O':'X';\n        this.checkWins(x, y);\n    } else {\n      alert('Please select an empty square!');\n      }\n    }\n  }\n\n  render(){\n    const style = {\n      margin:'auto',\n      width: \"auto\",\n      height:\"auto\",\n      backgroundColor:'darkorange',\n      color:'white',\n      fontSize:\"3em\",\n      tableLayout:'fixed',\n    }\n    const rows = this.state.grid.map((r, i) => {return (\n      \u003Ctr key={\"row_\"+i}>\n        {r.map((d, j) => {console.log('building'); return(\n          \u003CSquare\n            key={i+\"_\"+j}\n            dims={this.dims}\n            onClick={()=>{this.handleOnClick(i,j)}}\n            contents={d==\"+\"?\" \":d} />\n              )\n            }\n          )\n        }\n        \u003C/tr>)\n        }\n      );\n    return (\n      \u003Cdiv style={{textAlign:\"center\"}}>\n        \u003Ch1>Tic-Tac-React!\u003C/h1>\n        \u003Csmall>tic-tac-toe, written with \u003Cb>ReactJS\u003C/b>. Enjoy!\u003C/small>\n        \u003Cp>Current Player: {this.state.player}\u003C/p>\n        \u003Ctable cellSpacing=\"0\" id=\"table\" style={style}>\n          \u003Ctbody>\n            {rows}\n          \u003C/tbody>\n        \u003C/table>\n        \u003Cbr />\n        \u003Cbutton style={{margin:\"auto\"}} onClick={this.handleReset}>reset\u003C/button>\n        \u003Cbr />\u003Cbr />\n        \u003Cbutton onClick={()=>{this.state.dim==1?1:this.state.dim-=1;this.setState({dim:this.state.dim})}}>-\u003C/button>\n\n               \u003Cspan style={{color:'white'}}>{this.state.dim}\u003C/span>   \n\n        \u003Cbutton onClick={()=>{this.state.dim+=1;this.setState({dim:this.state.dim})}}>+\u003C/button>\n        \u003Cbr />\u003Cbr/>\u003Cbr/>\n      \u003C/div>\n  )\n  }\n}\n{% endraw %}\n",[30,77058,77059,77084,77098,77112,77116,77137,77149,77157,77168,77177,77225,77235,77244,77253,77258,77281,77303,77325,77377,77381,77385,77393,77444,77471,77475,77479,77494,77508,77512,77522,77535,77571,77579,77584,77588,77592,77601,77625,77667,77673,77677,77681,77685,77698,77713,77720,77724,77728,77741,77768,77774,77778,77782,77796,77809,77827,77854,77861,77871,77894,77900,77904,77908,77912,77916,77923,77930,77938,77946,77950,77954,77969,77981,77992,78006,78018,78033,78062,78074,78083,78095,78099,78103,78107,78111,78118,78128,78138,78148,78157,78167,78176,78186,78196,78200,78233,78257,78298,78305,78327,78343,78371,78398,78403,78407,78411,78415,78424,78428,78433,78439,78463,78476,78501,78525,78558,78566,78576,78584,78592,78600,78642,78655,78723,78727,78770,78774,78823,78841,78849,78853,78857,78861],{"__ignoreMap":464},[151,77060,77061,77063,77065,77068,77070,77072,77074,77077,77079,77082],{"class":469,"line":470},[151,77062,5729],{"class":503},[151,77064,44519],{"class":1869},[151,77066,77067],{"class":503}," raw ",[151,77069,44519],{"class":1869},[151,77071,2001],{"class":503},[151,77073,16859],{"class":1869},[151,77075,77076],{"class":503}," React ",[151,77078,16853],{"class":1869},[151,77080,77081],{"class":481}," 'react'",[151,77083,20086],{"class":503},[151,77085,77086,77088,77091,77093,77096],{"class":469,"line":488},[151,77087,16859],{"class":1869},[151,77089,77090],{"class":503}," { grid } ",[151,77092,16853],{"class":1869},[151,77094,77095],{"class":481}," '../data/grid.js'",[151,77097,20086],{"class":503},[151,77099,77100,77102,77105,77107,77110],{"class":469,"line":500},[151,77101,16859],{"class":1869},[151,77103,77104],{"class":503}," { Square } ",[151,77106,16853],{"class":1869},[151,77108,77109],{"class":481}," './Square'",[151,77111,20086],{"class":503},[151,77113,77114],{"class":469,"line":509},[151,77115,1090],{"emptyLinePlaceholder":609},[151,77117,77118,77120,77122,77125,77128,77131,77133,77135],{"class":469,"line":517},[151,77119,1870],{"class":1869},[151,77121,48323],{"class":12347},[151,77123,77124],{"class":15254}," Board",[151,77126,77127],{"class":1869}," extends",[151,77129,77130],{"class":15254}," React",[151,77132,643],{"class":503},[151,77134,1584],{"class":15260},[151,77136,19833],{"class":503},[151,77138,77139,77142,77144,77146],{"class":469,"line":534},[151,77140,77141],{"class":12347},"  constructor",[151,77143,12386],{"class":503},[151,77145,27681],{"class":15210},[151,77147,77148],{"class":503},"){\n",[151,77150,77151,77154],{"class":469,"line":1413},[151,77152,77153],{"class":15289},"    super",[151,77155,77156],{"class":503},"(props);\n",[151,77158,77159,77161,77164,77166],{"class":469,"line":1418},[151,77160,27842],{"class":15289},[151,77162,77163],{"class":503},".state ",[151,77165,1876],{"class":1869},[151,77167,19833],{"class":503},[151,77169,77170,77173,77175],{"class":469,"line":2462},[151,77171,77172],{"class":503},"      dim:",[151,77174,6557],{"class":477},[151,77176,9417],{"class":503},[151,77178,77179,77182,77185,77187,77189,77191,77194,77196,77198,77200,77202,77204,77206,77208,77210,77212,77214,77216,77218,77220,77223],{"class":469,"line":2471},[151,77180,77181],{"class":503},"      grid:",[151,77183,77184],{"class":473},"Array",[151,77186,12386],{"class":503},[151,77188,6557],{"class":477},[151,77190,13576],{"class":503},[151,77192,77193],{"class":473},"fill",[151,77195,12386],{"class":503},[151,77197,9181],{"class":477},[151,77199,13576],{"class":503},[151,77201,68343],{"class":473},[151,77203,12386],{"class":503},[151,77205,11126],{"class":15210},[151,77207,17166],{"class":12347},[151,77209,77184],{"class":473},[151,77211,12386],{"class":503},[151,77213,6557],{"class":477},[151,77215,13576],{"class":503},[151,77217,77193],{"class":473},[151,77219,12386],{"class":503},[151,77221,77222],{"class":481},"\"+\"",[151,77224,57361],{"class":503},[151,77226,77227,77230,77233],{"class":469,"line":2480},[151,77228,77229],{"class":503},"      player:",[151,77231,77232],{"class":481},"'X'",[151,77234,9417],{"class":503},[151,77236,77237,77240,77242],{"class":469,"line":2489},[151,77238,77239],{"class":503},"      winner:",[151,77241,9824],{"class":477},[151,77243,9417],{"class":503},[151,77245,77246,77249,77251],{"class":469,"line":2497},[151,77247,77248],{"class":503},"      active:",[151,77250,19726],{"class":477},[151,77252,9417],{"class":503},[151,77254,77255],{"class":469,"line":3140},[151,77256,77257],{"class":503},"    };\n",[151,77259,77260,77262,77265,77267,77269,77272,77275,77277,77279],{"class":469,"line":3149},[151,77261,27842],{"class":15289},[151,77263,77264],{"class":503},".handleOnClick ",[151,77266,1876],{"class":1869},[151,77268,2324],{"class":15289},[151,77270,77271],{"class":503},".handleOnClick.",[151,77273,77274],{"class":473},"bind",[151,77276,12386],{"class":503},[151,77278,23252],{"class":15289},[151,77280,20129],{"class":503},[151,77282,77283,77285,77288,77290,77292,77295,77297,77299,77301],{"class":469,"line":3158},[151,77284,27842],{"class":15289},[151,77286,77287],{"class":503},".checkWins ",[151,77289,1876],{"class":1869},[151,77291,2324],{"class":15289},[151,77293,77294],{"class":503},".checkWins.",[151,77296,77274],{"class":473},[151,77298,12386],{"class":503},[151,77300,23252],{"class":15289},[151,77302,20129],{"class":503},[151,77304,77305,77307,77310,77312,77314,77317,77319,77321,77323],{"class":469,"line":3167},[151,77306,27842],{"class":15289},[151,77308,77309],{"class":503},".handleReset ",[151,77311,1876],{"class":1869},[151,77313,2324],{"class":15289},[151,77315,77316],{"class":503},".handleReset.",[151,77318,77274],{"class":473},[151,77320,12386],{"class":503},[151,77322,23252],{"class":15289},[151,77324,20129],{"class":503},[151,77326,77327,77329,77332,77334,77336,77339,77341,77343,77345,77347,77350,77353,77355,77357,77359,77361,77363,77365,77368,77370,77373,77375],{"class":469,"line":3175},[151,77328,27842],{"class":15289},[151,77330,77331],{"class":503},".dims ",[151,77333,1876],{"class":1869},[151,77335,6604],{"class":503},[151,77337,77338],{"class":473},"parseFloat",[151,77340,12386],{"class":503},[151,77342,12208],{"class":477},[151,77344,19883],{"class":1869},[151,77346,23252],{"class":15289},[151,77348,77349],{"class":503},".state.grid.",[151,77351,77352],{"class":12360},"length",[151,77354,24817],{"class":503},[151,77356,77338],{"class":473},[151,77358,12386],{"class":503},[151,77360,12208],{"class":477},[151,77362,19883],{"class":1869},[151,77364,23252],{"class":15289},[151,77366,77367],{"class":503},".state.grid[",[151,77369,9181],{"class":477},[151,77371,77372],{"class":503},"].",[151,77374,77352],{"class":12360},[151,77376,44576],{"class":503},[151,77378,77379],{"class":469,"line":3184},[151,77380,19957],{"class":503},[151,77382,77383],{"class":469,"line":3193},[151,77384,1090],{"emptyLinePlaceholder":609},[151,77386,77387,77390],{"class":469,"line":3720},[151,77388,77389],{"class":473},"  handleReset",[151,77391,77392],{"class":503},"(){\n",[151,77394,77395,77397,77400,77402,77405,77407,77409,77412,77414,77416,77418,77420,77422,77424,77426,77428,77430,77432,77434,77436,77438,77440,77442],{"class":469,"line":3729},[151,77396,19860],{"class":12347},[151,77398,77399],{"class":12360}," newGrid",[151,77401,19865],{"class":1869},[151,77403,77404],{"class":473}," Array",[151,77406,12386],{"class":503},[151,77408,23252],{"class":15289},[151,77410,77411],{"class":503},".state.dim).",[151,77413,77193],{"class":473},[151,77415,12386],{"class":503},[151,77417,9181],{"class":477},[151,77419,13576],{"class":503},[151,77421,68343],{"class":473},[151,77423,12386],{"class":503},[151,77425,11126],{"class":15210},[151,77427,17166],{"class":12347},[151,77429,77184],{"class":473},[151,77431,12386],{"class":503},[151,77433,23252],{"class":15289},[151,77435,77411],{"class":503},[151,77437,77193],{"class":473},[151,77439,12386],{"class":503},[151,77441,77222],{"class":481},[151,77443,12451],{"class":503},[151,77445,77446,77448,77450,77453,77456,77459,77462,77465,77467,77469],{"class":469,"line":3735},[151,77447,27842],{"class":15289},[151,77449,643],{"class":503},[151,77451,77452],{"class":473},"setState",[151,77454,77455],{"class":503},"({",[151,77457,77458],{"class":481},"'grid'",[151,77460,77461],{"class":503},":newGrid, ",[151,77463,77464],{"class":481},"'player'",[151,77466,208],{"class":503},[151,77468,77232],{"class":481},[151,77470,20850],{"class":503},[151,77472,77473],{"class":469,"line":3745},[151,77474,19957],{"class":503},[151,77476,77477],{"class":469,"line":3754},[151,77478,1090],{"emptyLinePlaceholder":609},[151,77480,77481,77484,77486,77488,77490,77492],{"class":469,"line":3760},[151,77482,77483],{"class":473},"  checkWins",[151,77485,12386],{"class":503},[151,77487,11126],{"class":15210},[151,77489,106],{"class":503},[151,77491,25286],{"class":15210},[151,77493,77148],{"class":503},[151,77495,77496,77498,77501,77503,77505],{"class":469,"line":3773},[151,77497,19860],{"class":12347},[151,77499,77500],{"class":12360}," g",[151,77502,19865],{"class":1869},[151,77504,2324],{"class":15289},[151,77506,77507],{"class":503},".state.grid\n",[151,77509,77510],{"class":469,"line":3782},[151,77511,1090],{"emptyLinePlaceholder":609},[151,77513,77514,77517,77520],{"class":469,"line":3791},[151,77515,77516],{"class":12347},"    function",[151,77518,77519],{"class":473}," checkDiagonal1",[151,77521,77392],{"class":503},[151,77523,77524,77527,77530,77532],{"class":469,"line":3803},[151,77525,77526],{"class":1869},"      if",[151,77528,77529],{"class":503}," (x ",[151,77531,17223],{"class":1869},[151,77533,77534],{"class":503}," y){\n",[151,77536,77537,77540,77543,77545,77547,77550,77553,77555,77557,77559,77561,77564,77566,77568],{"class":469,"line":3811},[151,77538,77539],{"class":12347},"        const",[151,77541,77542],{"class":12360}," result",[151,77544,19865],{"class":1869},[151,77546,4236],{"class":1869},[151,77548,77549],{"class":473}," Set",[151,77551,77552],{"class":503},"(g.",[151,77554,68343],{"class":473},[151,77556,34211],{"class":503},[151,77558,34214],{"class":15210},[151,77560,106],{"class":503},[151,77562,77563],{"class":15210},"i",[151,77565,748],{"class":503},[151,77567,17166],{"class":12347},[151,77569,77570],{"class":503},"g[i][i]));\n",[151,77572,77573,77576],{"class":469,"line":3820},[151,77574,77575],{"class":473},"        announceWin",[151,77577,77578],{"class":503},"(result);\n",[151,77580,77581],{"class":469,"line":7084},[151,77582,77583],{"class":503},"      }\n",[151,77585,77586],{"class":469,"line":7148},[151,77587,9461],{"class":503},[151,77589,77590],{"class":469,"line":7211},[151,77591,1090],{"emptyLinePlaceholder":609},[151,77593,77594,77596,77599],{"class":469,"line":7273},[151,77595,77516],{"class":12347},[151,77597,77598],{"class":473}," checkDiagonal2",[151,77600,77392],{"class":503},[151,77602,77603,77605,77608,77610,77612,77614,77616,77618,77621,77623],{"class":469,"line":7335},[151,77604,77526],{"class":1869},[151,77606,77607],{"class":503}," (x",[151,77609,22885],{"class":1869},[151,77611,25286],{"class":503},[151,77613,22885],{"class":1869},[151,77615,6760],{"class":477},[151,77617,17288],{"class":1869},[151,77619,77620],{"class":503}," g.",[151,77622,77352],{"class":12360},[151,77624,77148],{"class":503},[151,77626,77627,77629,77631,77633,77635,77637,77639,77641,77643,77645,77647,77649,77651,77653,77656,77658,77660,77662,77664],{"class":469,"line":7398},[151,77628,77539],{"class":12347},[151,77630,77542],{"class":12360},[151,77632,19865],{"class":1869},[151,77634,4236],{"class":1869},[151,77636,77549],{"class":473},[151,77638,77552],{"class":503},[151,77640,68343],{"class":473},[151,77642,34211],{"class":503},[151,77644,34214],{"class":15210},[151,77646,106],{"class":503},[151,77648,77563],{"class":15210},[151,77650,748],{"class":503},[151,77652,17166],{"class":12347},[151,77654,77655],{"class":503},"g[i][g.",[151,77657,77352],{"class":12360},[151,77659,12445],{"class":1869},[151,77661,6760],{"class":477},[151,77663,12445],{"class":1869},[151,77665,77666],{"class":503},"i]))\n",[151,77668,77669,77671],{"class":469,"line":7462},[151,77670,77575],{"class":473},[151,77672,77578],{"class":503},[151,77674,77675],{"class":469,"line":7467},[151,77676,77583],{"class":503},[151,77678,77679],{"class":469,"line":7532},[151,77680,9461],{"class":503},[151,77682,77683],{"class":469,"line":7537},[151,77684,1090],{"emptyLinePlaceholder":609},[151,77686,77687,77689,77692,77694,77696],{"class":469,"line":7603},[151,77688,77516],{"class":12347},[151,77690,77691],{"class":473}," checkHorizontal",[151,77693,12386],{"class":503},[151,77695,11126],{"class":15210},[151,77697,77148],{"class":503},[151,77699,77700,77702,77704,77706,77708,77710],{"class":469,"line":7608},[151,77701,34174],{"class":12347},[151,77703,77542],{"class":12360},[151,77705,19865],{"class":1869},[151,77707,4236],{"class":1869},[151,77709,77549],{"class":473},[151,77711,77712],{"class":503},"(g[x]);\n",[151,77714,77715,77718],{"class":469,"line":7673},[151,77716,77717],{"class":473},"      announceWin",[151,77719,77578],{"class":503},[151,77721,77722],{"class":469,"line":7678},[151,77723,9461],{"class":503},[151,77725,77726],{"class":469,"line":7708},[151,77727,1090],{"emptyLinePlaceholder":609},[151,77729,77730,77732,77735,77737,77739],{"class":469,"line":7713},[151,77731,77516],{"class":12347},[151,77733,77734],{"class":473}," checkVertical",[151,77736,12386],{"class":503},[151,77738,25286],{"class":15210},[151,77740,77148],{"class":503},[151,77742,77743,77745,77747,77749,77751,77753,77755,77757,77759,77761,77763,77765],{"class":469,"line":7746},[151,77744,34174],{"class":12347},[151,77746,77542],{"class":12360},[151,77748,19865],{"class":1869},[151,77750,4236],{"class":1869},[151,77752,77549],{"class":473},[151,77754,77552],{"class":503},[151,77756,68343],{"class":473},[151,77758,34211],{"class":503},[151,77760,11126],{"class":15210},[151,77762,748],{"class":503},[151,77764,17166],{"class":12347},[151,77766,77767],{"class":503},"x[y]));\n",[151,77769,77770,77772],{"class":469,"line":7751},[151,77771,77717],{"class":473},[151,77773,77578],{"class":503},[151,77775,77776],{"class":469,"line":7816},[151,77777,9461],{"class":503},[151,77779,77780],{"class":469,"line":7821},[151,77781,1090],{"emptyLinePlaceholder":609},[151,77783,77784,77786,77789,77791,77794],{"class":469,"line":7847},[151,77785,77516],{"class":12347},[151,77787,77788],{"class":473}," announceWin",[151,77790,12386],{"class":503},[151,77792,77793],{"class":15210},"l",[151,77795,77148],{"class":503},[151,77797,77798,77800,77803,77805,77807],{"class":469,"line":7852},[151,77799,77526],{"class":1869},[151,77801,77802],{"class":503}," (l.size ",[151,77804,17223],{"class":1869},[151,77806,12448],{"class":477},[151,77808,77148],{"class":503},[151,77810,77811,77813,77816,77819,77821,77824],{"class":469,"line":7887},[151,77812,23357],{"class":1869},[151,77814,77815],{"class":503}," (l.",[151,77817,77818],{"class":473},"has",[151,77820,12386],{"class":503},[151,77822,77823],{"class":481},"\"X\"",[151,77825,77826],{"class":503},")){\n",[151,77828,77829,77832,77835,77837,77839,77842,77844,77847,77850,77852],{"class":469,"line":7892},[151,77830,77831],{"class":473},"          setTimeout",[151,77833,77834],{"class":503},"(()",[151,77836,17166],{"class":12347},[151,77838,5729],{"class":503},[151,77840,77841],{"class":473},"alert",[151,77843,12386],{"class":503},[151,77845,77846],{"class":481},"\"X wins\"",[151,77848,77849],{"class":503},")}, ",[151,77851,12423],{"class":477},[151,77853,20129],{"class":503},[151,77855,77856,77859],{"class":469,"line":7924},[151,77857,77858],{"class":1869},"          return",[151,77860,20086],{"class":503},[151,77862,77863,77866,77869],{"class":469,"line":7929},[151,77864,77865],{"class":503},"        } ",[151,77867,77868],{"class":1869},"else",[151,77870,19833],{"class":503},[151,77872,77873,77875,77877,77879,77881,77883,77885,77888,77890,77892],{"class":469,"line":7991},[151,77874,77831],{"class":473},[151,77876,77834],{"class":503},[151,77878,17166],{"class":12347},[151,77880,5729],{"class":503},[151,77882,77841],{"class":473},[151,77884,12386],{"class":503},[151,77886,77887],{"class":481},"\"O wins\"",[151,77889,77849],{"class":503},[151,77891,12423],{"class":477},[151,77893,20129],{"class":503},[151,77895,77896,77898],{"class":469,"line":7996},[151,77897,77858],{"class":1869},[151,77899,20086],{"class":503},[151,77901,77902],{"class":469,"line":8078},[151,77903,23390],{"class":503},[151,77905,77906],{"class":469,"line":8140},[151,77907,77583],{"class":503},[151,77909,77910],{"class":469,"line":8145},[151,77911,9461],{"class":503},[151,77913,77914],{"class":469,"line":8259},[151,77915,1090],{"emptyLinePlaceholder":609},[151,77917,77918,77921],{"class":469,"line":8264},[151,77919,77920],{"class":473},"    checkDiagonal1",[151,77922,20012],{"class":503},[151,77924,77925,77928],{"class":469,"line":8613},[151,77926,77927],{"class":473},"    checkDiagonal2",[151,77929,20012],{"class":503},[151,77931,77932,77935],{"class":469,"line":8678},[151,77933,77934],{"class":473},"    checkHorizontal",[151,77936,77937],{"class":503},"(x);\n",[151,77939,77940,77943],{"class":469,"line":8742},[151,77941,77942],{"class":473},"    checkVertical",[151,77944,77945],{"class":503},"(y);\n",[151,77947,77948],{"class":469,"line":8806},[151,77949,19957],{"class":503},[151,77951,77952],{"class":469,"line":8870},[151,77953,1090],{"emptyLinePlaceholder":609},[151,77955,77956,77959,77961,77963,77965,77967],{"class":469,"line":8875},[151,77957,77958],{"class":473},"  handleOnClick",[151,77960,12386],{"class":503},[151,77962,11126],{"class":15210},[151,77964,106],{"class":503},[151,77966,25286],{"class":15210},[151,77968,77148],{"class":503},[151,77970,77971,77973,77975,77977,77979],{"class":469,"line":8881},[151,77972,19860],{"class":12347},[151,77974,77500],{"class":12360},[151,77976,19865],{"class":1869},[151,77978,2324],{"class":15289},[151,77980,77507],{"class":503},[151,77982,77983,77985,77987,77989],{"class":469,"line":8886},[151,77984,23327],{"class":1869},[151,77986,129],{"class":503},[151,77988,23252],{"class":15289},[151,77990,77991],{"class":503},".state.active){\n",[151,77993,77994,77996,77999,78001,78004],{"class":469,"line":8892},[151,77995,77526],{"class":1869},[151,77997,77998],{"class":503}," (g[x][y] ",[151,78000,17223],{"class":1869},[151,78002,78003],{"class":481}," '+'",[151,78005,77148],{"class":503},[151,78007,78008,78011,78013,78015],{"class":469,"line":8963},[151,78009,78010],{"class":503},"        g[x][y] ",[151,78012,1876],{"class":1869},[151,78014,2324],{"class":15289},[151,78016,78017],{"class":503},".state.player;\n",[151,78019,78020,78022,78024,78026,78028,78030],{"class":469,"line":8969},[151,78021,34367],{"class":15289},[151,78023,643],{"class":503},[151,78025,77452],{"class":473},[151,78027,77455],{"class":503},[151,78029,77458],{"class":481},[151,78031,78032],{"class":503},":g});\n",[151,78034,78035,78037,78040,78042,78044,78046,78048,78051,78053,78056,78058,78060],{"class":469,"line":15001},[151,78036,34367],{"class":15289},[151,78038,78039],{"class":503},".state.player ",[151,78041,1876],{"class":1869},[151,78043,2324],{"class":15289},[151,78045,78039],{"class":503},[151,78047,17223],{"class":1869},[151,78049,78050],{"class":481}," 'X'",[151,78052,75197],{"class":1869},[151,78054,78055],{"class":481}," 'O'",[151,78057,208],{"class":1869},[151,78059,77232],{"class":481},[151,78061,20086],{"class":503},[151,78063,78064,78066,78068,78071],{"class":469,"line":15009},[151,78065,34367],{"class":15289},[151,78067,643],{"class":503},[151,78069,78070],{"class":473},"checkWins",[151,78072,78073],{"class":503},"(x, y);\n",[151,78075,78076,78079,78081],{"class":469,"line":15019},[151,78077,78078],{"class":503},"    } ",[151,78080,77868],{"class":1869},[151,78082,19833],{"class":503},[151,78084,78085,78088,78090,78093],{"class":469,"line":15027},[151,78086,78087],{"class":473},"      alert",[151,78089,12386],{"class":503},[151,78091,78092],{"class":481},"'Please select an empty square!'",[151,78094,20129],{"class":503},[151,78096,78097],{"class":469,"line":15037},[151,78098,77583],{"class":503},[151,78100,78101],{"class":469,"line":15045},[151,78102,9461],{"class":503},[151,78104,78105],{"class":469,"line":15055},[151,78106,19957],{"class":503},[151,78108,78109],{"class":469,"line":15060},[151,78110,1090],{"emptyLinePlaceholder":609},[151,78112,78113,78116],{"class":469,"line":15068},[151,78114,78115],{"class":473},"  render",[151,78117,77392],{"class":503},[151,78119,78120,78122,78124,78126],{"class":469,"line":15076},[151,78121,19860],{"class":12347},[151,78123,48548],{"class":12360},[151,78125,19865],{"class":1869},[151,78127,19833],{"class":503},[151,78129,78130,78133,78136],{"class":469,"line":15085},[151,78131,78132],{"class":503},"      margin:",[151,78134,78135],{"class":481},"'auto'",[151,78137,9417],{"class":503},[151,78139,78140,78143,78146],{"class":469,"line":15095},[151,78141,78142],{"class":503},"      width: ",[151,78144,78145],{"class":481},"\"auto\"",[151,78147,9417],{"class":503},[151,78149,78150,78153,78155],{"class":469,"line":15105},[151,78151,78152],{"class":503},"      height:",[151,78154,78145],{"class":481},[151,78156,9417],{"class":503},[151,78158,78159,78162,78165],{"class":469,"line":15110},[151,78160,78161],{"class":503},"      backgroundColor:",[151,78163,78164],{"class":481},"'darkorange'",[151,78166,9417],{"class":503},[151,78168,78169,78172,78174],{"class":469,"line":15118},[151,78170,78171],{"class":503},"      color:",[151,78173,45309],{"class":481},[151,78175,9417],{"class":503},[151,78177,78178,78181,78184],{"class":469,"line":15128},[151,78179,78180],{"class":503},"      fontSize:",[151,78182,78183],{"class":481},"\"3em\"",[151,78185,9417],{"class":503},[151,78187,78188,78191,78194],{"class":469,"line":15139},[151,78189,78190],{"class":503},"      tableLayout:",[151,78192,78193],{"class":481},"'fixed'",[151,78195,9417],{"class":503},[151,78197,78198],{"class":469,"line":31954},[151,78199,9461],{"class":503},[151,78201,78202,78204,78207,78209,78211,78213,78215,78217,78219,78221,78223,78225,78227,78229,78231],{"class":469,"line":31960},[151,78203,19860],{"class":12347},[151,78205,78206],{"class":12360}," rows",[151,78208,19865],{"class":1869},[151,78210,2324],{"class":15289},[151,78212,77349],{"class":503},[151,78214,68343],{"class":473},[151,78216,34211],{"class":503},[151,78218,58741],{"class":15210},[151,78220,106],{"class":503},[151,78222,77563],{"class":15210},[151,78224,16995],{"class":503},[151,78226,17166],{"class":12347},[151,78228,52023],{"class":503},[151,78230,63121],{"class":1869},[151,78232,37723],{"class":503},[151,78234,78235,78237,78239,78241,78243,78246,78249,78251,78253,78255],{"class":469,"line":31965},[151,78236,48222],{"class":503},[151,78238,1137],{"class":14368},[151,78240,2176],{"class":473},[151,78242,1876],{"class":1869},[151,78244,5729],{"class":78245},"s-ngx",[151,78247,78248],{"class":481},"\"row_\"",[151,78250,22885],{"class":1869},[151,78252,77563],{"class":503},[151,78254,2001],{"class":78245},[151,78256,3742],{"class":503},[151,78258,78259,78262,78265,78267,78269,78272,78274,78277,78279,78281,78284,78286,78288,78291,78294,78296],{"class":469,"line":31971},[151,78260,78261],{"class":78245},"        {",[151,78263,78264],{"class":503},"r.",[151,78266,68343],{"class":473},[151,78268,34211],{"class":503},[151,78270,78271],{"class":15210},"d",[151,78273,106],{"class":503},[151,78275,78276],{"class":15210},"j",[151,78278,16995],{"class":503},[151,78280,17166],{"class":12347},[151,78282,78283],{"class":503}," {console.",[151,78285,70339],{"class":473},[151,78287,12386],{"class":503},[151,78289,78290],{"class":481},"'building'",[151,78292,78293],{"class":503},"); ",[151,78295,63121],{"class":1869},[151,78297,15410],{"class":503},[151,78299,78300,78302],{"class":469,"line":31983},[151,78301,48318],{"class":503},[151,78303,78304],{"class":6205},"Square\n",[151,78306,78307,78310,78312,78314,78316,78318,78321,78323,78325],{"class":469,"line":31994},[151,78308,78309],{"class":473},"            key",[151,78311,1876],{"class":1869},[151,78313,5729],{"class":78245},[151,78315,77563],{"class":503},[151,78317,22885],{"class":1869},[151,78319,78320],{"class":481},"\"_\"",[151,78322,22885],{"class":1869},[151,78324,78276],{"class":503},[151,78326,6274],{"class":78245},[151,78328,78329,78332,78334,78336,78338,78341],{"class":469,"line":32007},[151,78330,78331],{"class":473},"            dims",[151,78333,1876],{"class":1869},[151,78335,5729],{"class":78245},[151,78337,23252],{"class":15289},[151,78339,78340],{"class":503},".dims",[151,78342,6274],{"class":78245},[151,78344,78345,78348,78350,78352,78355,78357,78359,78361,78363,78366,78369],{"class":469,"line":32018},[151,78346,78347],{"class":473},"            onClick",[151,78349,1876],{"class":1869},[151,78351,5729],{"class":78245},[151,78353,78354],{"class":503},"()",[151,78356,17166],{"class":12347},[151,78358,5729],{"class":503},[151,78360,23252],{"class":15289},[151,78362,643],{"class":503},[151,78364,78365],{"class":473},"handleOnClick",[151,78367,78368],{"class":503},"(i,j)}",[151,78370,6274],{"class":78245},[151,78372,78373,78376,78378,78380,78382,78384,78386,78388,78390,78392,78394,78396],{"class":469,"line":32026},[151,78374,78375],{"class":473},"            contents",[151,78377,1876],{"class":1869},[151,78379,5729],{"class":78245},[151,78381,78271],{"class":503},[151,78383,17223],{"class":1869},[151,78385,77222],{"class":481},[151,78387,10727],{"class":1869},[151,78389,24311],{"class":481},[151,78391,208],{"class":1869},[151,78393,78271],{"class":503},[151,78395,2001],{"class":78245},[151,78397,34675],{"class":503},[151,78399,78400],{"class":469,"line":32031},[151,78401,78402],{"class":503},"              )\n",[151,78404,78405],{"class":469,"line":32036},[151,78406,39498],{"class":503},[151,78408,78409],{"class":469,"line":32042},[151,78410,27913],{"class":503},[151,78412,78413],{"class":469,"line":32054},[151,78414,23390],{"class":78245},[151,78416,78417,78419,78421],{"class":469,"line":32067},[151,78418,21175],{"class":503},[151,78420,1137],{"class":14368},[151,78422,78423],{"class":503},">)\n",[151,78425,78426],{"class":469,"line":32086},[151,78427,23390],{"class":503},[151,78429,78430],{"class":469,"line":32097},[151,78431,78432],{"class":503},"      );\n",[151,78434,78435,78437],{"class":469,"line":25585},[151,78436,17496],{"class":1869},[151,78438,37723],{"class":503},[151,78440,78441,78443,78445,78447,78449,78451,78454,78457,78459,78461],{"class":469,"line":32112},[151,78442,48222],{"class":503},[151,78444,23950],{"class":14368},[151,78446,48548],{"class":473},[151,78448,1876],{"class":1869},[151,78450,5729],{"class":78245},[151,78452,78453],{"class":503},"{textAlign:",[151,78455,78456],{"class":481},"\"center\"",[151,78458,2001],{"class":503},[151,78460,2001],{"class":78245},[151,78462,3742],{"class":503},[151,78464,78465,78467,78469,78472,78474],{"class":469,"line":32117},[151,78466,21070],{"class":503},[151,78468,14063],{"class":14368},[151,78470,78471],{"class":503},">Tic-Tac-React!\u003C/",[151,78473,14063],{"class":14368},[151,78475,3742],{"class":503},[151,78477,78478,78480,78483,78486,78489,78492,78494,78497,78499],{"class":469,"line":32123},[151,78479,21070],{"class":503},[151,78481,78482],{"class":14368},"small",[151,78484,78485],{"class":503},">tic-tac-toe, written with \u003C",[151,78487,78488],{"class":14368},"b",[151,78490,78491],{"class":503},">ReactJS\u003C/",[151,78493,78488],{"class":14368},[151,78495,78496],{"class":503},">. Enjoy!\u003C/",[151,78498,78482],{"class":14368},[151,78500,3742],{"class":503},[151,78502,78503,78505,78507,78510,78512,78514,78517,78519,78521,78523],{"class":469,"line":32151},[151,78504,21070],{"class":503},[151,78506,11],{"class":14368},[151,78508,78509],{"class":503},">Current Player: ",[151,78511,5729],{"class":78245},[151,78513,23252],{"class":15289},[151,78515,78516],{"class":503},".state.player",[151,78518,2001],{"class":78245},[151,78520,19966],{"class":503},[151,78522,11],{"class":14368},[151,78524,3742],{"class":503},[151,78526,78527,78529,78531,78534,78536,78539,78541,78543,78546,78548,78550,78552,78554,78556],{"class":469,"line":32156},[151,78528,21070],{"class":503},[151,78530,1131],{"class":14368},[151,78532,78533],{"class":473}," cellSpacing",[151,78535,1876],{"class":1869},[151,78537,78538],{"class":481},"\"0\"",[151,78540,48210],{"class":473},[151,78542,1876],{"class":1869},[151,78544,78545],{"class":481},"\"table\"",[151,78547,48548],{"class":473},[151,78549,1876],{"class":1869},[151,78551,5729],{"class":78245},[151,78553,589],{"class":503},[151,78555,2001],{"class":78245},[151,78557,3742],{"class":503},[151,78559,78560,78562,78564],{"class":469,"line":32162},[151,78561,48318],{"class":503},[151,78563,1153],{"class":14368},[151,78565,3742],{"class":503},[151,78567,78568,78571,78574],{"class":469,"line":32168},[151,78569,78570],{"class":78245},"            {",[151,78572,78573],{"class":503},"rows",[151,78575,6274],{"class":78245},[151,78577,78578,78580,78582],{"class":469,"line":32180},[151,78579,48417],{"class":503},[151,78581,1153],{"class":14368},[151,78583,3742],{"class":503},[151,78585,78586,78588,78590],{"class":469,"line":32192},[151,78587,21175],{"class":503},[151,78589,1131],{"class":14368},[151,78591,3742],{"class":503},[151,78593,78594,78596,78598],{"class":469,"line":32207},[151,78595,21070],{"class":503},[151,78597,1205],{"class":14368},[151,78599,34675],{"class":503},[151,78601,78602,78604,78606,78608,78610,78612,78615,78617,78619,78621,78624,78626,78628,78630,78633,78635,78638,78640],{"class":469,"line":32217},[151,78603,21070],{"class":503},[151,78605,63037],{"class":14368},[151,78607,48548],{"class":473},[151,78609,1876],{"class":1869},[151,78611,5729],{"class":78245},[151,78613,78614],{"class":503},"{margin:",[151,78616,78145],{"class":481},[151,78618,2001],{"class":503},[151,78620,2001],{"class":78245},[151,78622,78623],{"class":473}," onClick",[151,78625,1876],{"class":1869},[151,78627,5729],{"class":78245},[151,78629,23252],{"class":15289},[151,78631,78632],{"class":503},".handleReset",[151,78634,2001],{"class":78245},[151,78636,78637],{"class":503},">reset\u003C/",[151,78639,63037],{"class":14368},[151,78641,3742],{"class":503},[151,78643,78644,78646,78648,78651,78653],{"class":469,"line":32226},[151,78645,21070],{"class":503},[151,78647,1205],{"class":14368},[151,78649,78650],{"class":503}," />\u003C",[151,78652,1205],{"class":14368},[151,78654,34675],{"class":503},[151,78656,78657,78659,78661,78663,78665,78667,78669,78671,78673,78675,78678,78680,78682,78684,78686,78688,78690,78692,78695,78697,78700,78702,78704,78706,78709,78711,78714,78716,78719,78721],{"class":469,"line":32231},[151,78658,21070],{"class":503},[151,78660,63037],{"class":14368},[151,78662,78623],{"class":473},[151,78664,1876],{"class":1869},[151,78666,5729],{"class":78245},[151,78668,78354],{"class":503},[151,78670,17166],{"class":12347},[151,78672,5729],{"class":503},[151,78674,23252],{"class":15289},[151,78676,78677],{"class":503},".state.dim",[151,78679,17223],{"class":1869},[151,78681,6760],{"class":477},[151,78683,10727],{"class":1869},[151,78685,6760],{"class":477},[151,78687,208],{"class":1869},[151,78689,23252],{"class":15289},[151,78691,78677],{"class":503},[151,78693,78694],{"class":1869},"-=",[151,78696,6760],{"class":477},[151,78698,78699],{"class":503},";",[151,78701,23252],{"class":15289},[151,78703,643],{"class":503},[151,78705,77452],{"class":473},[151,78707,78708],{"class":503},"({dim:",[151,78710,23252],{"class":15289},[151,78712,78713],{"class":503},".state.dim})}",[151,78715,2001],{"class":78245},[151,78717,78718],{"class":503},">-\u003C/",[151,78720,63037],{"class":14368},[151,78722,3742],{"class":503},[151,78724,78725],{"class":469,"line":32236},[151,78726,1090],{"emptyLinePlaceholder":609},[151,78728,78729,78732,78734,78736,78738,78740,78742,78745,78747,78749,78751,78753,78755,78757,78759,78761,78763,78765,78767],{"class":469,"line":32244},[151,78730,78731],{"class":477},"               ",[151,78733,3613],{"class":503},[151,78735,151],{"class":14368},[151,78737,48548],{"class":473},[151,78739,1876],{"class":1869},[151,78741,5729],{"class":78245},[151,78743,78744],{"class":503},"{color:",[151,78746,45309],{"class":481},[151,78748,2001],{"class":503},[151,78750,2001],{"class":78245},[151,78752,3663],{"class":503},[151,78754,5729],{"class":78245},[151,78756,23252],{"class":15289},[151,78758,78677],{"class":503},[151,78760,2001],{"class":78245},[151,78762,19966],{"class":503},[151,78764,151],{"class":14368},[151,78766,3663],{"class":503},[151,78768,78769],{"class":477},"   \n",[151,78771,78772],{"class":469,"line":32249},[151,78773,1090],{"emptyLinePlaceholder":609},[151,78775,78776,78778,78780,78782,78784,78786,78788,78790,78792,78794,78796,78798,78800,78802,78804,78806,78808,78810,78812,78814,78816,78819,78821],{"class":469,"line":32255},[151,78777,21070],{"class":503},[151,78779,63037],{"class":14368},[151,78781,78623],{"class":473},[151,78783,1876],{"class":1869},[151,78785,5729],{"class":78245},[151,78787,78354],{"class":503},[151,78789,17166],{"class":12347},[151,78791,5729],{"class":503},[151,78793,23252],{"class":15289},[151,78795,78677],{"class":503},[151,78797,24780],{"class":1869},[151,78799,6760],{"class":477},[151,78801,78699],{"class":503},[151,78803,23252],{"class":15289},[151,78805,643],{"class":503},[151,78807,77452],{"class":473},[151,78809,78708],{"class":503},[151,78811,23252],{"class":15289},[151,78813,78713],{"class":503},[151,78815,2001],{"class":78245},[151,78817,78818],{"class":503},">+\u003C/",[151,78820,63037],{"class":14368},[151,78822,3742],{"class":503},[151,78824,78825,78827,78829,78831,78833,78836,78838],{"class":469,"line":32272},[151,78826,21070],{"class":503},[151,78828,1205],{"class":14368},[151,78830,78650],{"class":503},[151,78832,1205],{"class":14368},[151,78834,78835],{"class":503},"/>\u003C",[151,78837,1205],{"class":14368},[151,78839,78840],{"class":503},"/>\n",[151,78842,78843,78845,78847],{"class":469,"line":32277},[151,78844,48707],{"class":503},[151,78846,23950],{"class":14368},[151,78848,3742],{"class":503},[151,78850,78851],{"class":469,"line":32283},[151,78852,31481],{"class":503},[151,78854,78855],{"class":469,"line":32295},[151,78856,19957],{"class":503},[151,78858,78859],{"class":469,"line":32307},[151,78860,6274],{"class":503},[151,78862,78863,78865,78867,78870,78872],{"class":469,"line":32320},[151,78864,5729],{"class":503},[151,78866,44519],{"class":1869},[151,78868,78869],{"class":503}," endraw ",[151,78871,44519],{"class":1869},[151,78873,6274],{"class":503},[11,78875,78876],{},"I had so much fun putting together this tic-tac-toe app that I decided to write another one of my favorite games called Gomoku. This game is somwhat similar to tic-tac-toe, but the objective is to connect 5 stones of the same color on a much larger board.",[11,78878,78879],{},"Here's a look at the result of my Gomoku game:",[11,78881,78882],{},[2718,78883],{"alt":20386,"src":78884},"/static/react/wuziqi.png",[11,78886,77047,78887],{},[20,78888,13074],{"href":78889,"target":59345},"/static/react/wuziqi/wuziqi.html",[11,78891,78892],{},"Here's the main component for this game with heavy commenting:",[459,78894,78896],{"className":19459,"code":78895,"language":19461,"meta":464,"style":464},"{% raw %}//import React and Square component\nimport React from 'react';\nimport { Square } from './Square';\nimport { Button } from './Button';\n\n//main board component with game logic\nexport class Board extends React.Component{\n  constructor(props){\n    super(props);\n    this.state = {\n      //white goes first\n      'isWhite':true,\n      //this sets up an empty board\n      //\"+\"\" represenets an empty square, \"b\" is a black stone and \"w\" is a white stone\n      'grid':Array(19).fill().map(x => Array(19).fill(\"+\")),\n    };\n    //bind this word to helper functions\n    this.handleClick = this.handleClick.bind(this);\n    this.handleReset = this.handleReset.bind(this);\n  }\n\n  //generate a new empty grid and set it to the grid state with setState\n  handleReset(){\n    let newGrid = Array(19).fill().map(x => Array(19).fill(\"+\"));\n    this.setState({'grid':newGrid});\n  }\n\n  handleClick(x, y){\n    //only add a peice and check for wins if the clicked square is empty\n    if (this.state.grid[x][y] === '+'){\n      //we don't want to mutate state directly, so we store the reference to 'grid' in a const\n      const g = this.state.grid;\n      //set the grid square cooresponding to the clicked square to the color of the current player\n      g[x][y] = this.state.isWhite === true ? 'w':'b';\n      //set the state with the new grid data\n      this.setState({'grid':g, 'isWhite':!this.state.isWhite})\n\n      //helper function for\n      function checkDir(x_, y_, color){\n        //track how many squares of a given color there are in a given dirention (specified by x_ and y_)\n        //for example checkDir(0,1, 'w') checks how many white stones there are in a row to the right )\n        let tracked = 0;\n        let _x = x;\n        let _y = y;\n        //stop tracking stones when the color is not equal to the specified stone or we have gone past the edge of the board\n        while (g[_x] !== undefined && g[_x][_y] === color){\n          //increment the number of tracked stones\n          tracked += 1;\n          //increment/decrement to check the next square in the specified direction\n          _y += y_;\n          _x += x_;\n        }\n        return tracked;\n      }\n      //sum the directions (left+right, up+down, 2 diagonals)\n      const w_horizontal = checkDir(0, 1, 'w') + checkDir(0, -1, 'w') -1;\n      const b_horizontal = checkDir(0, 1, 'b') + checkDir(0, -1, 'b') -1;\n\n      const w_vertical = checkDir(1, 0, 'w') + checkDir(-1, 0, 'w') -1;\n      const b_vertical = checkDir(1, 0, 'b') + checkDir(-1, 0, 'b') -1;\n\n      const w_diag1 = checkDir(1, 1, 'w') + checkDir(-1, -1, 'w') -1;\n      const b_diag1 = checkDir(1, 1, 'b') + checkDir(-1, -1, 'b') -1;\n\n      const w_diag2 = checkDir(1, 1, 'w') + checkDir(-1, -1, 'w') -1;\n      const b_diag2 = checkDir(-1, 1, 'b') + checkDir(1, -1, 'b') -1;\n\n      //check to see if there are any sums greater than or equal to 5 and alert the players of a win\n      //setTimeout is called so that the alert() function does not hold up the rendering of the board.\n      if (w_horizontal >=  5 || w_vertical >=  5 || w_diag1 >=  5 || w_diag2 >=  5){\n        setTimeout(()=>{alert('white wins')}, 1);\n      }\n\n      if (b_horizontal >= 5 || b_vertical >= 5 || b_diag1 >= 5 || b_diag2 >= 5){\n        setTimeout(()=>{alert('black wins')}, 1);\n      }\n    }\n  }\n  render(){\n    //define styles for the \u003Ctable> element in the return() function below\n    const style={\n             textAlign: \"center\",\n             margin:\"auto\",\n             height: \"auto\",\n             width:\"500px\",\n             border:\"1px solid black\",\n             tableLayout:'fixed',\n           };\n    const g = this.state.grid;\n    //loop through the squares in each row and generate a new Square component,\n    //passing in props to the Square component in the nested map() function\n    const board = g.map((row, i) => { return (\n      \u003Ctr key={\"row_\"+i}>\n        {row.map((col, j) => {\n          //set the color of the square based on state.grid\n          const color_ = g[i][j] === '+' ? '#e4e4a1': g[i][j] === 'w' ? 'white':'black';\n          //return Square component, passing in the following as props:\n          //square color defined above in color_,\n          //a value for the key which React needs (I think) and\n          //a function to handle clicks with grid coordinates passed in as arguments\n          return (\n            \u003CSquare handleClick={()=>this.handleClick(i,j)} color={color_} key={i+\"_\"+j} />\n              )\n            }\n          )\n        }\n      \u003C/tr>)\n    });\n\n    //returns the board with the Square Components in {board},\n    //as well as a simple Button component that takes the handleReset function as a prop\n    //this could be further refactored to separate the layout and styling, but it isn't that complicated so I will leave it like this\n    return (\n      \u003Cdiv style={{ textAlign:'center'}}>\n      \u003Ch2>\u003Ca href=\"https://en.wikipedia.org/wiki/Gomoku\" style={{textDecoration:\"none\"}}>五子棋\u003C/a>\u003C/h2>\n      \u003Cdiv style={{margin: 'auto', width:\"40%\"}}>\n      \u003Ctable cellSpacing=\"0\" style={style}>\n        \u003Ctbody>\n          {board}\n        \u003C/tbody>\n      \u003C/table>\n      \u003C/div>\n      \u003Cbr />\n      \u003CButton onClick={this.handleReset} />\n      \u003C/div>\n    )\n  }\n}\n{% endraw %}\n",[30,78897,78898,78913,78925,78937,78951,78955,78960,78978,78988,78994,79004,79009,79020,79025,79030,79073,79077,79082,79104,79124,79128,79132,79137,79143,79188,79203,79207,79211,79226,79231,79248,79253,79266,79271,79299,79304,79331,79335,79340,79365,79370,79375,79389,79401,79413,79418,79443,79448,79459,79464,79474,79484,79488,79495,79499,79504,79555,79606,79610,79661,79712,79716,79769,79822,79826,79879,79932,79936,79941,79946,79989,80013,80017,80021,80061,80084,80088,80092,80096,80102,80107,80117,80126,80135,80144,80154,80164,80173,80178,80190,80195,80200,80232,80254,80278,80283,80324,80329,80334,80339,80344,80350,80412,80416,80420,80424,80428,80436,80440,80444,80449,80454,80459,80465,80489,80534,80563,80587,80595,80605,80613,80621,80629,80637,80658,80666,80670,80674,80678],{"__ignoreMap":464},[151,78899,78900,78902,78904,78906,78908,78910],{"class":469,"line":470},[151,78901,5729],{"class":503},[151,78903,44519],{"class":1869},[151,78905,77067],{"class":503},[151,78907,44519],{"class":1869},[151,78909,2001],{"class":503},[151,78911,78912],{"class":1527},"//import React and Square component\n",[151,78914,78915,78917,78919,78921,78923],{"class":469,"line":488},[151,78916,16859],{"class":1869},[151,78918,77076],{"class":503},[151,78920,16853],{"class":1869},[151,78922,77081],{"class":481},[151,78924,20086],{"class":503},[151,78926,78927,78929,78931,78933,78935],{"class":469,"line":500},[151,78928,16859],{"class":1869},[151,78930,77104],{"class":503},[151,78932,16853],{"class":1869},[151,78934,77109],{"class":481},[151,78936,20086],{"class":503},[151,78938,78939,78941,78944,78946,78949],{"class":469,"line":509},[151,78940,16859],{"class":1869},[151,78942,78943],{"class":503}," { Button } ",[151,78945,16853],{"class":1869},[151,78947,78948],{"class":481}," './Button'",[151,78950,20086],{"class":503},[151,78952,78953],{"class":469,"line":517},[151,78954,1090],{"emptyLinePlaceholder":609},[151,78956,78957],{"class":469,"line":534},[151,78958,78959],{"class":1527},"//main board component with game logic\n",[151,78961,78962,78964,78966,78968,78970,78972,78974,78976],{"class":469,"line":1413},[151,78963,1870],{"class":1869},[151,78965,48323],{"class":12347},[151,78967,77124],{"class":15254},[151,78969,77127],{"class":1869},[151,78971,77130],{"class":15254},[151,78973,643],{"class":503},[151,78975,1584],{"class":15260},[151,78977,12966],{"class":503},[151,78979,78980,78982,78984,78986],{"class":469,"line":1418},[151,78981,77141],{"class":12347},[151,78983,12386],{"class":503},[151,78985,27681],{"class":15210},[151,78987,77148],{"class":503},[151,78989,78990,78992],{"class":469,"line":2462},[151,78991,77153],{"class":15289},[151,78993,77156],{"class":503},[151,78995,78996,78998,79000,79002],{"class":469,"line":2471},[151,78997,27842],{"class":15289},[151,78999,77163],{"class":503},[151,79001,1876],{"class":1869},[151,79003,19833],{"class":503},[151,79005,79006],{"class":469,"line":2480},[151,79007,79008],{"class":1527},"      //white goes first\n",[151,79010,79011,79014,79016,79018],{"class":469,"line":2489},[151,79012,79013],{"class":481},"      'isWhite'",[151,79015,208],{"class":503},[151,79017,19726],{"class":477},[151,79019,9417],{"class":503},[151,79021,79022],{"class":469,"line":2497},[151,79023,79024],{"class":1527},"      //this sets up an empty board\n",[151,79026,79027],{"class":469,"line":3140},[151,79028,79029],{"class":1527},"      //\"+\"\" represenets an empty square, \"b\" is a black stone and \"w\" is a white stone\n",[151,79031,79032,79035,79037,79039,79041,79043,79045,79047,79049,79051,79053,79055,79057,79059,79061,79063,79065,79067,79069,79071],{"class":469,"line":3149},[151,79033,79034],{"class":481},"      'grid'",[151,79036,208],{"class":503},[151,79038,77184],{"class":473},[151,79040,12386],{"class":503},[151,79042,42212],{"class":477},[151,79044,13576],{"class":503},[151,79046,77193],{"class":473},[151,79048,37880],{"class":503},[151,79050,68343],{"class":473},[151,79052,12386],{"class":503},[151,79054,11126],{"class":15210},[151,79056,20832],{"class":12347},[151,79058,77404],{"class":473},[151,79060,12386],{"class":503},[151,79062,42212],{"class":477},[151,79064,13576],{"class":503},[151,79066,77193],{"class":473},[151,79068,12386],{"class":503},[151,79070,77222],{"class":481},[151,79072,57361],{"class":503},[151,79074,79075],{"class":469,"line":3158},[151,79076,77257],{"class":503},[151,79078,79079],{"class":469,"line":3167},[151,79080,79081],{"class":1527},"    //bind this word to helper functions\n",[151,79083,79084,79086,79089,79091,79093,79096,79098,79100,79102],{"class":469,"line":3175},[151,79085,27842],{"class":15289},[151,79087,79088],{"class":503},".handleClick ",[151,79090,1876],{"class":1869},[151,79092,2324],{"class":15289},[151,79094,79095],{"class":503},".handleClick.",[151,79097,77274],{"class":473},[151,79099,12386],{"class":503},[151,79101,23252],{"class":15289},[151,79103,20129],{"class":503},[151,79105,79106,79108,79110,79112,79114,79116,79118,79120,79122],{"class":469,"line":3184},[151,79107,27842],{"class":15289},[151,79109,77309],{"class":503},[151,79111,1876],{"class":1869},[151,79113,2324],{"class":15289},[151,79115,77316],{"class":503},[151,79117,77274],{"class":473},[151,79119,12386],{"class":503},[151,79121,23252],{"class":15289},[151,79123,20129],{"class":503},[151,79125,79126],{"class":469,"line":3193},[151,79127,19957],{"class":503},[151,79129,79130],{"class":469,"line":3720},[151,79131,1090],{"emptyLinePlaceholder":609},[151,79133,79134],{"class":469,"line":3729},[151,79135,79136],{"class":1527},"  //generate a new empty grid and set it to the grid state with setState\n",[151,79138,79139,79141],{"class":469,"line":3735},[151,79140,77389],{"class":473},[151,79142,77392],{"class":503},[151,79144,79145,79147,79150,79152,79154,79156,79158,79160,79162,79164,79166,79168,79170,79172,79174,79176,79178,79180,79182,79184,79186],{"class":469,"line":3745},[151,79146,29455],{"class":12347},[151,79148,79149],{"class":503}," newGrid ",[151,79151,1876],{"class":1869},[151,79153,77404],{"class":473},[151,79155,12386],{"class":503},[151,79157,42212],{"class":477},[151,79159,13576],{"class":503},[151,79161,77193],{"class":473},[151,79163,37880],{"class":503},[151,79165,68343],{"class":473},[151,79167,12386],{"class":503},[151,79169,11126],{"class":15210},[151,79171,20832],{"class":12347},[151,79173,77404],{"class":473},[151,79175,12386],{"class":503},[151,79177,42212],{"class":477},[151,79179,13576],{"class":503},[151,79181,77193],{"class":473},[151,79183,12386],{"class":503},[151,79185,77222],{"class":481},[151,79187,29331],{"class":503},[151,79189,79190,79192,79194,79196,79198,79200],{"class":469,"line":3754},[151,79191,27842],{"class":15289},[151,79193,643],{"class":503},[151,79195,77452],{"class":473},[151,79197,77455],{"class":503},[151,79199,77458],{"class":481},[151,79201,79202],{"class":503},":newGrid});\n",[151,79204,79205],{"class":469,"line":3760},[151,79206,19957],{"class":503},[151,79208,79209],{"class":469,"line":3773},[151,79210,1090],{"emptyLinePlaceholder":609},[151,79212,79213,79216,79218,79220,79222,79224],{"class":469,"line":3782},[151,79214,79215],{"class":473},"  handleClick",[151,79217,12386],{"class":503},[151,79219,11126],{"class":15210},[151,79221,106],{"class":503},[151,79223,25286],{"class":15210},[151,79225,77148],{"class":503},[151,79227,79228],{"class":469,"line":3791},[151,79229,79230],{"class":1527},"    //only add a peice and check for wins if the clicked square is empty\n",[151,79232,79233,79235,79237,79239,79242,79244,79246],{"class":469,"line":3803},[151,79234,23327],{"class":1869},[151,79236,129],{"class":503},[151,79238,23252],{"class":15289},[151,79240,79241],{"class":503},".state.grid[x][y] ",[151,79243,34077],{"class":1869},[151,79245,78003],{"class":481},[151,79247,77148],{"class":503},[151,79249,79250],{"class":469,"line":3811},[151,79251,79252],{"class":1527},"      //we don't want to mutate state directly, so we store the reference to 'grid' in a const\n",[151,79254,79255,79257,79259,79261,79263],{"class":469,"line":3820},[151,79256,34174],{"class":12347},[151,79258,77500],{"class":12360},[151,79260,19865],{"class":1869},[151,79262,2324],{"class":15289},[151,79264,79265],{"class":503},".state.grid;\n",[151,79267,79268],{"class":469,"line":7084},[151,79269,79270],{"class":1527},"      //set the grid square cooresponding to the clicked square to the color of the current player\n",[151,79272,79273,79276,79278,79280,79283,79285,79287,79289,79292,79294,79297],{"class":469,"line":7148},[151,79274,79275],{"class":503},"      g[x][y] ",[151,79277,1876],{"class":1869},[151,79279,2324],{"class":15289},[151,79281,79282],{"class":503},".state.isWhite ",[151,79284,34077],{"class":1869},[151,79286,529],{"class":477},[151,79288,75197],{"class":1869},[151,79290,79291],{"class":481}," 'w'",[151,79293,208],{"class":1869},[151,79295,79296],{"class":481},"'b'",[151,79298,20086],{"class":503},[151,79300,79301],{"class":469,"line":7211},[151,79302,79303],{"class":1527},"      //set the state with the new grid data\n",[151,79305,79306,79308,79310,79312,79314,79316,79319,79322,79324,79326,79328],{"class":469,"line":7273},[151,79307,34086],{"class":15289},[151,79309,643],{"class":503},[151,79311,77452],{"class":473},[151,79313,77455],{"class":503},[151,79315,77458],{"class":481},[151,79317,79318],{"class":503},":g, ",[151,79320,79321],{"class":481},"'isWhite'",[151,79323,208],{"class":503},[151,79325,12282],{"class":1869},[151,79327,23252],{"class":15289},[151,79329,79330],{"class":503},".state.isWhite})\n",[151,79332,79333],{"class":469,"line":7335},[151,79334,1090],{"emptyLinePlaceholder":609},[151,79336,79337],{"class":469,"line":7398},[151,79338,79339],{"class":1527},"      //helper function for\n",[151,79341,79342,79345,79348,79350,79353,79355,79358,79360,79363],{"class":469,"line":7462},[151,79343,79344],{"class":12347},"      function",[151,79346,79347],{"class":473}," checkDir",[151,79349,12386],{"class":503},[151,79351,79352],{"class":15210},"x_",[151,79354,106],{"class":503},[151,79356,79357],{"class":15210},"y_",[151,79359,106],{"class":503},[151,79361,79362],{"class":15210},"color",[151,79364,77148],{"class":503},[151,79366,79367],{"class":469,"line":7467},[151,79368,79369],{"class":1527},"        //track how many squares of a given color there are in a given dirention (specified by x_ and y_)\n",[151,79371,79372],{"class":469,"line":7532},[151,79373,79374],{"class":1527},"        //for example checkDir(0,1, 'w') checks how many white stones there are in a row to the right )\n",[151,79376,79377,79380,79383,79385,79387],{"class":469,"line":7537},[151,79378,79379],{"class":12347},"        let",[151,79381,79382],{"class":503}," tracked ",[151,79384,1876],{"class":1869},[151,79386,57890],{"class":477},[151,79388,20086],{"class":503},[151,79390,79391,79393,79396,79398],{"class":469,"line":7603},[151,79392,79379],{"class":12347},[151,79394,79395],{"class":503}," _x ",[151,79397,1876],{"class":1869},[151,79399,79400],{"class":503}," x;\n",[151,79402,79403,79405,79408,79410],{"class":469,"line":7608},[151,79404,79379],{"class":12347},[151,79406,79407],{"class":503}," _y ",[151,79409,1876],{"class":1869},[151,79411,79412],{"class":503}," y;\n",[151,79414,79415],{"class":469,"line":7673},[151,79416,79417],{"class":1527},"        //stop tracking stones when the color is not equal to the specified stone or we have gone past the edge of the board\n",[151,79419,79420,79423,79426,79429,79432,79435,79438,79440],{"class":469,"line":7678},[151,79421,79422],{"class":1869},"        while",[151,79424,79425],{"class":503}," (g[_x] ",[151,79427,79428],{"class":1869},"!==",[151,79430,79431],{"class":477}," undefined",[151,79433,79434],{"class":1869}," &&",[151,79436,79437],{"class":503}," g[_x][_y] ",[151,79439,34077],{"class":1869},[151,79441,79442],{"class":503}," color){\n",[151,79444,79445],{"class":469,"line":7708},[151,79446,79447],{"class":1527},"          //increment the number of tracked stones\n",[151,79449,79450,79453,79455,79457],{"class":469,"line":7713},[151,79451,79452],{"class":503},"          tracked ",[151,79454,24780],{"class":1869},[151,79456,12448],{"class":477},[151,79458,20086],{"class":503},[151,79460,79461],{"class":469,"line":7746},[151,79462,79463],{"class":1527},"          //increment/decrement to check the next square in the specified direction\n",[151,79465,79466,79469,79471],{"class":469,"line":7751},[151,79467,79468],{"class":503},"          _y ",[151,79470,24780],{"class":1869},[151,79472,79473],{"class":503}," y_;\n",[151,79475,79476,79479,79481],{"class":469,"line":7816},[151,79477,79478],{"class":503},"          _x ",[151,79480,24780],{"class":1869},[151,79482,79483],{"class":503}," x_;\n",[151,79485,79486],{"class":469,"line":7821},[151,79487,23390],{"class":503},[151,79489,79490,79492],{"class":469,"line":7847},[151,79491,16833],{"class":1869},[151,79493,79494],{"class":503}," tracked;\n",[151,79496,79497],{"class":469,"line":7852},[151,79498,77583],{"class":503},[151,79500,79501],{"class":469,"line":7887},[151,79502,79503],{"class":1527},"      //sum the directions (left+right, up+down, 2 diagonals)\n",[151,79505,79506,79508,79511,79513,79515,79517,79519,79521,79523,79525,79527,79529,79531,79533,79535,79537,79539,79541,79543,79545,79547,79549,79551,79553],{"class":469,"line":7892},[151,79507,34174],{"class":12347},[151,79509,79510],{"class":12360}," w_horizontal",[151,79512,19865],{"class":1869},[151,79514,79347],{"class":473},[151,79516,12386],{"class":503},[151,79518,9181],{"class":477},[151,79520,106],{"class":503},[151,79522,6760],{"class":477},[151,79524,106],{"class":503},[151,79526,71153],{"class":481},[151,79528,16995],{"class":503},[151,79530,22885],{"class":1869},[151,79532,79347],{"class":473},[151,79534,12386],{"class":503},[151,79536,9181],{"class":477},[151,79538,106],{"class":503},[151,79540,12445],{"class":1869},[151,79542,6760],{"class":477},[151,79544,106],{"class":503},[151,79546,71153],{"class":481},[151,79548,16995],{"class":503},[151,79550,12445],{"class":1869},[151,79552,6760],{"class":477},[151,79554,20086],{"class":503},[151,79556,79557,79559,79562,79564,79566,79568,79570,79572,79574,79576,79578,79580,79582,79584,79586,79588,79590,79592,79594,79596,79598,79600,79602,79604],{"class":469,"line":7924},[151,79558,34174],{"class":12347},[151,79560,79561],{"class":12360}," b_horizontal",[151,79563,19865],{"class":1869},[151,79565,79347],{"class":473},[151,79567,12386],{"class":503},[151,79569,9181],{"class":477},[151,79571,106],{"class":503},[151,79573,6760],{"class":477},[151,79575,106],{"class":503},[151,79577,79296],{"class":481},[151,79579,16995],{"class":503},[151,79581,22885],{"class":1869},[151,79583,79347],{"class":473},[151,79585,12386],{"class":503},[151,79587,9181],{"class":477},[151,79589,106],{"class":503},[151,79591,12445],{"class":1869},[151,79593,6760],{"class":477},[151,79595,106],{"class":503},[151,79597,79296],{"class":481},[151,79599,16995],{"class":503},[151,79601,12445],{"class":1869},[151,79603,6760],{"class":477},[151,79605,20086],{"class":503},[151,79607,79608],{"class":469,"line":7929},[151,79609,1090],{"emptyLinePlaceholder":609},[151,79611,79612,79614,79617,79619,79621,79623,79625,79627,79629,79631,79633,79635,79637,79639,79641,79643,79645,79647,79649,79651,79653,79655,79657,79659],{"class":469,"line":7991},[151,79613,34174],{"class":12347},[151,79615,79616],{"class":12360}," w_vertical",[151,79618,19865],{"class":1869},[151,79620,79347],{"class":473},[151,79622,12386],{"class":503},[151,79624,6760],{"class":477},[151,79626,106],{"class":503},[151,79628,9181],{"class":477},[151,79630,106],{"class":503},[151,79632,71153],{"class":481},[151,79634,16995],{"class":503},[151,79636,22885],{"class":1869},[151,79638,79347],{"class":473},[151,79640,12386],{"class":503},[151,79642,12445],{"class":1869},[151,79644,6760],{"class":477},[151,79646,106],{"class":503},[151,79648,9181],{"class":477},[151,79650,106],{"class":503},[151,79652,71153],{"class":481},[151,79654,16995],{"class":503},[151,79656,12445],{"class":1869},[151,79658,6760],{"class":477},[151,79660,20086],{"class":503},[151,79662,79663,79665,79668,79670,79672,79674,79676,79678,79680,79682,79684,79686,79688,79690,79692,79694,79696,79698,79700,79702,79704,79706,79708,79710],{"class":469,"line":7996},[151,79664,34174],{"class":12347},[151,79666,79667],{"class":12360}," b_vertical",[151,79669,19865],{"class":1869},[151,79671,79347],{"class":473},[151,79673,12386],{"class":503},[151,79675,6760],{"class":477},[151,79677,106],{"class":503},[151,79679,9181],{"class":477},[151,79681,106],{"class":503},[151,79683,79296],{"class":481},[151,79685,16995],{"class":503},[151,79687,22885],{"class":1869},[151,79689,79347],{"class":473},[151,79691,12386],{"class":503},[151,79693,12445],{"class":1869},[151,79695,6760],{"class":477},[151,79697,106],{"class":503},[151,79699,9181],{"class":477},[151,79701,106],{"class":503},[151,79703,79296],{"class":481},[151,79705,16995],{"class":503},[151,79707,12445],{"class":1869},[151,79709,6760],{"class":477},[151,79711,20086],{"class":503},[151,79713,79714],{"class":469,"line":8078},[151,79715,1090],{"emptyLinePlaceholder":609},[151,79717,79718,79720,79723,79725,79727,79729,79731,79733,79735,79737,79739,79741,79743,79745,79747,79749,79751,79753,79755,79757,79759,79761,79763,79765,79767],{"class":469,"line":8140},[151,79719,34174],{"class":12347},[151,79721,79722],{"class":12360}," w_diag1",[151,79724,19865],{"class":1869},[151,79726,79347],{"class":473},[151,79728,12386],{"class":503},[151,79730,6760],{"class":477},[151,79732,106],{"class":503},[151,79734,6760],{"class":477},[151,79736,106],{"class":503},[151,79738,71153],{"class":481},[151,79740,16995],{"class":503},[151,79742,22885],{"class":1869},[151,79744,79347],{"class":473},[151,79746,12386],{"class":503},[151,79748,12445],{"class":1869},[151,79750,6760],{"class":477},[151,79752,106],{"class":503},[151,79754,12445],{"class":1869},[151,79756,6760],{"class":477},[151,79758,106],{"class":503},[151,79760,71153],{"class":481},[151,79762,16995],{"class":503},[151,79764,12445],{"class":1869},[151,79766,6760],{"class":477},[151,79768,20086],{"class":503},[151,79770,79771,79773,79776,79778,79780,79782,79784,79786,79788,79790,79792,79794,79796,79798,79800,79802,79804,79806,79808,79810,79812,79814,79816,79818,79820],{"class":469,"line":8145},[151,79772,34174],{"class":12347},[151,79774,79775],{"class":12360}," b_diag1",[151,79777,19865],{"class":1869},[151,79779,79347],{"class":473},[151,79781,12386],{"class":503},[151,79783,6760],{"class":477},[151,79785,106],{"class":503},[151,79787,6760],{"class":477},[151,79789,106],{"class":503},[151,79791,79296],{"class":481},[151,79793,16995],{"class":503},[151,79795,22885],{"class":1869},[151,79797,79347],{"class":473},[151,79799,12386],{"class":503},[151,79801,12445],{"class":1869},[151,79803,6760],{"class":477},[151,79805,106],{"class":503},[151,79807,12445],{"class":1869},[151,79809,6760],{"class":477},[151,79811,106],{"class":503},[151,79813,79296],{"class":481},[151,79815,16995],{"class":503},[151,79817,12445],{"class":1869},[151,79819,6760],{"class":477},[151,79821,20086],{"class":503},[151,79823,79824],{"class":469,"line":8259},[151,79825,1090],{"emptyLinePlaceholder":609},[151,79827,79828,79830,79833,79835,79837,79839,79841,79843,79845,79847,79849,79851,79853,79855,79857,79859,79861,79863,79865,79867,79869,79871,79873,79875,79877],{"class":469,"line":8264},[151,79829,34174],{"class":12347},[151,79831,79832],{"class":12360}," w_diag2",[151,79834,19865],{"class":1869},[151,79836,79347],{"class":473},[151,79838,12386],{"class":503},[151,79840,6760],{"class":477},[151,79842,106],{"class":503},[151,79844,6760],{"class":477},[151,79846,106],{"class":503},[151,79848,71153],{"class":481},[151,79850,16995],{"class":503},[151,79852,22885],{"class":1869},[151,79854,79347],{"class":473},[151,79856,12386],{"class":503},[151,79858,12445],{"class":1869},[151,79860,6760],{"class":477},[151,79862,106],{"class":503},[151,79864,12445],{"class":1869},[151,79866,6760],{"class":477},[151,79868,106],{"class":503},[151,79870,71153],{"class":481},[151,79872,16995],{"class":503},[151,79874,12445],{"class":1869},[151,79876,6760],{"class":477},[151,79878,20086],{"class":503},[151,79880,79881,79883,79886,79888,79890,79892,79894,79896,79898,79900,79902,79904,79906,79908,79910,79912,79914,79916,79918,79920,79922,79924,79926,79928,79930],{"class":469,"line":8613},[151,79882,34174],{"class":12347},[151,79884,79885],{"class":12360}," b_diag2",[151,79887,19865],{"class":1869},[151,79889,79347],{"class":473},[151,79891,12386],{"class":503},[151,79893,12445],{"class":1869},[151,79895,6760],{"class":477},[151,79897,106],{"class":503},[151,79899,6760],{"class":477},[151,79901,106],{"class":503},[151,79903,79296],{"class":481},[151,79905,16995],{"class":503},[151,79907,22885],{"class":1869},[151,79909,79347],{"class":473},[151,79911,12386],{"class":503},[151,79913,6760],{"class":477},[151,79915,106],{"class":503},[151,79917,12445],{"class":1869},[151,79919,6760],{"class":477},[151,79921,106],{"class":503},[151,79923,79296],{"class":481},[151,79925,16995],{"class":503},[151,79927,12445],{"class":1869},[151,79929,6760],{"class":477},[151,79931,20086],{"class":503},[151,79933,79934],{"class":469,"line":8678},[151,79935,1090],{"emptyLinePlaceholder":609},[151,79937,79938],{"class":469,"line":8742},[151,79939,79940],{"class":1527},"      //check to see if there are any sums greater than or equal to 5 and alert the players of a win\n",[151,79942,79943],{"class":469,"line":8806},[151,79944,79945],{"class":1527},"      //setTimeout is called so that the alert() function does not hold up the rendering of the board.\n",[151,79947,79948,79950,79953,79956,79959,79962,79965,79967,79969,79971,79974,79976,79978,79980,79983,79985,79987],{"class":469,"line":8870},[151,79949,77526],{"class":1869},[151,79951,79952],{"class":503}," (w_horizontal ",[151,79954,79955],{"class":1869},">=",[151,79957,79958],{"class":477},"  5",[151,79960,79961],{"class":1869}," ||",[151,79963,79964],{"class":503}," w_vertical ",[151,79966,79955],{"class":1869},[151,79968,79958],{"class":477},[151,79970,79961],{"class":1869},[151,79972,79973],{"class":503}," w_diag1 ",[151,79975,79955],{"class":1869},[151,79977,79958],{"class":477},[151,79979,79961],{"class":1869},[151,79981,79982],{"class":503}," w_diag2 ",[151,79984,79955],{"class":1869},[151,79986,79958],{"class":477},[151,79988,77148],{"class":503},[151,79990,79991,79994,79996,79998,80000,80002,80004,80007,80009,80011],{"class":469,"line":8875},[151,79992,79993],{"class":473},"        setTimeout",[151,79995,77834],{"class":503},[151,79997,17166],{"class":12347},[151,79999,5729],{"class":503},[151,80001,77841],{"class":473},[151,80003,12386],{"class":503},[151,80005,80006],{"class":481},"'white wins'",[151,80008,77849],{"class":503},[151,80010,6760],{"class":477},[151,80012,20129],{"class":503},[151,80014,80015],{"class":469,"line":8881},[151,80016,77583],{"class":503},[151,80018,80019],{"class":469,"line":8886},[151,80020,1090],{"emptyLinePlaceholder":609},[151,80022,80023,80025,80028,80030,80032,80034,80037,80039,80041,80043,80046,80048,80050,80052,80055,80057,80059],{"class":469,"line":8892},[151,80024,77526],{"class":1869},[151,80026,80027],{"class":503}," (b_horizontal ",[151,80029,79955],{"class":1869},[151,80031,546],{"class":477},[151,80033,79961],{"class":1869},[151,80035,80036],{"class":503}," b_vertical ",[151,80038,79955],{"class":1869},[151,80040,546],{"class":477},[151,80042,79961],{"class":1869},[151,80044,80045],{"class":503}," b_diag1 ",[151,80047,79955],{"class":1869},[151,80049,546],{"class":477},[151,80051,79961],{"class":1869},[151,80053,80054],{"class":503}," b_diag2 ",[151,80056,79955],{"class":1869},[151,80058,546],{"class":477},[151,80060,77148],{"class":503},[151,80062,80063,80065,80067,80069,80071,80073,80075,80078,80080,80082],{"class":469,"line":8963},[151,80064,79993],{"class":473},[151,80066,77834],{"class":503},[151,80068,17166],{"class":12347},[151,80070,5729],{"class":503},[151,80072,77841],{"class":473},[151,80074,12386],{"class":503},[151,80076,80077],{"class":481},"'black wins'",[151,80079,77849],{"class":503},[151,80081,6760],{"class":477},[151,80083,20129],{"class":503},[151,80085,80086],{"class":469,"line":8969},[151,80087,77583],{"class":503},[151,80089,80090],{"class":469,"line":15001},[151,80091,9461],{"class":503},[151,80093,80094],{"class":469,"line":15009},[151,80095,19957],{"class":503},[151,80097,80098,80100],{"class":469,"line":15019},[151,80099,78115],{"class":473},[151,80101,77392],{"class":503},[151,80103,80104],{"class":469,"line":15027},[151,80105,80106],{"class":1527},"    //define styles for the \u003Ctable> element in the return() function below\n",[151,80108,80109,80111,80113,80115],{"class":469,"line":15037},[151,80110,19860],{"class":12347},[151,80112,48548],{"class":12360},[151,80114,1876],{"class":1869},[151,80116,12966],{"class":503},[151,80118,80119,80122,80124],{"class":469,"line":15045},[151,80120,80121],{"class":503},"             textAlign: ",[151,80123,78456],{"class":481},[151,80125,9417],{"class":503},[151,80127,80128,80131,80133],{"class":469,"line":15055},[151,80129,80130],{"class":503},"             margin:",[151,80132,78145],{"class":481},[151,80134,9417],{"class":503},[151,80136,80137,80140,80142],{"class":469,"line":15060},[151,80138,80139],{"class":503},"             height: ",[151,80141,78145],{"class":481},[151,80143,9417],{"class":503},[151,80145,80146,80149,80152],{"class":469,"line":15068},[151,80147,80148],{"class":503},"             width:",[151,80150,80151],{"class":481},"\"500px\"",[151,80153,9417],{"class":503},[151,80155,80156,80159,80162],{"class":469,"line":15076},[151,80157,80158],{"class":503},"             border:",[151,80160,80161],{"class":481},"\"1px solid black\"",[151,80163,9417],{"class":503},[151,80165,80166,80169,80171],{"class":469,"line":15085},[151,80167,80168],{"class":503},"             tableLayout:",[151,80170,78193],{"class":481},[151,80172,9417],{"class":503},[151,80174,80175],{"class":469,"line":15095},[151,80176,80177],{"class":503},"           };\n",[151,80179,80180,80182,80184,80186,80188],{"class":469,"line":15105},[151,80181,19860],{"class":12347},[151,80183,77500],{"class":12360},[151,80185,19865],{"class":1869},[151,80187,2324],{"class":15289},[151,80189,79265],{"class":503},[151,80191,80192],{"class":469,"line":15110},[151,80193,80194],{"class":1527},"    //loop through the squares in each row and generate a new Square component,\n",[151,80196,80197],{"class":469,"line":15118},[151,80198,80199],{"class":1527},"    //passing in props to the Square component in the nested map() function\n",[151,80201,80202,80204,80207,80209,80211,80213,80215,80218,80220,80222,80224,80226,80228,80230],{"class":469,"line":15128},[151,80203,19860],{"class":12347},[151,80205,80206],{"class":12360}," board",[151,80208,19865],{"class":1869},[151,80210,77620],{"class":503},[151,80212,68343],{"class":473},[151,80214,34211],{"class":503},[151,80216,80217],{"class":15210},"row",[151,80219,106],{"class":503},[151,80221,77563],{"class":15210},[151,80223,16995],{"class":503},[151,80225,17166],{"class":12347},[151,80227,12351],{"class":503},[151,80229,63121],{"class":1869},[151,80231,37723],{"class":503},[151,80233,80234,80236,80238,80240,80242,80244,80246,80248,80250,80252],{"class":469,"line":15139},[151,80235,48222],{"class":503},[151,80237,1137],{"class":14368},[151,80239,2176],{"class":473},[151,80241,1876],{"class":1869},[151,80243,5729],{"class":78245},[151,80245,78248],{"class":481},[151,80247,22885],{"class":1869},[151,80249,77563],{"class":503},[151,80251,2001],{"class":78245},[151,80253,3742],{"class":503},[151,80255,80256,80258,80261,80263,80265,80268,80270,80272,80274,80276],{"class":469,"line":31954},[151,80257,78261],{"class":78245},[151,80259,80260],{"class":503},"row.",[151,80262,68343],{"class":473},[151,80264,34211],{"class":503},[151,80266,80267],{"class":15210},"col",[151,80269,106],{"class":503},[151,80271,78276],{"class":15210},[151,80273,16995],{"class":503},[151,80275,17166],{"class":12347},[151,80277,19833],{"class":503},[151,80279,80280],{"class":469,"line":31960},[151,80281,80282],{"class":1527},"          //set the color of the square based on state.grid\n",[151,80284,80285,80288,80291,80293,80296,80298,80300,80302,80305,80307,80309,80311,80313,80315,80318,80320,80322],{"class":469,"line":31965},[151,80286,80287],{"class":12347},"          const",[151,80289,80290],{"class":12360}," color_",[151,80292,19865],{"class":1869},[151,80294,80295],{"class":503}," g[i][j] ",[151,80297,34077],{"class":1869},[151,80299,78003],{"class":481},[151,80301,75197],{"class":1869},[151,80303,80304],{"class":481}," '#e4e4a1'",[151,80306,208],{"class":1869},[151,80308,80295],{"class":503},[151,80310,34077],{"class":1869},[151,80312,79291],{"class":481},[151,80314,75197],{"class":1869},[151,80316,80317],{"class":481}," 'white'",[151,80319,208],{"class":1869},[151,80321,45401],{"class":481},[151,80323,20086],{"class":503},[151,80325,80326],{"class":469,"line":31971},[151,80327,80328],{"class":1527},"          //return Square component, passing in the following as props:\n",[151,80330,80331],{"class":469,"line":31983},[151,80332,80333],{"class":1527},"          //square color defined above in color_,\n",[151,80335,80336],{"class":469,"line":31994},[151,80337,80338],{"class":1527},"          //a value for the key which React needs (I think) and\n",[151,80340,80341],{"class":469,"line":32007},[151,80342,80343],{"class":1527},"          //a function to handle clicks with grid coordinates passed in as arguments\n",[151,80345,80346,80348],{"class":469,"line":32018},[151,80347,77858],{"class":1869},[151,80349,37723],{"class":503},[151,80351,80352,80354,80357,80360,80362,80364,80366,80368,80370,80372,80375,80378,80380,80383,80385,80387,80390,80392,80394,80396,80398,80400,80402,80404,80406,80408,80410],{"class":469,"line":32026},[151,80353,21079],{"class":503},[151,80355,80356],{"class":6205},"Square",[151,80358,80359],{"class":473}," handleClick",[151,80361,1876],{"class":1869},[151,80363,5729],{"class":78245},[151,80365,78354],{"class":503},[151,80367,17166],{"class":12347},[151,80369,23252],{"class":15289},[151,80371,643],{"class":503},[151,80373,80374],{"class":473},"handleClick",[151,80376,80377],{"class":503},"(i,j)",[151,80379,2001],{"class":78245},[151,80381,80382],{"class":473}," color",[151,80384,1876],{"class":1869},[151,80386,5729],{"class":78245},[151,80388,80389],{"class":503},"color_",[151,80391,2001],{"class":78245},[151,80393,2176],{"class":473},[151,80395,1876],{"class":1869},[151,80397,5729],{"class":78245},[151,80399,77563],{"class":503},[151,80401,22885],{"class":1869},[151,80403,78320],{"class":481},[151,80405,22885],{"class":1869},[151,80407,78276],{"class":503},[151,80409,2001],{"class":78245},[151,80411,34675],{"class":503},[151,80413,80414],{"class":469,"line":32031},[151,80415,78402],{"class":503},[151,80417,80418],{"class":469,"line":32036},[151,80419,39498],{"class":503},[151,80421,80422],{"class":469,"line":32042},[151,80423,27913],{"class":503},[151,80425,80426],{"class":469,"line":32054},[151,80427,23390],{"class":78245},[151,80429,80430,80432,80434],{"class":469,"line":32067},[151,80431,48707],{"class":503},[151,80433,1137],{"class":14368},[151,80435,78423],{"class":503},[151,80437,80438],{"class":469,"line":32086},[151,80439,26978],{"class":503},[151,80441,80442],{"class":469,"line":32097},[151,80443,1090],{"emptyLinePlaceholder":609},[151,80445,80446],{"class":469,"line":25585},[151,80447,80448],{"class":1527},"    //returns the board with the Square Components in {board},\n",[151,80450,80451],{"class":469,"line":32112},[151,80452,80453],{"class":1527},"    //as well as a simple Button component that takes the handleReset function as a prop\n",[151,80455,80456],{"class":469,"line":32117},[151,80457,80458],{"class":1527},"    //this could be further refactored to separate the layout and styling, but it isn't that complicated so I will leave it like this\n",[151,80460,80461,80463],{"class":469,"line":32123},[151,80462,17496],{"class":1869},[151,80464,37723],{"class":503},[151,80466,80467,80469,80471,80473,80475,80477,80480,80483,80485,80487],{"class":469,"line":32151},[151,80468,48222],{"class":503},[151,80470,23950],{"class":14368},[151,80472,48548],{"class":473},[151,80474,1876],{"class":1869},[151,80476,5729],{"class":78245},[151,80478,80479],{"class":503},"{ textAlign:",[151,80481,80482],{"class":481},"'center'",[151,80484,2001],{"class":503},[151,80486,2001],{"class":78245},[151,80488,3742],{"class":503},[151,80490,80491,80493,80495,80498,80500,80502,80504,80507,80509,80511,80513,80516,80519,80521,80523,80526,80528,80530,80532],{"class":469,"line":32156},[151,80492,48222],{"class":503},[151,80494,56],{"class":14368},[151,80496,80497],{"class":503},">\u003C",[151,80499,20],{"class":14368},[151,80501,61092],{"class":473},[151,80503,1876],{"class":1869},[151,80505,80506],{"class":481},"\"https://en.wikipedia.org/wiki/Gomoku\"",[151,80508,48548],{"class":473},[151,80510,1876],{"class":1869},[151,80512,5729],{"class":78245},[151,80514,80515],{"class":503},"{textDecoration:",[151,80517,80518],{"class":481},"\"none\"",[151,80520,2001],{"class":503},[151,80522,2001],{"class":78245},[151,80524,80525],{"class":503},">五子棋\u003C/",[151,80527,20],{"class":14368},[151,80529,62997],{"class":503},[151,80531,56],{"class":14368},[151,80533,3742],{"class":503},[151,80535,80536,80538,80540,80542,80544,80546,80549,80551,80554,80557,80559,80561],{"class":469,"line":32162},[151,80537,48222],{"class":503},[151,80539,23950],{"class":14368},[151,80541,48548],{"class":473},[151,80543,1876],{"class":1869},[151,80545,5729],{"class":78245},[151,80547,80548],{"class":503},"{margin: ",[151,80550,78135],{"class":481},[151,80552,80553],{"class":503},", width:",[151,80555,80556],{"class":481},"\"40%\"",[151,80558,2001],{"class":503},[151,80560,2001],{"class":78245},[151,80562,3742],{"class":503},[151,80564,80565,80567,80569,80571,80573,80575,80577,80579,80581,80583,80585],{"class":469,"line":32168},[151,80566,48222],{"class":503},[151,80568,1131],{"class":14368},[151,80570,78533],{"class":473},[151,80572,1876],{"class":1869},[151,80574,78538],{"class":481},[151,80576,48548],{"class":473},[151,80578,1876],{"class":1869},[151,80580,5729],{"class":78245},[151,80582,589],{"class":503},[151,80584,2001],{"class":78245},[151,80586,3742],{"class":503},[151,80588,80589,80591,80593],{"class":469,"line":32180},[151,80590,21070],{"class":503},[151,80592,1153],{"class":14368},[151,80594,3742],{"class":503},[151,80596,80597,80600,80603],{"class":469,"line":32192},[151,80598,80599],{"class":78245},"          {",[151,80601,80602],{"class":503},"board",[151,80604,6274],{"class":78245},[151,80606,80607,80609,80611],{"class":469,"line":32207},[151,80608,21175],{"class":503},[151,80610,1153],{"class":14368},[151,80612,3742],{"class":503},[151,80614,80615,80617,80619],{"class":469,"line":32217},[151,80616,48707],{"class":503},[151,80618,1131],{"class":14368},[151,80620,3742],{"class":503},[151,80622,80623,80625,80627],{"class":469,"line":32226},[151,80624,48707],{"class":503},[151,80626,23950],{"class":14368},[151,80628,3742],{"class":503},[151,80630,80631,80633,80635],{"class":469,"line":32231},[151,80632,48222],{"class":503},[151,80634,1205],{"class":14368},[151,80636,34675],{"class":503},[151,80638,80639,80641,80644,80646,80648,80650,80652,80654,80656],{"class":469,"line":32236},[151,80640,48222],{"class":503},[151,80642,80643],{"class":6205},"Button",[151,80645,78623],{"class":473},[151,80647,1876],{"class":1869},[151,80649,5729],{"class":78245},[151,80651,23252],{"class":15289},[151,80653,78632],{"class":503},[151,80655,2001],{"class":78245},[151,80657,34675],{"class":503},[151,80659,80660,80662,80664],{"class":469,"line":32244},[151,80661,48707],{"class":503},[151,80663,23950],{"class":14368},[151,80665,3742],{"class":503},[151,80667,80668],{"class":469,"line":32249},[151,80669,39567],{"class":503},[151,80671,80672],{"class":469,"line":32255},[151,80673,19957],{"class":503},[151,80675,80676],{"class":469,"line":32272},[151,80677,6274],{"class":503},[151,80679,80680,80682,80684,80686,80688],{"class":469,"line":32277},[151,80681,5729],{"class":503},[151,80683,44519],{"class":1869},[151,80685,78869],{"class":503},[151,80687,44519],{"class":1869},[151,80689,6274],{"class":503},[11,80691,80692],{},"And here is the square component:",[459,80694,80696],{"className":19459,"code":80695,"language":19461,"meta":464,"style":464},"{% raw %}import React from 'react';\n\nexport class Square extends React.Component{\n  render(){\n    const color_ = this.props.color;\n    return (\n      \u003Ctd\n        style={{\n          overflow:'hidden',\n          width:'auto',\n          height:'25px',\n          backgroundColor:'#e4e4a1',\n          color:'red',\n          boarderColor: 'black',\n          border:\".5px solid black\"\n        }}\n      onClick={this.props.handleClick} >\n        \u003Cdiv\n          style={{color:color_,\n                  border:\"1px solid\",\n                  backgroundColor: color_,\n                  borderRadius: \"50%\",\n                  borderColor: color_,\n                  height:25}} >\n        \u003C/div>\n      \u003C/td>\n    )\n  }\n}\n{% endraw %}\n",[30,80697,80698,80720,80724,80743,80749,80762,80768,80775,80786,80796,80805,80815,80825,80835,80844,80852,80859,80878,80884,80896,80906,80911,80921,80926,80940,80948,80956,80960,80964,80968],{"__ignoreMap":464},[151,80699,80700,80702,80704,80706,80708,80710,80712,80714,80716,80718],{"class":469,"line":470},[151,80701,5729],{"class":503},[151,80703,44519],{"class":1869},[151,80705,77067],{"class":503},[151,80707,44519],{"class":1869},[151,80709,2001],{"class":503},[151,80711,16859],{"class":1869},[151,80713,77076],{"class":503},[151,80715,16853],{"class":1869},[151,80717,77081],{"class":481},[151,80719,20086],{"class":503},[151,80721,80722],{"class":469,"line":488},[151,80723,1090],{"emptyLinePlaceholder":609},[151,80725,80726,80728,80730,80733,80735,80737,80739,80741],{"class":469,"line":500},[151,80727,1870],{"class":1869},[151,80729,48323],{"class":12347},[151,80731,80732],{"class":15254}," Square",[151,80734,77127],{"class":1869},[151,80736,77130],{"class":15254},[151,80738,643],{"class":503},[151,80740,1584],{"class":15260},[151,80742,12966],{"class":503},[151,80744,80745,80747],{"class":469,"line":509},[151,80746,78115],{"class":473},[151,80748,77392],{"class":503},[151,80750,80751,80753,80755,80757,80759],{"class":469,"line":517},[151,80752,19860],{"class":12347},[151,80754,80290],{"class":12360},[151,80756,19865],{"class":1869},[151,80758,2324],{"class":15289},[151,80760,80761],{"class":503},".props.color;\n",[151,80763,80764,80766],{"class":469,"line":534},[151,80765,17496],{"class":1869},[151,80767,37723],{"class":503},[151,80769,80770,80772],{"class":469,"line":1413},[151,80771,48222],{"class":503},[151,80773,80774],{"class":14368},"td\n",[151,80776,80777,80780,80782,80784],{"class":469,"line":1418},[151,80778,80779],{"class":473},"        style",[151,80781,1876],{"class":1869},[151,80783,5729],{"class":78245},[151,80785,12966],{"class":503},[151,80787,80788,80791,80794],{"class":469,"line":2462},[151,80789,80790],{"class":503},"          overflow:",[151,80792,80793],{"class":481},"'hidden'",[151,80795,9417],{"class":503},[151,80797,80798,80801,80803],{"class":469,"line":2471},[151,80799,80800],{"class":503},"          width:",[151,80802,78135],{"class":481},[151,80804,9417],{"class":503},[151,80806,80807,80810,80813],{"class":469,"line":2480},[151,80808,80809],{"class":503},"          height:",[151,80811,80812],{"class":481},"'25px'",[151,80814,9417],{"class":503},[151,80816,80817,80820,80823],{"class":469,"line":2489},[151,80818,80819],{"class":503},"          backgroundColor:",[151,80821,80822],{"class":481},"'#e4e4a1'",[151,80824,9417],{"class":503},[151,80826,80827,80830,80833],{"class":469,"line":2497},[151,80828,80829],{"class":503},"          color:",[151,80831,80832],{"class":481},"'red'",[151,80834,9417],{"class":503},[151,80836,80837,80840,80842],{"class":469,"line":3140},[151,80838,80839],{"class":503},"          boarderColor: ",[151,80841,45401],{"class":481},[151,80843,9417],{"class":503},[151,80845,80846,80849],{"class":469,"line":3149},[151,80847,80848],{"class":503},"          border:",[151,80850,80851],{"class":481},"\".5px solid black\"\n",[151,80853,80854,80857],{"class":469,"line":3158},[151,80855,80856],{"class":503},"        }",[151,80858,6274],{"class":78245},[151,80860,80861,80864,80866,80868,80870,80873,80875],{"class":469,"line":3167},[151,80862,80863],{"class":473},"      onClick",[151,80865,1876],{"class":1869},[151,80867,5729],{"class":78245},[151,80869,23252],{"class":15289},[151,80871,80872],{"class":503},".props.handleClick",[151,80874,2001],{"class":78245},[151,80876,80877],{"class":503}," >\n",[151,80879,80880,80882],{"class":469,"line":3175},[151,80881,21070],{"class":503},[151,80883,48450],{"class":14368},[151,80885,80886,80889,80891,80893],{"class":469,"line":3184},[151,80887,80888],{"class":473},"          style",[151,80890,1876],{"class":1869},[151,80892,5729],{"class":78245},[151,80894,80895],{"class":503},"{color:color_,\n",[151,80897,80898,80901,80904],{"class":469,"line":3193},[151,80899,80900],{"class":503},"                  border:",[151,80902,80903],{"class":481},"\"1px solid\"",[151,80905,9417],{"class":503},[151,80907,80908],{"class":469,"line":3720},[151,80909,80910],{"class":503},"                  backgroundColor: color_,\n",[151,80912,80913,80916,80919],{"class":469,"line":3729},[151,80914,80915],{"class":503},"                  borderRadius: ",[151,80917,80918],{"class":481},"\"50%\"",[151,80920,9417],{"class":503},[151,80922,80923],{"class":469,"line":3735},[151,80924,80925],{"class":503},"                  borderColor: color_,\n",[151,80927,80928,80931,80934,80936,80938],{"class":469,"line":3745},[151,80929,80930],{"class":503},"                  height:",[151,80932,80933],{"class":477},"25",[151,80935,2001],{"class":503},[151,80937,2001],{"class":78245},[151,80939,80877],{"class":503},[151,80941,80942,80944,80946],{"class":469,"line":3754},[151,80943,21175],{"class":503},[151,80945,23950],{"class":14368},[151,80947,3742],{"class":503},[151,80949,80950,80952,80954],{"class":469,"line":3760},[151,80951,48707],{"class":503},[151,80953,1158],{"class":14368},[151,80955,3742],{"class":503},[151,80957,80958],{"class":469,"line":3773},[151,80959,39567],{"class":503},[151,80961,80962],{"class":469,"line":3782},[151,80963,19957],{"class":503},[151,80965,80966],{"class":469,"line":3791},[151,80967,6274],{"class":503},[151,80969,80970,80972,80974,80976,80978],{"class":469,"line":3803},[151,80971,5729],{"class":503},[151,80973,44519],{"class":1869},[151,80975,78869],{"class":503},[151,80977,44519],{"class":1869},[151,80979,6274],{"class":503},[11,80981,80982],{},"So far, I love using React. These examples only scratch the surface of what you can do with the library, but even with these examples you can see how easy it is to keep track of state. By making use of the virtual DOM, React only rerenders the part of the actual DOM for which there has been a change in state.",[11,80984,80985],{},"There's more work to do on these games, but I will be putting them aside to learn more about React's lifecycle methods and how to do routing with react-router. Ultimately I'm working toward building React as the frontend to a Django Rest Framework API in some type of single page application (or not, there are actually a few different ways to combine React with Django and I hope to get into that soon!) I will also be learning more about Redux to see how complicated state can be easily managed by using React and Redux together.",[11,80987,80988],{},"Leave a comment if you have any questions or want to point out any errors I may have made in the code above. Thanks!",[589,80990,80991],{},"html pre.shiki code .sMOD_, html code.shiki .sMOD_{--shiki-default:#24292E;--shiki-dark:#E1E4E8;--shiki-sepia:#F8F8F2}html pre.shiki code .sC2Qs, html code.shiki .sC2Qs{--shiki-default:#D73A49;--shiki-dark:#F97583;--shiki-sepia:#F92672}html pre.shiki code .sstjo, html code.shiki .sstjo{--shiki-default:#032F62;--shiki-dark:#9ECBFF;--shiki-sepia:#E6DB74}html pre.shiki code .sq6CD, html code.shiki .sq6CD{--shiki-default:#D73A49;--shiki-default-font-style:inherit;--shiki-dark:#F97583;--shiki-dark-font-style:inherit;--shiki-sepia:#66D9EF;--shiki-sepia-font-style:italic}html pre.shiki code .sz2Vg, html code.shiki .sz2Vg{--shiki-default:#6F42C1;--shiki-default-text-decoration:inherit;--shiki-dark:#B392F0;--shiki-dark-text-decoration:inherit;--shiki-sepia:#A6E22E;--shiki-sepia-text-decoration:underline}html pre.shiki code .s30JN, html code.shiki .s30JN{--shiki-default:#6F42C1;--shiki-default-font-style:inherit;--shiki-default-text-decoration:inherit;--shiki-dark:#B392F0;--shiki-dark-font-style:inherit;--shiki-dark-text-decoration:inherit;--shiki-sepia:#A6E22E;--shiki-sepia-font-style:italic;--shiki-sepia-text-decoration:underline}html pre.shiki code .sTHNf, html code.shiki .sTHNf{--shiki-default:#E36209;--shiki-default-font-style:inherit;--shiki-dark:#FFAB70;--shiki-dark-font-style:inherit;--shiki-sepia:#FD971F;--shiki-sepia-font-style:italic}html pre.shiki code .sP7S_, html code.shiki .sP7S_{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#FD971F}html pre.shiki code .s7F3e, html code.shiki .s7F3e{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#AE81FF}html pre.shiki code .srTi1, html code.shiki .srTi1{--shiki-default:#6F42C1;--shiki-dark:#B392F0;--shiki-sepia:#A6E22E}html pre.shiki code .sXSQT, html code.shiki .sXSQT{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#F8F8F2}html pre.shiki code .s5clZ, html code.shiki .s5clZ{--shiki-default:#22863A;--shiki-dark:#85E89D;--shiki-sepia:#F92672}html pre.shiki code .s-ngx, html code.shiki .s-ngx{--shiki-default:#24292E;--shiki-dark:#E1E4E8;--shiki-sepia:#F92672}html pre.shiki code .s-m8C, html code.shiki .s-m8C{--shiki-default:#005CC5;--shiki-default-font-style:inherit;--shiki-dark:#79B8FF;--shiki-dark-font-style:inherit;--shiki-sepia:#66D9EF;--shiki-sepia-font-style:italic}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html .sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html.sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html pre.shiki code .s8-w5, html code.shiki .s8-w5{--shiki-default:#6A737D;--shiki-dark:#6A737D;--shiki-sepia:#88846F}",{"title":464,"searchDepth":488,"depth":488,"links":80993},[],"2017-10-03","To ease into learning ReactJS, I took a shot at implementing a simple tic-tac-toe game with React. This is covered in the official Facebook React tutorial, but I haven't actually looked at how they did this yet. Instead, I wanted to see how far I could get on my own, and then fallback to the tutorial if I needed help. I heard that there are many different ways that React components can be organized and structured in a React project, so I wanted to see how my results compared to what the official tutorial recommends.",{"layout":48045},"/2017/10/03/simple-games-in-react",{"title":77025,"description":80995},"2017/10/03/simple-games-in-react",[81001],"react","Fepvnw-VgL4u2Qp85nPyu6DGbG3P5IZd5Mz_nnFfnwg",{"id":81004,"title":81005,"body":81006,"comments":609,"date":83565,"description":83566,"draft":602,"extension":605,"external":606,"image":83567,"meta":83568,"navigation":609,"path":83569,"seo":83570,"stem":83571,"tags":83572,"__hash__":83573},"blog/2017/08/03/arch-linux-installation-guide.md","Arch Linux Installation Guide",{"type":8,"value":81007,"toc":83488},[81008,81017,81021,81035,81042,81045,81048,81054,81057,81060,81069,81081,81085,81088,81094,81097,81103,81106,81112,81115,81121,81128,81133,81136,81139,81145,81152,81156,81182,81186,81211,81215,81244,81248,81254,81273,81283,81286,81292,81296,81299,81305,81311,81317,81323,81327,81333,81339,81345,81348,81352,81358,81361,81365,81368,81374,81380,81386,81389,81395,81398,81403,81407,81413,81423,81426,81432,81439,81443,81446,81448,81454,81467,81471,81481,81487,81492,81496,81502,81509,81515,81521,81527,81534,81540,81546,81549,81555,81561,81565,81571,81575,81582,81588,81592,81598,81604,81607,81612,81616,81621,81627,81633,81637,81640,81646,81650,81656,81659,81663,81677,81683,81686,81692,81696,81702,81706,81712,81721,81726,81732,81735,81741,81746,81749,81755,81777,81781,81784,81790,81793,81799,81804,81813,81816,81822,81827,81831,81834,81840,81844,81847,81850,81856,81861,81867,81870,81876,81880,81886,81892,81898,81901,81907,81910,81916,81919,81925,81928,81934,81937,81941,81947,81951,81954,81957,81963,81966,81972,81975,81981,81984,81990,81996,82002,82008,82011,82017,82023,82029,82035,82038,82045,82049,82056,82062,82067,82071,82077,82081,82087,82091,82097,82101,82107,82112,82115,82121,82124,82130,82135,82141,82148,82154,82157,82161,82172,82185,82188,82194,82197,82200,82206,82213,82217,82220,82227,82230,82236,82239,82245,82248,82251,82257,82260,82277,82286,82292,82309,82312,82318,82321,82327,82330,82336,82343,82354,82358,82370,82373,82379,82382,82415,82418,82422,82425,82431,82437,82443,82450,82456,82460,82463,82467,82474,82478,82488,82492,82498,82502,82505,82509,82516,82526,82530,82533,82538,82541,82549,82556,82561,82567,82570,82576,82579,82584,82587,82593,82603,82608,82611,82617,82626,82641,82647,82650,82656,82667,82671,82674,82680,82684,82688,82691,82697,82710,82717,82720,82725,82731,82734,82740,82745,82749,82752,82758,82761,82764,82768,82777,82784,82790,82793,82799,82802,82808,82814,82820,82827,82833,82836,82842,82851,82857,82863,82866,82870,82876,82878,82884,82891,82895,82898,82900,82906,82912,82918,82922,82928,82937,82943,82946,82952,82956,82964,82970,82973,82979,82986,82989,82994,83000,83003,83005,83011,83017,83025,83033,83038,83044,83049,83055,83058,83062,83065,83071,83077,83087,83092,83096,83101,83105,83111,83117,83120,83123,83129,83132,83138,83145,83151,83157,83166,83169,83175,83177,83183,83187,83190,83196,83199,83205,83212,83215,83221,83224,83228,83231,83237,83240,83246,83250,83253,83259,83262,83268,83272,83286,83292,83296,83299,83303,83306,83320,83323,83329,83338,83342,83345,83351,83358,83362,83365,83371,83374,83378,83381,83387,83390,83396,83400,83403,83406],[11,81009,81010,81011,81016],{},"This post will be a comprehensive guide to installing Arch Linux. I will be installing Arch Linux on a refurbished ",[20,81012,81015],{"href":81013,"rel":81014},"http://www3.lenovo.com/us/en/laptops/thinkpad/t-series/t430/",[24],"ThinkPad T430 Laptop",", first as a guest OS on a Windows 10 Pro host, and then on a secondary SSD that will replace the optical drive. I will try to cover absolutely everything you need from creating a bootable USB drive to customizing the desktop interface and installing development environments and tools.",[56,81018,81020],{"id":81019},"put-arch-linux-on-a-usb-drive","Put Arch Linux on a USB drive",[11,81022,81023,81024,81029,81030,81034],{},"I recommend that you download a BitTorrent client such as ",[20,81025,81028],{"href":81026,"rel":81027},"http://dev.deluge-torrent.org/wiki/Download",[24],"Deluge",", and then grab the ISO magnet link from ",[20,81031,81032],{"href":81032,"rel":81033},"https://www.archlinux.org/download/",[24],". It should take just a few minutes to complete the download via BitTorrent.",[11,81036,81037,81038,81041],{},"If you are using a Mac to build the bootable drive, I have read that you can use the ",[30,81039,81040],{},"dd"," command. This didn't work for me.",[11,81043,81044],{},"I found out that my 2011 MacBook Air is not capable of creating Windows 10 bootable media. Only the options for Windows 7 and 8 were available in Boot Camp Assistant (the Mac utility for installing Windows and creating bootable media). I also found that the latest MacBooks can't create bootable media. I was able to find another Mac that had the option for creating \"Windows 7 or later\", and this finally worked.",[11,81046,81047],{},"Another solution that is probably much faster is using Rufus, a Windows/Linux utility for burning ISO images to bootable media. The Arch Wiki has a highlighted note about using Rufus:",[459,81049,81052],{"className":81050,"code":81051,"language":997},[995],"Note: Be sure to select DD mode or the image will be transferred incorrectly.\n",[30,81053,81051],{"__ignoreMap":464},[11,81055,81056],{},"A problem I found is that you can't select the ISO with DD mode selected. I was able to select ISO (in ISO mode), choose the ISO file in my downloads folder, and then select DD mode.",[14063,81058,81059],{"id":69706},"VirtualBox",[11,81061,81062,81063,81068],{},"The T430 came with a copy of Windows Pro, which I installed right away. I also picked up a SSD sleeve that can replace the optical drive for dual booting or additional storage. I have installed Arch Linux on my Windows desktop machine, as well as my MacBook Air. Land of Linux has a ",[20,81064,81067],{"href":81065,"rel":81066},"http://landoflinux.com/linux_install_archlinux_process.html",[24],"great tutorial on installing Arch Linux in Virtual Box",", but it is frustrating because there is no date and no comments. This part of the guide will closely follow this guide and provide some corrections where you might get stuck following the directions command for command.",[11,81070,81071,81072,63177,81074,19883,81077,81080],{},"First we want to set up a new virtual machine with the options Linx / ArchLinux / 64-bit. You are probably using a 64-bit machine (although some old netbooks are 32-bit). The first obstacle is you will probably face with VirtualBox is that there are no options for 64-bit. This is because Intel Virtualization is often disabled by default. You will need to go into the BIOS (by pressing ",[30,81073,34791],{},[30,81075,81076],{},"F10",[30,81078,81079],{},"F12"," at boot time) and then change this option, save the BIOS settings and restart. With this done, you will want to create a virtual machine with at at least 20GB of storage and around 4GB of memory.",[54716,81082,81084],{"id":81083},"quick-note-about-fonts","Quick note about fonts",[11,81086,81087],{},"The default font when installing Arch Linux is quite small, so you may want to increase the font size.",[459,81089,81092],{"className":81090,"code":81091,"language":997},[995],"ls /usr/share/kbd/consolefonts/\n",[30,81093,81091],{"__ignoreMap":464},[11,81095,81096],{},"This command will list the available fonts. To set a font, run the following command:",[459,81098,81101],{"className":81099,"code":81100,"language":997},[995],"setfont sun12x22\n",[30,81102,81100],{"__ignoreMap":464},[11,81104,81105],{},"To revert back to the default font, just run:",[459,81107,81110],{"className":81108,"code":81109,"language":997},[995],"setfont\n",[30,81111,81109],{"__ignoreMap":464},[11,81113,81114],{},"You can also view the current font with",[459,81116,81119],{"className":81117,"code":81118,"language":997},[995],"showconsolefont\n",[30,81120,81118],{"__ignoreMap":464},[736,81122,32889,81124,81127],{"id":81123},"using-fdisk-for-partitioning",[30,81125,81126],{},"fdisk"," for partitioning",[210,81129,81130],{},[11,81131,81132],{},"fdisk is a dialogue-driven program for creation and manipulation of partition tables.",[11,81134,81135],{},"This part of the guide will be similar for installation on virtual machines and regular installions (on harddrives in desktops and laptops). If you are installing on a physical machine that has other disks of similar size to your target disk, I recommend that you first remove all other drives so you don't accidentaly format partitions (I did this by accident, but luckily it was just my Windows Boot Partition and I was able to recover my Windows installation).",[11,81137,81138],{},"We want to run fdisk and enter the device we want to work with as the first argument after the command:",[459,81140,81143],{"className":81141,"code":81142,"language":997},[995],"fdisk /dev/sda\n",[30,81144,81142],{"__ignoreMap":464},[11,81146,81147,81148,81151],{},"On your Virtual Machine it will most likely be ",[15,81149,81150],{},"sda",", but on a physical machine it could vary slightly.",[54716,81153,81155],{"id":81154},"note-about-fdisk","Note about fdisk",[11,81157,81158,81159,30583,81162,30583,81165,81168,81169,81172,81173,49861,81176,81178,81179,81181],{},"When installing Arch Linux on a disk that has Windows or Linux installed (and you want delete the existing Windows or Linux completely to make room for a fresh Arch Linux istallation), you can delete the partitions on the drive (which is usually labeled ",[30,81160,81161],{},"/dev/sda",[30,81163,81164],{},"/dev/sdb",[30,81166,81167],{},"/dev/sdc","). Run ",[30,81170,81171],{},"lsblk"," to determine which drive you want to format, then run ",[30,81174,81175],{},"fdisk /dev/sdX",[30,81177,56498],{}," corresponds to the drive we will be using. You should be inside the fdisk menu. Press ",[30,81180,78271],{}," to start deleting partitions. You may get some warnings about a partition containing some type of trace (such as ext4). This is fine. You can delete all of the partitions.",[736,81183,81185],{"id":81184},"create-a-root-partition","Create a root partition",[11,81187,81188,81189,81191,81192,81194,81195,81198,81199,81202,81203,81206,81207,81210],{},"Press ",[30,81190,8521],{}," and then ",[30,81193,11],{},". When prompted for ",[30,81196,81197],{},"first sector"," press ",[30,81200,81201],{},"ENTER",", and for ",[30,81204,81205],{},"last sector"," enter ",[30,81208,81209],{},"+10G"," for a 10GB partition.",[736,81212,81214],{"id":81213},"create-a-swap-partition","Create a SWAP partition",[11,81216,81188,81217,81219,81220,81222,81223,81225,81226,81228,81229,81232,81233,81236,81237,81239,81240,81243],{},[30,81218,8521],{}," for a new partition, press ",[30,81221,11],{}," for primary partition. Select partition number ",[30,81224,6619],{},". Press ",[30,81227,81201],{}," to select the default start sector, and then type ",[30,81230,81231],{},"+1G"," to create a 1GB swap partition. Press ",[30,81234,81235],{},"t"," to indicate the type of partition this will be. Specify that we are still working with partition ",[30,81238,6619],{},", and then enter ",[30,81241,81242],{},"82",", the code for SWAP partition.",[736,81245,81247],{"id":81246},"create-home-partition","Create home partition",[11,81249,81250,81251,81253],{},"Enter the following ",[30,81252,81126],{}," commands:",[11,81255,81256,106,81258,106,81260,106,81262,106,81264,106,81266,106,81268,106,81270,643],{},[30,81257,8521],{},[30,81259,11],{},[30,81261,6557],{},[30,81263,81201],{},[30,81265,81201],{},[30,81267,81235],{},[30,81269,6557],{},[30,81271,81272],{},"83",[11,81274,81275,81276,81278,81279,81282],{},"Before exiting the ",[30,81277,81126],{}," interface, enter the ",[30,81280,81281],{},"w"," command to write the proposed changes.",[11,81284,81285],{},"We should see the following:",[459,81287,81290],{"className":81288,"code":81289,"language":997},[995],"The partition table has been altered.\nCalling ioctl() to re-read partition table.\nSyncing disks.\n",[30,81291,81289],{"__ignoreMap":464},[736,81293,81295],{"id":81294},"create-filesystems-and-format-partitions","Create Filesystems and Format Partitions",[11,81297,81298],{},"Make filesystems for partitions 1 and 3:",[459,81300,81303],{"className":81301,"code":81302,"language":997},[995],"mkfs.ext4 /dev/sda1\nmkfs.ext4 /dev/sda3\n",[30,81304,81302],{"__ignoreMap":464},[459,81306,81309],{"className":81307,"code":81308,"language":997},[995],"mkswap /dev/sda2\nswapon /dev/sda2\n",[30,81310,81308],{"__ignoreMap":464},[736,81312,81314,81315],{"id":81313},"verify-our-layout-with-lsblk","Verify our layout with ",[30,81316,81171],{},[459,81318,81321],{"className":81319,"code":81320,"language":997},[995],"lsblk\n",[30,81322,81320],{"__ignoreMap":464},[736,81324,81326],{"id":81325},"mount-filesystems","Mount Filesystems",[459,81328,81331],{"className":81329,"code":81330,"language":997},[995],"mount /dev/sda1 /mnt\n",[30,81332,81330],{"__ignoreMap":464},[459,81334,81337],{"className":81335,"code":81336,"language":997},[995],"mkdir /mnt/home\n",[30,81338,81336],{"__ignoreMap":464},[459,81340,81343],{"className":81341,"code":81342,"language":997},[995],"mount /dev/sda3 /mnt/home\n",[30,81344,81342],{"__ignoreMap":464},[11,81346,81347],{},"(The swap partition has already been initialized.)",[736,81349,81351],{"id":81350},"backup-mirrorlist","Backup mirrorlist",[459,81353,81356],{"className":81354,"code":81355,"language":997},[995],"cp -vf /etc/pacman.d/mirrorlist /etc/pacman.d/mirrorlist.backup\n",[30,81357,81355],{"__ignoreMap":464},[11,81359,81360],{},"We can edit the mirrorlist to select the closest mirror, but this is not necessary.",[736,81362,81364],{"id":81363},"install-reflector","Install reflector",[11,81366,81367],{},"This synchronizes package databases:",[459,81369,81372],{"className":81370,"code":81371,"language":997},[995],"pacman -Syy\n",[30,81373,81371],{"__ignoreMap":464},[11,81375,81376,81377,643],{},"Next we want to install our first package: ",[30,81378,81379],{},"reflector",[459,81381,81384],{"className":81382,"code":81383,"language":997},[995],"pacman -S reflector\n",[30,81385,81383],{"__ignoreMap":464},[11,81387,81388],{},"The following command will sort the five best mirrors based on your location:",[459,81390,81393],{"className":81391,"code":81392,"language":997},[995],"reflector --verbose -l 5 --sort rate --save /etc/pacman.d/mirrorlist\n",[30,81394,81392],{"__ignoreMap":464},[11,81396,81397],{},"Re-issue the sync command:",[459,81399,81401],{"className":81400,"code":81371,"language":997},[995],[30,81402,81371],{"__ignoreMap":464},[736,81404,81406],{"id":81405},"installing-the-base","Installing the base",[11,81408,81409,81410,643],{},"The base installation is installed using a special script called ",[30,81411,81412],{},"pacstrap",[11,81414,81415,81416,81418,81419,81422],{},"We will be using the ",[30,81417,26473],{}," group and also the ",[30,81420,81421],{},"base-devel"," group that we will need later.",[11,81424,81425],{},"Issue the following command:",[459,81427,81430],{"className":81428,"code":81429,"language":997},[995],"pacstrap /mnt base base-devel\n",[30,81431,81429],{"__ignoreMap":464},[11,81433,81434,81435,81438],{},"This will install lots of packages; it takes a few minutes. It took me ",[30,81436,81437],{},"4:27.06"," on my MacBook Air and about 2.5 minutes on my ThinkPad.",[736,81440,81442],{"id":81441},"create-etcfstab","Create /etc/fstab",[11,81444,81445],{},"The next step is to create a mount table that will be responsible for automatically mounting filesystems at boot time.",[11,81447,81425],{},[459,81449,81452],{"className":81450,"code":81451,"language":997},[995],"genfstab -U -p /mnt >> /mnt/etc/fstab\n",[30,81453,81451],{"__ignoreMap":464},[11,81455,81456,81457,81460,81461,49861,81464,81466],{},"We can verify the entries with ",[30,81458,81459],{},"cat /mnt/etc/fstab",", or ",[30,81462,81463],{},"blkid /dev/sd#",[30,81465,23702],{}," is the partition number.",[736,81468,81470],{"id":81469},"chroot-configuring-the-base-system","Chroot - Configuring the base system",[11,81472,81473,81474,81477,81478,81480],{},"To configure our new installation, we need to issue the ",[30,81475,81476],{},"arch-chroot"," command. This command is used as follows: ",[30,81479,81476],{}," followed by the new root directory as the first argument.",[459,81482,81485],{"className":81483,"code":81484,"language":997},[995],"root@archiso ~ # arch-chroot /mnt /bin/bash\n[root@archiso /]#\n",[30,81486,81484],{"__ignoreMap":464},[11,81488,81489,81490,643],{},"We could pass in additional arguments to specify a particular shell, but here we are using ",[30,81491,463],{},[736,81493,81495],{"id":81494},"configuring-locale-settings","Configuring locale settings",[459,81497,81500],{"className":81498,"code":81499,"language":997},[995],"nano /etc/locale.gen\n",[30,81501,81499],{"__ignoreMap":464},[11,81503,81504,81505,81508],{},"Uncomment ",[30,81506,81507],{},"en_US.UTF-8 UTF-8"," for US English.",[11,81510,81511,81512,643],{},"Once the desired locales have been uncommented, run ",[30,81513,81514],{},"local-gen",[11,81516,81517,81518,208],{},"Next we will create ",[30,81519,81520],{},"/etc/locale.conf",[459,81522,81525],{"className":81523,"code":81524,"language":997},[995],"echo LANG=en_US.UTF-8 > /etc/locale.conf\nexport LANG=en_US.UTF-8\n",[30,81526,81524],{"__ignoreMap":464},[11,81528,81529,81530,81533],{},"Create the following file: ",[30,81531,81532],{},"/etc/vconsole.conf"," and add the following line:",[459,81535,81538],{"className":81536,"code":81537,"language":997},[995],"KEYMAP=us\n",[30,81539,81537],{"__ignoreMap":464},[11,81541,81542,81543,643],{},"To configure the time zone, we need to create a link to a file called ",[30,81544,81545],{},"/etc/localtime",[11,81547,81548],{},"The following command won't work, we will do this in a later step, so skip the following command for now.",[459,81550,81553],{"className":81551,"code":81552,"language":997},[995],"ln -s /usr/share/zoneinfo/US/Eastern /etc/localtime\n",[30,81554,81552],{"__ignoreMap":464},[11,81556,81557,81558,81560],{},"This worked on my MacBook Air in VirtualBox, but in doing this on my Windows host in VirtualBox trying to link to the file ",[30,81559,81545],{}," gave an error that the file already existed. Skipping this step on my ThinkPad was OK. Looking in the file, it seemed that UTC was already configured.",[736,81562,81564],{"id":81563},"set-the-hardware-clock","Set the Hardware Clock",[459,81566,81569],{"className":81567,"code":81568,"language":997},[995],"hwclock --systohc --utc\n",[30,81570,81568],{"__ignoreMap":464},[736,81572,81574],{"id":81573},"set-the-host-name-for-the-system","Set the Host Name for the System",[11,81576,81577,81578,81581],{},"I'm using the hostname ",[30,81579,81580],{},"archthinkpad",", you can pick whatever host name you want.",[459,81583,81586],{"className":81584,"code":81585,"language":997},[995],"echo archthinkpad > /etc/hostname\n",[30,81587,81585],{"__ignoreMap":464},[736,81589,81591],{"id":81590},"enable-multilib-repository","Enable Multilib Repository",[11,81593,81594,81595,208],{},"Add the following to ",[30,81596,81597],{},"/etc/pacman.conf",[459,81599,81602],{"className":81600,"code":81601,"language":997},[995],"[multilib]\nInclude = /etc/pacman.d/mirrorlist\n",[30,81603,81601],{"__ignoreMap":464},[11,81605,81606],{},"And then run",[459,81608,81610],{"className":81609,"code":81371,"language":997},[995],[30,81611,81371],{"__ignoreMap":464},[736,81613,81615],{"id":81614},"add-support-for-yaourt-package-tool","Add support for Yaourt Package Tool",[11,81617,81618,81619,208],{},"Add the following to the end of ",[30,81620,81597],{},[459,81622,81625],{"className":81623,"code":81624,"language":997},[995],"[archlinuxfr]\nSigLevel = Never\nServer = http://repo.archlinux.fr/$arch\n",[30,81626,81624],{"__ignoreMap":464},[11,81628,81629,81630,643],{},"Then run ",[30,81631,81632],{},"pacman -Syy",[736,81634,81636],{"id":81635},"synchronize-and-update-database-packages","Synchronize and Update Database Packages",[11,81638,81639],{},"Run the \"update system\" command:",[459,81641,81644],{"className":81642,"code":81643,"language":997},[995],"pacman -Syu\n",[30,81645,81643],{"__ignoreMap":464},[736,81647,81649],{"id":81648},"create-a-password-for-the-root-account","Create a password for the root account",[459,81651,81654],{"className":81652,"code":81653,"language":997},[995],"passwd\n",[30,81655,81653],{"__ignoreMap":464},[11,81657,81658],{},"and enter a password for the root user.",[736,81660,81662],{"id":81661},"add-a-new-user-with-sudo-privileges","Add a new user with sudo privileges",[11,81664,81665,81666,81669,81670,81673,81674,643],{},"To add a new user we issue ",[30,81667,81668],{},"useradd"," along with primary and secondary groups. ",[30,81671,81672],{},"wheel"," group members will be able to issue commands prefixed with ",[30,81675,81676],{},"sudo",[459,81678,81681],{"className":81679,"code":81680,"language":997},[995],"useradd -mg users -G wheel,storage,power -s /bin/bash brian\n",[30,81682,81680],{"__ignoreMap":464},[11,81684,81685],{},"Then set a password for this user:",[459,81687,81690],{"className":81688,"code":81689,"language":997},[995],"passwd brian\n",[30,81691,81689],{"__ignoreMap":464},[736,81693,81695],{"id":81694},"install-sudo-package","Install sudo Package",[459,81697,81700],{"className":81698,"code":81699,"language":997},[995],"pacman -S sudo\n",[30,81701,81699],{"__ignoreMap":464},[736,81703,81705],{"id":81704},"configuring-a-sudo-adding-user-to-wheel-group","Configuring a sudo - adding user to wheel group",[11,81707,81708,81709,643],{},"We now need to configure the sudo configuration file. In order to edit this file we must use a special command called ",[30,81710,81711],{},"visudo",[11,81713,81714,81715,30583,81718,81720],{},"This is an editor like ",[30,81716,81717],{},"vi",[30,81719,76719],{}," and it edits the sudo config file.",[11,81722,69440,81723,81725],{},[30,81724,81711],{},", we might see the follwoing error:",[459,81727,81730],{"className":81728,"code":81729,"language":997},[995],"visudo: specified editor (vim) doesn't exist.\n",[30,81731,81729],{"__ignoreMap":464},[11,81733,81734],{},"Let's install vim:",[459,81736,81739],{"className":81737,"code":81738,"language":997},[995],"pacman -S vim\n",[30,81740,81738],{"__ignoreMap":464},[11,81742,81743,81744,643],{},"Now we should be able to run ",[30,81745,81711],{},[11,81747,81748],{},"We need to uncomment the following line:",[459,81750,81753],{"className":81751,"code":81752,"language":997},[995],"# %wheel ALL=(ALL) ALL\n",[30,81754,81752],{"__ignoreMap":464},[11,81756,81757,81758,81760,81761,81763,81764,81767,81768,81191,81770,81767,81772,81774,81775,643],{},"To do this, press the down arrow until we are on the line we need to uncomment, then press ",[30,81759,77563],{},", then remove the ",[30,81762,23702],{}," and space. Next, press ",[30,81765,81766],{},"esc"," and then press ",[30,81769,208],{},[30,81771,11126],{},[30,81773,81201],{},". That should save our changes. Arch Linux warns that this file ONLY be edited with ",[30,81776,76719],{},[736,81778,81780],{"id":81779},"installing-a-bootloader","Installing a bootloader",[11,81782,81783],{},"At this point the directions will fork based on your motherboard firmware. By default grub-install uses target x86_64-efi. If you don't have a UEFI system you should use the following command:",[459,81785,81788],{"className":81786,"code":81787,"language":997},[995],"pacman -S grub\ngrub-install --target=i386-pc --recheck /dev/sdb\n",[30,81789,81787],{"__ignoreMap":464},[11,81791,81792],{},"If this works properly you should see the following:",[459,81794,81797],{"className":81795,"code":81796,"language":997},[995],"Installing for i386-pc platform.\nInstallation finished. No error reported.\n",[30,81798,81796],{"__ignoreMap":464},[210,81800,81801],{},[11,81802,81803],{},"--target=i386-pc instructs grub-install to install for BIOS systems only. It is recommended to always use this option to remove ambiguity in grub-install.",[11,81805,81806,81807,81812],{},"Here is ",[20,81808,81811],{"href":81809,"rel":81810},"https://wiki.archlinux.org/index.php/GRUB",[24],"more information about GRUB"," from the Arch Wiki.",[11,81814,81815],{},"Finally run the following command:",[459,81817,81820],{"className":81818,"code":81819,"language":997},[995],"grub-mkconfig -o /boot/grub/grub.cfg\n",[30,81821,81819],{"__ignoreMap":464},[11,81823,81824],{},[151,81825,81826],{},"Note: If you had a multi OS environment, you could install the \"os-prober\" package with \"pacman -S os-prober\" before running the \"grub-mkconfig\" command. Os-prober will detect other operating systems when generating the grub.cfg file.",[736,81828,81830],{"id":81829},"arch-linux-is-now-installed","Arch Linux is now installed",[11,81832,81833],{},"Issue the following commands to exit:",[459,81835,81838],{"className":81836,"code":81837,"language":997},[995],"exit\numount /mnt/home\numount /mnt\nreboot\n",[30,81839,81837],{"__ignoreMap":464},[736,81841,81843],{"id":81842},"boot-from-existing-os","Boot from existing OS",[11,81845,81846],{},"At the startup screen, select \"Boot existing OS\".",[11,81848,81849],{},"I got a strange error message at login:",[459,81851,81854],{"className":81852,"code":81853,"language":997},[995],"arch20815 login: [9.921142] piix4_smbus 0000:00:07.0: SMBus base address uninitialized - upgrade BIOS or use force_addr=0xaddr\n",[30,81855,81853],{"__ignoreMap":464},[11,81857,81188,81858,81860],{},[30,81859,81201],{}," to return to the login prompt.",[11,81862,81863,81864,81866],{},"Login with the ",[30,81865,81676],{}," user we created.",[11,81868,81869],{},"We can check to see if this user was setup properly by running:",[459,81871,81874],{"className":81872,"code":81873,"language":997},[995],"[brian@arch20815 ~]$ sudo whoami\n\nWe trust you have received the usual lecture from the local System Administrator. It usually boils down to these three things:\n\n    #1) Respect the privacy of others.\n    #2) Think before you type.\n    #3) With great power comes great responsibility.\n\n[sudo] password for brian:\nroot\n[brian@arch20815 ~]$\n",[30,81875,81873],{"__ignoreMap":464},[736,81877,81879],{"id":81878},"configuring-the-network","Configuring the Network",[11,81881,81882,81883,643],{},"After the reboot we will have lost network connectivity. We need to create a new definition to our interface. We can isssue the command ",[30,81884,81885],{},"ip link",[11,81887,10635,81888,81891],{},[30,81889,81890],{},"sudo vi /etc/systemd/network/enp0s3.netork"," and add the following to this file:",[459,81893,81896],{"className":81894,"code":81895,"language":997},[995],"[Match]\nName=en*\n\n[Network]\nDHCP=yes\n",[30,81897,81895],{"__ignoreMap":464},[11,81899,81900],{},"Then we need to run the following commands:",[459,81902,81905],{"className":81903,"code":81904,"language":997},[995],"sudo systemctl restart systemd-networkd\nsudo systemctl enable systemd-networkd\n",[30,81906,81904],{"__ignoreMap":464},[11,81908,81909],{},"Next issue the following command:",[459,81911,81914],{"className":81912,"code":81913,"language":997},[995],"sudo vi /etc/resolv.conf\n",[30,81915,81913],{"__ignoreMap":464},[11,81917,81918],{},"And add the following to this file:",[459,81920,81923],{"className":81921,"code":81922,"language":997},[995],"nameserver 8.8.8.8\nnameserver 8.8.4.4\n",[30,81924,81922],{"__ignoreMap":464},[11,81926,81927],{},"Next we can check connectivity:",[459,81929,81932],{"className":81930,"code":81931,"language":997},[995],"ip a s\nping -c 3 www.google.com\n",[30,81933,81931],{"__ignoreMap":464},[11,81935,81936],{},"We should see that we have been assigned an IP address and that we can ping google.com.",[736,81938,81940],{"id":81939},"run-a-full-system-update","Run a full System update",[459,81942,81945],{"className":81943,"code":81944,"language":997},[995],"sudo pacman -Syu\n",[30,81946,81944],{"__ignoreMap":464},[736,81948,81950],{"id":81949},"installing-the-x-environment","Installing the X Environment",[11,81952,81953],{},"I have had problems with this step in the past.",[11,81955,81956],{},"The next step in the tutorial is to run:",[459,81958,81961],{"className":81959,"code":81960,"language":997},[995],"sudo pacman -S xorg-server xorg-xinit xorg-utils xorg-server-utils mesa xorg-twm xterm xorg-xclock\n",[30,81962,81960],{"__ignoreMap":464},[11,81964,81965],{},"But this will result in:",[459,81967,81970],{"className":81968,"code":81969,"language":997},[995],"error: target not found: xorg-utils\nerror: target not found: xorg-server-utils\n",[30,81971,81969],{"__ignoreMap":464},[11,81973,81974],{},"It seems that these packages have been deprecated. I think a solution to this is to run:",[459,81976,81979],{"className":81977,"code":81978,"language":997},[995],"sudo pacman -S xorg-apps\n",[30,81980,81978],{"__ignoreMap":464},[11,81982,81983],{},"and then install the remaining packages:",[459,81985,81988],{"className":81986,"code":81987,"language":997},[995],"sudo pacman -S xorg-server xorg-xinit mesa xorg-twm xterm xorg-xclock\n",[30,81989,81987],{"__ignoreMap":464},[11,81991,81992,81993,81995],{},"Select option ",[30,81994,6760],{}," (don't select the options that include the word nvidia).",[11,81997,81998,81999,208],{},"Next we need to install ",[30,82000,82001],{},"linux-headers",[459,82003,82006],{"className":82004,"code":82005,"language":997},[995],"sudo pacman -S linux-headers\n",[30,82007,82005],{"__ignoreMap":464},[11,82009,82010],{},"Next we need to install some VirtualBox packages:",[459,82012,82015],{"className":82013,"code":82014,"language":997},[995],"sudo pacman -S virtualbox-guest-utils virtualbox-guest-dkms\n",[30,82016,82014],{"__ignoreMap":464},[11,82018,82019,82020,82022],{},"The tutorial didn't mention the ",[30,82021,82001],{}," package, but the error message that comes up if you skip this step say that this package is needed.",[11,82024,82025,82028],{},[30,82026,82027],{},"xorg-server-utils"," seems to have been removed, but it was a meta-package that simply included the following:",[459,82030,82033],{"className":82031,"code":82032,"language":997},[995],"xorg-iceauth xorg-sessreg xorg-xcmsdb xorg-xbacklight xorg-xgamma xorg-xhost xorg-xinput xorg-xmodmap xorg-xrandr xorg-xrdb xorg-xrefresh xorg-xset xorg-xsetroot\n",[30,82034,82032],{"__ignoreMap":464},[11,82036,82037],{},"Installing these packages individually did not help.",[11,82039,82040,82041,82044],{},"This does a full install of xorg (reinstalling many packages I just installed). After running this command, running ",[30,82042,82043],{},"startx"," gives no errors and lets me enter the X environment.",[736,82046,82048],{"id":82047},"installing-gnome-desktop","Installing GNOME Desktop",[11,82050,82051,82052,82055],{},"Next, let's install ",[30,82053,82054],{},"GNOME"," desktop. Issue the following command:",[459,82057,82060],{"className":82058,"code":82059,"language":997},[995],"sudo pacman -S gnome gnome-extra gdm\n",[30,82061,82059],{"__ignoreMap":464},[11,82063,82064,82065,643],{},"Accept all of the defaults and then confirm by pressing ",[30,82066,56502],{},[736,82068,82070],{"id":82069},"installing-networking-tools","Installing Networking Tools",[459,82072,82075],{"className":82073,"code":82074,"language":997},[995],"sudo pacman -S net-tools\n",[30,82076,82074],{"__ignoreMap":464},[736,82078,82080],{"id":82079},"installing-popular-packages","Installing popular packages",[459,82082,82085],{"className":82083,"code":82084,"language":997},[995],"sudo pacman -S pulseaudio pulseaudio-alsa pavucontrol gnome-terminal firefox flashplugin vlc deluge smplayer audacious qmmp gimp xfburn gedit gnome-system-monitor\n",[30,82086,82084],{"__ignoreMap":464},[736,82088,82090],{"id":82089},"installing-codecs-for-audio-and-video","Installing codecs for audio and video",[459,82092,82095],{"className":82093,"code":82094,"language":997},[995],"sudo pacman -S a52dec faac faad2 flac jasper lame libdca libdv libmad libmpeg2 libtheora libvorbis libxv wavpack x264 xvidcore\n",[30,82096,82094],{"__ignoreMap":464},[736,82098,82100],{"id":82099},"install-yaourt","Install Yaourt",[459,82102,82105],{"className":82103,"code":82104,"language":997},[995],"sudo pacman -S yaourt\n",[30,82106,82104],{"__ignoreMap":464},[11,82108,82109],{},[15,82110,82111],{},"NEVER run yaourt with sudo or as root",[11,82113,82114],{},"Finally, enable GNOME to start automatically at boot:",[459,82116,82119],{"className":82117,"code":82118,"language":997},[995],"sudo systemctl enable gdm\n",[30,82120,82118],{"__ignoreMap":464},[11,82122,82123],{},"The last issue I had was getting GNOME Desktop to display in full-screen in VirtualBox. To do this, click on the Devices Tab and then Insert Guest Additions Image.",[459,82125,82128],{"className":82126,"code":82127,"language":997},[995],"reboot\n",[30,82129,82127],{"__ignoreMap":464},[11,82131,82132,82133,208],{},"Login with the user account that you created. At this point, trying to open Terminal will probably not work. This is because of an issue mentioned earlier about a locale file. Let's look at ",[30,82134,81520],{},[459,82136,82139],{"className":82137,"code":82138,"language":997},[995],"cat /etc/locale.conf\n",[30,82140,82138],{"__ignoreMap":464},[11,82142,82143,82144,82147],{},"We just need to make sure that the language we want is uncommented in ",[30,82145,82146],{},"/etc/locale.gen",". We did this previously. Now we need to run the following command:",[459,82149,82152],{"className":82150,"code":82151,"language":997},[995],"sudo locale-gen\n",[30,82153,82151],{"__ignoreMap":464},[11,82155,82156],{},"This should allow us to access the terminal.",[736,82158,82160],{"id":82159},"enable-wi-fi","Enable Wi-Fi",[11,82162,82163,82164,82167,82168,82171],{},"To enable Wi-Fi, first check ",[30,82165,82166],{},"lspci -k"," if you have an onboard wifi card or ",[30,82169,82170],{},"lsusb -v"," if you have a USB wifi adapter.",[11,82173,82174,82175,82177,82178,82180,82181,82184],{},"Next run ",[30,82176,81885],{}," and see if there is an entry that starts with ",[30,82179,81281],{},". This is wireless interface. If you don't have WiFi working it probably says ",[30,82182,82183],{},"state DOWN"," in the description.",[11,82186,82187],{},"We need to run the following command to bring it up:",[459,82189,82192],{"className":82190,"code":82191,"language":997},[995],"sudo ip link set \u003Cthe code that starts with w> up\n",[30,82193,82191],{"__ignoreMap":464},[11,82195,82196],{},"If we go to \"Settings\" we will see an error that says \"NetworkManager needs to be running.\"",[11,82198,82199],{},"To get NetworkManager running, issue the following commands:",[459,82201,82204],{"className":82202,"code":82203,"language":997},[995],"sudo systemctl enable NetworkManager.service\nsudo systemctl start NetworkManager.service\n",[30,82205,82203],{"__ignoreMap":464},[11,82207,82208,82209,82212],{},"This will prompt you for authentication, then you can go back to ",[30,82210,82211],{},"Settings > Network"," and you should be able to find any available Wi-Fi networks.",[736,82214,82216],{"id":82215},"customizing-gnome-desktop","Customizing GNOME Desktop",[11,82218,82219],{},"Next let's work on customizing our GUI.",[11,82221,82222,82223,82226],{},"We will now start making use of the ",[30,82224,82225],{},"yaourt"," utility to build packages for programs we want to install.",[11,82228,82229],{},"Yaourt gives us a useful flag to search packages:",[459,82231,82234],{"className":82232,"code":82233,"language":997},[995],"yaourt -Ss search_term1 search_term2\n",[30,82235,82233],{"__ignoreMap":464},[11,82237,82238],{},"This returns a list of packages which match search terms either in their title or description. When you find the right package you can copy the exact name and then download it with:",[459,82240,82243],{"className":82241,"code":82242,"language":997},[995],"yaourt -S package_name\n",[30,82244,82242],{"__ignoreMap":464},[11,82246,82247],{},"Let's start with nice custom theme, numix.",[11,82249,82250],{},"Run the following:",[459,82252,82255],{"className":82253,"code":82254,"language":997},[995],"yaourt -Ss numix\n",[30,82256,82254],{"__ignoreMap":464},[11,82258,82259],{},"We will see many packages, and there are three that we should install:",[76,82261,82262,82267,82272],{},[79,82263,82264],{},[30,82265,82266],{},"numix-gtk-theme",[79,82268,82269],{},[30,82270,82271],{},"numix-circle-theme-git",[79,82273,82274],{},[30,82275,82276],{},"numix-cursor-theme-git",[11,82278,82279,82280,82282,82283,82285],{},"The installations process for ",[30,82281,82266],{}," is quite long. You need to confirm ",[30,82284,13436],{}," and it will also ask you the following:",[459,82287,82290],{"className":82288,"code":82289,"language":997},[995],"Edit PKGBUILD with:\n",[30,82291,82289],{"__ignoreMap":464},[11,82293,82294,82295,82298,82299,82301,82302,82304,82305,82308],{},"Here you can type ",[30,82296,82297],{},"nano"," and press ",[30,82300,81201],{},". It will then show you a file in ",[30,82303,82297],{}," or the editor you select. You can press ",[30,82306,82307],{},"Ctrl-X"," and the installation script will continue running.",[11,82310,82311],{},"You will also see the following message before being prompted to select an editor:",[459,82313,82316],{"className":82314,"code":82315,"language":997},[995],"Please add $VISUAL to your environment variables.\n",[30,82317,82315],{"__ignoreMap":464},[11,82319,82320],{},"Let's do that now so we are not prompted to do it later:",[459,82322,82325],{"className":82323,"code":82324,"language":997},[995],"sudo nano ~/.bashrc\n",[30,82326,82324],{"__ignoreMap":464},[11,82328,82329],{},"Add the following line to the end of the file:",[459,82331,82334],{"className":82332,"code":82333,"language":997},[995],"export VISUAL=\"nano\"\n",[30,82335,82333],{"__ignoreMap":464},[11,82337,82338,82339,82342],{},"You could also set ",[30,82340,82341],{},"VISUAL=\"vim\""," or some other editor.",[11,82344,82345,82346,82349,82350,82353],{},"Now that we have installed the themes, let's activate them. Press ",[30,82347,82348],{},"ALT F2"," and type ",[30,82351,82352],{},"gnome-tweak-tool",". Under the appearance tab you can select the newly installed GTK+ theme, icons and cursor.",[736,82355,82357],{"id":82356},"gnome-shell-extensions","GNOME Shell Extensions",[11,82359,82360,82361,82364,82365,82369],{},"You can do a lot of neat customization with extensions. You will see an ",[30,82362,82363],{},"Extensions"," tab in the Tweak Tool. Click on \"Get more extensions\". This will take you to the ",[20,82366,82357],{"href":82367,"rel":82368},"https://extensions.gnome.org",[24]," site. Before you start using extensions, you need to add a Firefox extensions and install a package. Click the link in the blue bar at the top of the page and install the plugin.",[11,82371,82372],{},"Next you need to install the following package:",[459,82374,82377],{"className":82375,"code":82376,"language":997},[995],"yaourt -S chrome-gnome-shell-git\n",[30,82378,82376],{"__ignoreMap":464},[11,82380,82381],{},"Now refresh the GNOME Shell Extensions page and the blue message should go away. Here are the extensions I use:",[76,82383,82384,82387,82390,82393,82396,82399,82402,82405,82408],{},[79,82385,82386],{},"Applications menu",[79,82388,82389],{},"Removable Drive menu",[79,82391,82392],{},"Dash to Dock",[79,82394,82395],{},"Drop Down Terminal",[79,82397,82398],{},"Dynamic Top Bar",[79,82400,82401],{},"User Themes",[79,82403,82404],{},"System Monitor",[79,82406,82407],{},"Sensory Perception",[79,82409,82410,82411,82414],{},"Screen Saver and hibernate buttons (requires ",[30,82412,82413],{},"gnome-screensaver"," package and a GNOME Shell Extension called \"Suspend and Lock Button\")",[11,82416,82417],{},"Dash to Dock lets you place the dock wherever you want, you can also change its appearance and behavior.",[736,82419,82421],{"id":82420},"screenfetch","Screenfetch",[11,82423,82424],{},"I like to use Screenfetch, a package that prints out ASCII art for your distro and some customizable system information.",[459,82426,82429],{"className":82427,"code":82428,"language":997},[995],"yaourt -S screenfetch\n",[30,82430,82428],{"__ignoreMap":464},[11,82432,82433,82434,82436],{},"Now edit ",[30,82435,76255],{}," by adding the following line at the bottom of the file:",[459,82438,82441],{"className":82439,"code":82440,"language":997},[995],"screenfetch\n",[30,82442,82440],{"__ignoreMap":464},[11,82444,82445,82446,82449],{},"Screefetch can be further customized, check out ",[30,82447,82448],{},"man screenfetch"," for more options. For example, you can exlude certain lines about system information or add custom lines like the model of your PC, etc:",[459,82451,82454],{"className":82452,"code":82453,"language":997},[995],"                  -`\n                 .o+`                 brian@archthinkpad\n                `ooo/                 OS: Arch Linux\n               `+oooo:                Kernel: x86_64 Linux 4.12.5-1-ARCH\n              `+oooooo:               Uptime: 5h 8m\n              -+oooooo+:              Packages: 1044\n            `/:-:++oooo+:             Shell: bash 4.4.12\n           `/++++/+++++++:            Resolution: 1366x768\n          `/++++++++++++++:           DE: GNOME\n         `/+++ooooooooooooo/`         WM: GNOME Shell\n        ./ooosssso++osssssso+`        GTK Theme: Numix [GTK2/3]\n       .oossssso-````/ossssss+`       Icon Theme: Numix-Circle\n      -osssssso.      :ssssssso.      Font: Cantarell 11\n     :osssssss/        osssso+++.     CPU: Intel Core i5-3320M @ 4x 3.3GHz [44.0°C]\n    /ossssssss/        +ssssooo/-     GPU: intel\n  `/ossssso+/:-        -:/+osssso+-   RAM: 4315MiB / 7685MiB\n  `+sso+:-`                 `.-/+oso:\n `++:.                           `-/+/\n.`                                 `/\n\n\n",[30,82455,82453],{"__ignoreMap":464},[54716,82457,82459],{"id":82458},"quick-note-on-terminal-colors","Quick Note on Terminal Colors",[11,82461,82462],{},"I don't like the default white background of the terminal, so you can change this by editing the default profile. Unselect \"Use colors from system theme\" and select \"Solarized Dark\" (my preferred color scheme).",[54716,82464,82466],{"id":82465},"quick-note-on-yaourt","Quick Note on yaourt",[11,82468,82469,82470,82473],{},"You can add a ",[30,82471,82472],{},"--no-confirm"," flag to yaourt that will prevent it from asking you if you want to proceed with each step of the installation process.",[736,82475,82477],{"id":82476},"window-options","Window options",[11,82479,82480,82481,82483,82484,82487],{},"By default, GNOME only shows the ",[30,82482,56498],{}," in the top right corner of windows. You can add the maximize and minimize buttons in the ",[30,82485,82486],{},"Windows"," tab of the GNOME Tweak Tool.",[736,82489,82491],{"id":82490},"icons-on-desktop","Icons on Desktop",[11,82493,82494,82495,82497],{},"Another default setting that may be strange for you is \"Icons on Desktop\" which is set to ",[30,82496,40598],{}," by default. The is found under the Desktop tab in the GNOME Tweak Tool.",[736,82499,82501],{"id":82500},"my-other-setup-notes","My other setup notes",[11,82503,82504],{},"I install Chromium and set it to my default browser. My must have extension for Chrome (and any other browser) is Last Pass for account and password management.",[56,82506,82508],{"id":82507},"default-applications","Default Applications",[11,82510,82511,82512,82515],{},"You can set default applications by opening ",[30,82513,82514],{},"Settings > Details > Default Applications",". I use Image Viewer for photos instead of GIMP and VLC for Video.",[11,82517,82518,82521,82522,82525],{},[30,82519,82520],{},"pacman"," offers similar functionality, and so does ",[30,82523,82524],{},"pacaur",". There are several options for how to download software with Arch Linux.",[736,82527,82529],{"id":82528},"other-packages","Other packages",[11,82531,82532],{},"Here's a list of other helpful packages:",[76,82534,82535],{},[79,82536,82537],{},"htop (system monitoring)",[11,82539,82540],{},"htop can be customized to show some helpful information. Press F2 or click in the terminal where it says \"Setup\" and you will open a menu to select custom options for htop.",[76,82542,82543,82546],{},[79,82544,82545],{},"blender (video editing and 3d modeling)",[79,82547,82548],{},"OBS (screen recording and streaming software)",[11,82550,82551,82552,82555],{},"OBS took a little bit of configuring to get right. Out of the box, doing a screen capture showed just a black screen. I think the solution is to install ",[30,82553,82554],{},"obs-studio-git"," and then reboot your computer.",[76,82557,82558],{},[79,82559,82560],{},"Dropbox",[459,82562,82565],{"className":82563,"code":82564,"language":997},[995],"yaourt -S dropbox\n",[30,82566,82564],{"__ignoreMap":464},[11,82568,82569],{},"Run the following command:",[459,82571,82574],{"className":82572,"code":82573,"language":997},[995],"dropbox\n",[30,82575,82573],{"__ignoreMap":464},[11,82577,82578],{},"And then follow the dialogue to sign into your Dropbox account. You will see a Dropbox folder in the File Manager.",[76,82580,82581],{},[79,82582,82583],{},"Google Drive",[11,82585,82586],{},"You can get access to Google Drive by going into:",[459,82588,82591],{"className":82589,"code":82590,"language":997},[995],"Settings > Online Accounts > Google\n",[30,82592,82590],{"__ignoreMap":464},[11,82594,82595,82596,82599,82600,82602],{},"If you have ",[30,82597,82598],{},"Files"," set to ",[30,82601,40595],{},", you will find your Google Drive Folder in the side bar of the File Viewer. Clicking on the icon mounts your Google Drive and gives you access to the files.",[76,82604,82605],{},[79,82606,82607],{},"hexchat (IRC client)",[11,82609,82610],{},"Internet Relay Chat (IRC) is an easy way to start talking to people in real-time about any topic you can think of. Hexchat is a popular GUI client for IRC. Here is how to install hexchat:",[459,82612,82615],{"className":82613,"code":82614,"language":997},[995],"sudo pacman -S hexchat\n",[30,82616,82614],{"__ignoreMap":464},[11,82618,82619,82620,82625],{},"Once you have installed hexchat, you will want to register a nickname. ",[20,82621,82624],{"href":82622,"rel":82623},"https://freenode.net/kb/answer/registration",[24],"This article"," says that the steps for registering a nickname are to:",[11,82627,82628,82629,82632,82633,82636,82637,82640],{},"Send the following message to ",[30,82630,82631],{},"freenode"," (replace ",[30,82634,82635],{},"some_password"," with a secure password of your choice and ",[30,82638,82639],{},"youremail@example.com"," with your email address):",[459,82642,82645],{"className":82643,"code":82644,"language":997},[995],"/msg NickServ REGISTER some_password youremail@example.com\n",[30,82646,82644],{"__ignoreMap":464},[11,82648,82649],{},"After you send this message, check your email and you will see instructions to confirm your email that look like this:",[459,82651,82654],{"className":82652,"code":82653,"language":997},[995],"/msg NickServ VERIFY REGISTER \u003Cyour_name> \u003Csome_code_they_give_you>\n",[30,82655,82653],{"__ignoreMap":464},[11,82657,82658,82659,82662,82663,82666],{},"This should confirm your nickname. Now you can start chatting on channels. Press ",[30,82660,82661],{},"Alt + S",", and then ",[30,82664,82665],{},"J",", and then type the name of the channel you want to join.",[736,82668,82670],{"id":82669},"configure-vpn","Configure VPN",[11,82672,82673],{},"I setup TorGuard VPN service by installing:",[459,82675,82678],{"className":82676,"code":82677,"language":997},[995],"yaourt -S torguard\n",[30,82679,82677],{"__ignoreMap":464},[56,82681,82683],{"id":82682},"setting-up-development-tools","Setting Up Development Tools",[736,82685,82687],{"id":82686},"atom-and-other-text-editors","Atom and other text editors",[11,82689,82690],{},"My preferred text editor is Atom:",[459,82692,82695],{"className":82693,"code":82694,"language":997},[995],"sudo pacman -S atom\n",[30,82696,82694],{"__ignoreMap":464},[11,82698,82699,82700,82705,82706,82709],{},"Spell checker didn't seem to be working for me, and this seems to be a ",[20,82701,82704],{"href":82702,"rel":82703},"https://github.com/atom/spell-check/issues/129",[24],"known issue",", so I recommend you install the ",[30,82707,82708],{},"atom/spell-checker"," package if you like to use atom and want spell checking.",[11,82711,82712,82713,82716],{},"The Arch Linux installation also came with an interesting editor called ",[30,82714,82715],{},"Builder"," which seems pretty interesting as well.",[736,82718,81059],{"id":82719},"virtualbox-1",[11,82721,82722],{},[51,82723,82724],{},"Skip this step if you are installing Arch Linux as guest OS in a VirtualBox (you can't run a virtual machine in a virtual machine).",[459,82726,82729],{"className":82727,"code":82728,"language":997},[995],"sudo pacman -S virtualbox\n",[30,82730,82728],{"__ignoreMap":464},[11,82732,82733],{},"You will see a message that says:",[459,82735,82738],{"className":82736,"code":82737,"language":997},[995],":: There are 2 providers available for VIRTUALBOX-HOST-MODULES\n:: Repository community\n   1) virtualbox-host-dkms 2) virtualbox-host-modules-arch\n",[30,82739,82737],{"__ignoreMap":464},[11,82741,82742],{},[15,82743,82744],{},"Select option 2, continue the installation process and then reboot.",[736,82746,82748],{"id":82747},"quick-note-on-deluge","Quick note on deluge",[11,82750,82751],{},"You will probably want to use a BitTorrent client to download ISO images for virtual machines you want to run. Deluge works well for this, and we already downloaded it during setup. We can add one more package for a theme that Deluge uses:",[459,82753,82756],{"className":82754,"code":82755,"language":997},[995],"yaourt -S gtk-engine-murrine\n",[30,82757,82755],{"__ignoreMap":464},[11,82759,82760],{},"Now grab a torrent link or file for an OS you want to download and add it to Deluge.",[11,82762,82763],{},"When it is finished we can launch VirtualBox and create a virtual machine with our ISO.",[736,82765,82767],{"id":82766},"npm-and-nodejs","npm and node.js",[11,82769,82770,82771,82776],{},"Here's ",[20,82772,82775],{"href":82773,"rel":82774},"https://docs.npmjs.com/getting-started/fixing-npm-permissions",[24],"a helpful article"," that is important to following when setting up npm and node.js. Be very careful here as some of the steps can completely mess up permissions on your entire system.",[11,82778,82779,82780,82783],{},"Here's a walkthrough of the steps to get ",[30,82781,82782],{},"npm"," setup correctly.",[459,82785,82788],{"className":82786,"code":82787,"language":997},[995],"sudo pacman -S npm\n",[30,82789,82787],{"__ignoreMap":464},[11,82791,82792],{},"Next run:",[459,82794,82797],{"className":82795,"code":82796,"language":997},[995],"npm config get prefix\n",[30,82798,82796],{"__ignoreMap":464},[11,82800,82801],{},"The output of this command is probably:",[459,82803,82806],{"className":82804,"code":82805,"language":997},[995],"/usr\n",[30,82807,82805],{"__ignoreMap":464},[11,82809,82810,82811,208],{},"If this is the case, we MUST follow the following steps (option 2 from ",[20,82812,49337],{"href":82773,"rel":82813},[24],[459,82815,82818],{"className":82816,"code":82817,"language":997},[995],"mkdir ~/.npm-global\nnpm config set prefix '~/.npm-global'\n",[30,82819,82817],{"__ignoreMap":464},[11,82821,82822,82823,82826],{},"If you don't already have a ",[30,82824,82825],{},"~/.profile"," file, make one, and then add the following line:",[459,82828,82831],{"className":82829,"code":82830,"language":997},[995],"export PATH=~/.npm-global/bin:$PATH\n",[30,82832,82830],{"__ignoreMap":464},[11,82834,82835],{},"Then run:",[459,82837,82840],{"className":82838,"code":82839,"language":997},[995],"source ~/.profile\n",[30,82841,82839],{"__ignoreMap":464},[11,82843,82844,82845,82847,82848,82850],{},"This will \"refresh\" ",[30,82846,82825],{}," so that using the ",[30,82849,82782],{}," command will work without needing sudo.",[11,82852,82853,82854,82856],{},"Next we can test that ",[30,82855,82782],{}," works by trying to install a package:",[459,82858,82861],{"className":82859,"code":82860,"language":997},[995],"npm install -g jshint\n",[30,82862,82860],{"__ignoreMap":464},[11,82864,82865],{},"This should install the package with no errors.",[736,82867,82869],{"id":82868},"heroku","Heroku",[459,82871,82874],{"className":82872,"code":82873,"language":997},[995],"yaourt -S heroku-cli\n",[30,82875,82873],{"__ignoreMap":464},[11,82877,82835],{},[459,82879,82882],{"className":82880,"code":82881,"language":997},[995],"heroku keys:add\n",[30,82883,82881],{"__ignoreMap":464},[11,82885,82886,82887,82890],{},"and then type ",[30,82888,82889],{},"yes"," at the prompt. This will your computer's public key to your Heroku account.",[54716,82892,82894],{"id":82893},"note-about-python-versions-on-heroku","Note about Python versions on Heroku",[11,82896,82897],{},"I have only been able to deploy with Python 3.5.2.",[736,82899,27501],{"id":27500},[459,82901,82904],{"className":82902,"code":82903,"language":997},[995],"sudo pacman -S redis\n",[30,82905,82903],{"__ignoreMap":464},[11,82907,82908,82909],{},"Next, start and enable ",[30,82910,82911],{},"redis.service",[459,82913,82916],{"className":82914,"code":82915,"language":997},[995],"sudo systemctl start redis.service\nsudo systemctl enable redis.service\n",[30,82917,82915],{"__ignoreMap":464},[736,82919,82921],{"id":82920},"ssh","SSH",[459,82923,82926],{"className":82924,"code":82925,"language":997},[995],"sudo pacman -S openssh\n",[30,82927,82925],{"__ignoreMap":464},[736,82929,82931,82932,748],{"id":82930},"autoenv-github","autoenv (",[20,82933,82936],{"href":82934,"rel":82935},"https://github.com/kennethreitz/autoenv",[24],"GitHub",[459,82938,82941],{"className":82939,"code":82940,"language":997},[995],"yaourt -S autoenv\n",[30,82942,82940],{"__ignoreMap":464},[11,82944,82945],{},"You need to source activate.sh in your bashrc afterwards:",[459,82947,82950],{"className":82948,"code":82949,"language":997},[995],"echo 'source /usr/share/autoenv/activate.sh' >> ~/.bashrc\n",[30,82951,82949],{"__ignoreMap":464},[736,82953,82955],{"id":82954},"ruby-and-jekyll","Ruby and Jekyll",[11,82957,82958,82959,643],{},"I use Jekyll for my personal site and blog, which is a static site generator written in Ruby. To start using Jekyll we need to install a ruby gem, and to install the gem we will need to first install ruby. Here is the ",[20,82960,82963],{"href":82961,"rel":82962},"https://wiki.archlinux.org/index.php/ruby",[24],"ruby page from the Arch Wiki",[459,82965,82968],{"className":82966,"code":82967,"language":997},[995],"sudo pacman -S ruby\n",[30,82969,82967],{"__ignoreMap":464},[11,82971,82972],{},"Next we can install the Jekyll gem:",[459,82974,82977],{"className":82975,"code":82976,"language":997},[995],"gem install jekyll bundler\n",[30,82978,82976],{"__ignoreMap":464},[11,82980,82981,82982,643],{},"It should say something like \"20 gems installed\" if everything was successful. You can read more about Jekyll ",[20,82983,13074],{"href":82984,"rel":82985},"https://jekyllrb.com/",[24],[11,82987,82988],{},"For now I will not be installing RVM (Ruby Version Manager), but the Arch Wiki has good notes on how this can be installed.",[11,82990,82991,82992,37166],{},"Before we use the Jekyll gem, we need to add the following to our ",[30,82993,76255],{},[459,82995,82998],{"className":82996,"code":82997,"language":997},[995],"PATH=\"$(ruby -e 'print Gem.user_dir')/bin:$PATH\"\n",[30,82999,82997],{"__ignoreMap":464},[11,83001,83002],{},"This will allow you to run Jekyll commands.",[736,83004,11356],{"id":12886},[11,83006,83007,83008,208],{},"In the installation pIssuesrocess, the latest versions of Python has been installed (3.6.2 at the time of writing this tutorial). We need to get ",[30,83009,83010],{},"pip",[459,83012,83015],{"className":83013,"code":83014,"language":997},[995],"yaourt -S python-pip\n",[30,83016,83014],{"__ignoreMap":464},[11,83018,82770,83019,83024],{},[20,83020,83023],{"href":83021,"rel":83022},"https://docs.python.org/3/library/venv.html",[24],"an article from the Python documentation"," that covers creating virtual environments.",[11,83026,81806,83027,83032],{},[20,83028,83031],{"href":83029,"rel":83030},"https://packaging.python.org/tutorials/installing-packages/",[24],"another article"," about creating virtual environments in python.",[210,83034,83035],{},[11,83036,83037],{},"Currently, there are two viable tools for creating Python virtual environments:\nvenv is available by default in Python 3.3 and later, and installs pip and setuptools into created virtual environments in Python 3.4 and later.\nvirtualenv needs to be installed separately, but supports Python 2.6+ and Python 3.3+, and pip, setuptools and wheel are always installed into created virtual environments by default (regardless of Python version).\nThe basic usage is like so:\nUsing virtualenv:",[459,83039,83042],{"className":83040,"code":83041,"language":997},[995],"virtualenv \u003CDIR>\nsource \u003CDIR>/bin/activate\n",[30,83043,83041],{"__ignoreMap":464},[210,83045,83046],{},[11,83047,83048],{},"Using venv:",[459,83050,83053],{"className":83051,"code":83052,"language":997},[995],"python3 -m venv \u003CDIR>\nsource \u003CDIR>/bin/activate\n",[30,83054,83052],{"__ignoreMap":464},[11,83056,83057],{},"For more information, see the virtualenv docs or the venv docs.",[54716,83059,83061],{"id":83060},"jupyter-notebook-and-ipython-notebook","Jupyter Notebook and IPython Notebook",[11,83063,83064],{},"IPython notebooks (also known as Jupyter notebooks), provide a nice environment for combining python runtime environments, Markdown, images, graphs and other interactive Python tools such as Bokeh. You can install both Jupyter Notebooks and IPython Notebooks:",[459,83066,83069],{"className":83067,"code":83068,"language":997},[995],"yaort -S jupyter-notebook\n",[30,83070,83068],{"__ignoreMap":464},[459,83072,83075],{"className":83073,"code":83074,"language":997},[995],"yaourt -S ipython2-notebook\n",[30,83076,83074],{"__ignoreMap":464},[11,83078,83079,83080,30583,83083,83086],{},"You can launch either of these with ",[30,83081,83082],{},"jupyter notebook",[30,83084,83085],{},"ipython notebook"," from the terminal.",[11,83088,83089],{},[51,83090,83091],{},"There seems to be an issue with LaTeX",[54716,83093,83095],{"id":83094},"anaconda-distribution-from-continuum-analytics","Anaconda Distribution from Continuum Analytics",[11,83097,83098,83099,10552],{},"If you are doing scientific computing, statistical analysis or an type of machine learning, you will want to download the Anaconda distribution. This is Python bundled with a bunch of great packages and C libraries for doing heavy lifting. Anaconda uses a slightly different tool for managing virtual environments and package management, but it is very easy to do both using the ",[30,83100,70052],{},[736,83102,83104],{"id":83103},"postgresql","PostgreSQL",[459,83106,83109],{"className":83107,"code":83108,"language":997},[995],"sudo pacman -S postgresql\n",[30,83110,83108],{"__ignoreMap":464},[459,83112,83115],{"className":83113,"code":83114,"language":997},[995],"sudo -u postgres -i\n",[30,83116,83114],{"__ignoreMap":464},[11,83118,83119],{},"This switches you to the PostgreSQL user.",[11,83121,83122],{},"Before PostgreSQL can function correctly, the database cluster must be initialized:",[459,83124,83127],{"className":83125,"code":83126,"language":997},[995],"initdb --locale $LANG -E UTF8 -D '/var/lib/postgres/data'\n",[30,83128,83126],{"__ignoreMap":464},[11,83130,83131],{},"You should see the following:",[459,83133,83136],{"className":83134,"code":83135,"language":997},[995],"[postgres@archthinkpad ~]$ initdb --locale $LANG -E UTF8 -D '/var/lib/postgres/data'\nThe files belonging to this database system will be owned by user \"postgres\".\nThis user must also own the server process.\n\nThe database cluster will be initialized with locale \"en_US.UTF-8\".\nThe default text search configuration will be set to \"english\".\n\nData page checksums are disabled.\n\nfixing permissions on existing directory /var/lib/postgres/data ... ok\ncreating subdirectories ... ok\nselecting default max_connections ... 100\nselecting default shared_buffers ... 128MB\nselecting dynamic shared memory implementation ... posix\ncreating configuration files ... ok\nrunning bootstrap script ... ok\nperforming post-bootstrap initialization ... ok\nsyncing data to disk ... ok\n\nWARNING: enabling \"trust\" authentication for local connections\nYou can change this by editing pg_hba.conf or using the option -A, or\n--auth-local and --auth-host, the next time you run initdb.\n\nSuccess. You can now start the database server using:\n\n    pg_ctl -D /var/lib/postgres/data -l logfile start\n",[30,83137,83135],{"__ignoreMap":464},[11,83139,83140,83141,83144],{},"Now type ",[30,83142,83143],{},"exit"," to return to your regular user.",[11,83146,82908,83147,83150],{},[30,83148,83149],{},"postgresql.service"," as root:",[459,83152,83155],{"className":83153,"code":83154,"language":997},[995],"[brian@archthinkpad ~]$ sudo systemctl start postgresql.service\n[sudo] password for brian:\n[brian@archthinkpad ~]$ sudo systemctl enable postgresql.service\nCreated symlink /etc/systemd/system/multi-user.target.wants/postgresql.service → /usr/lib/systemd/system/postgresql.service.\n[brian@archthinkpad ~]$\n",[30,83156,83154],{"__ignoreMap":464},[11,83158,83159,83160,83165],{},"See the ",[20,83161,83164],{"href":83162,"rel":83163},"https://wiki.archlinux.org/index.php/PostgreSQL",[24],"Arch Wiki article on PostgreSQL"," for more information.",[736,83167,83168],{"id":83168},"tmux",[459,83170,83173],{"className":83171,"code":83172,"language":997},[995],"sudo pacman -S tmux\n",[30,83174,83172],{"__ignoreMap":464},[736,83176,53860],{"id":53860},[459,83178,83181],{"className":83179,"code":83180,"language":997},[995],"sudo pacman -S tree\n",[30,83182,83180],{"__ignoreMap":464},[736,83184,83186],{"id":83185},"languages-and-input-sources","Languages and Input Sources",[11,83188,83189],{},"Here is how to add support for Chinese characters and also Chinese pinyin input:",[459,83191,83194],{"className":83192,"code":83193,"language":997},[995],"yaourt -S adobe-source-han-sans-cn-fonts\n",[30,83195,83193],{"__ignoreMap":464},[11,83197,83198],{},"This will make Chinese characters visible. In order to type Chinese using the pinyin input method, run the following commands:",[459,83200,83203],{"className":83201,"code":83202,"language":997},[995],"sudo pacman -S ibus\nsudo pacman -S ibus-libpinyin\n",[30,83204,83202],{"__ignoreMap":464},[11,83206,83207,83208,83211],{},"可以打汉字了！其它语言和文字 packages 可以参考",[20,83209,83210],{"href":464},"这个 Arch Wiki 文章","。",[11,83213,83214],{},"Next you can go into:",[459,83216,83219],{"className":83217,"code":83218,"language":997},[995],"Settings > Region & Language > Input Source > + > Other\n",[30,83220,83218],{"__ignoreMap":464},[11,83222,83223],{},"Select Chinese (Intelligent Pin Yin) from the list and you should see a language menu in the top bar. At this point I was able to see both English and Chinese in Language menu in the top bar, but the Chinese pinyin only worked after rebooting.",[736,83225,83227],{"id":83226},"spotify","Spotify",[11,83229,83230],{},"Get some tunes going with the Spotfy app. It is not officially supported, but it seems to work alright. Facebook login did not work for me. You can create a new account with your existing email like this:",[459,83232,83235],{"className":83233,"code":83234,"language":997},[995],"youremail+spotify@email.com\n",[30,83236,83234],{"__ignoreMap":464},[11,83238,83239],{},"Using my main email to sign up for a new account didn't work because that email was linked to the Facebook login.",[459,83241,83244],{"className":83242,"code":83243,"language":997},[995],"yaourt -S spotify\n",[30,83245,83243],{"__ignoreMap":464},[736,83247,83249],{"id":83248},"word-processing-libre-office","Word Processing (Libre Office)",[11,83251,83252],{},"Libre Office is an open-source Office Suite similar in functionality to Microsoft Word.",[459,83254,83257],{"className":83255,"code":83256,"language":997},[995],"sudo pacman -S libreoffice-fresh\n",[30,83258,83256],{"__ignoreMap":464},[11,83260,83261],{},"You can also install any language packs you may need for Libre Office:",[459,83263,83266],{"className":83264,"code":83265,"language":997},[995],"yaourt -S libreoffice-fresh-zh-CN\n",[30,83267,83265],{"__ignoreMap":464},[736,83269,83271],{"id":83270},"disk-usage-visualizations","Disk Usage Visualizations",[11,83273,83274,83275,83278,83279,187,83282,83285],{},"You can get disk usage with the ",[30,83276,83277],{},"du"," command. There are other programs for disk space visualizations, so far I have found ",[30,83280,83281],{},"gdmap",[30,83283,83284],{},"filelight"," to be helpful. Both are available in the AUR:",[459,83287,83290],{"className":83288,"code":83289,"language":997},[995],"yaourt -S gdmap filelight\n",[30,83291,83289],{"__ignoreMap":464},[736,83293,83295],{"id":83294},"sign-up-for-the-arch-wiki","Sign up for the Arch Wiki",[11,83297,83298],{},"The best way to ask for help is to ask the community. You should sign up for the Arch Wiki and post any questions you have after doing research and trying a few different solutions. The more detailed you are in your post, the better help you will get.",[56,83300,83302],{"id":83301},"miscelaneous-items","Miscelaneous Items",[11,83304,83305],{},"I have experienced unexpected behavior with some programs. Here's a list of some major issues and workarounds I have found:",[76,83307,83308],{},[79,83309,83310,83313,83314,83319],{},[15,83311,83312],{},"Google Hangouts",": I use Google Hangouts a lot for screen sharing and video conferencing. Trying to share my screen only showed a black screen with a cursor, although sharing invidual windows seemed to work fine (similar to what was happening with OBS). I found a ",[20,83315,83318],{"href":83316,"rel":83317},"https://superuser.com/questions/1166765/google-hangouts-screen-share-black-screen-error",[24],"Superuser thread"," describing the same issue. The solution is simply to select an Xorg session on login (instead of GNOME Classic or regular GNOME session). This is something I still don't know much about, but the solution worked for me and I have been able to share my entire screen in Google Hangouts.",[11,83321,83322],{},"You can check which session type you are using with the following command:",[459,83324,83327],{"className":83325,"code":83326,"language":997},[995],"echo $XDG_SESSION_TYPE\n",[30,83328,83326],{"__ignoreMap":464},[11,83330,83331,83332,30583,83335,643],{},"and you should either see ",[30,83333,83334],{},"x11",[30,83336,83337],{},"wayland",[736,83339,83341],{"id":83340},"keyboard-shortcuts","Keyboard shortcuts",[11,83343,83344],{},"You can configure keyboard shortcuts by going into:",[459,83346,83349],{"className":83347,"code":83348,"language":997},[995],"Settings > Keyboard\n",[30,83350,83348],{"__ignoreMap":464},[11,83352,83353,83354,83357],{},"Search for or find the setting that says \"Hide all normal windows\". I set mine to ",[30,83355,83356],{},"Shift + Alt + D",". You can configure other keyboard shortcuts here.",[736,83359,83361],{"id":83360},"night-light","Night Light",[11,83363,83364],{},"If you are coming from Mac or Windows, you might be used to using f.lux to dim the blue colors from your screen. There are few options for Linux such as Red Shift and xflux. I finf that in GNOME, using the built in Nigh Light works the best for reduce eye strain. Just go into:",[459,83366,83369],{"className":83367,"code":83368,"language":997},[995],"Settings > Displays\n",[30,83370,83368],{"__ignoreMap":464},[11,83372,83373],{},"and click on Night Light. Like flux, you can set it for regular hours. There aren't as many modes or options, but it does a pretty goog job of what I need it to do.",[736,83375,83377],{"id":83376},"helpful-commands","Helpful Commands",[11,83379,83380],{},"To view all available commands, run the following command:",[459,83382,83385],{"className":83383,"code":83384,"language":997},[995],"compgen -c\n",[30,83386,83384],{"__ignoreMap":464},[11,83388,83389],{},"To view all installed packages, run:",[459,83391,83394],{"className":83392,"code":83393,"language":997},[995],"pacman -Q\n",[30,83395,83393],{"__ignoreMap":464},[56,83397,83399],{"id":83398},"conclusion-and-next-steps","Conclusion and Next Steps",[11,83401,83402],{},"Installing Arch Linux is like trying to build a house with just a spoon. You won't get very far with a spoon, but you have the benefit of being able to order unlimited copies of free parts and materials online (the AUR, or Arch User Repository), and a great community of people that are extremely knowledgable about \"building houses\". Pretty soon you are able to quickly put together a foundation, scaffolding, wiring, appliances and yes even wallpaper. Arch Linux stays with its rolling release cycle, so you don't have to rebuild your house every 9 months to stay current.",[11,83404,83405],{},"Arch Linux is a lot of work to set up compared to popular Linux distributions like Ubuntu. Just like everyone says, you learn a lot by going through the process, breaking things and starting the installations process from scratch. I totally messed up my permissions while installing and quickly found that the only solutions was to reinstall Arch. This guide is primarily for personal use, and I am sure there are things that can be improved and even done completely differently. Here's a list of things I can start with:",[76,83407,83408,83424,83430,83436,83442,83448,83454,83460,83466,83472,83480,83483,83486],{},[79,83409,83410,6208,83413,83420,83421,13576],{},[15,83411,83412],{},"Fix OBS Studio",[83414,83415,83416,83417],"del",{},"I got this to work in my earlier install and can't seem to remember how I fixed it. Currently OBS displays a black screen when set to ",[30,83418,83419],{},"Screen Capture",". I finally got OBS studio to work and I think the issue was as simple as rebooting (or possibly an update with ",[30,83422,83423],{},"sudo pacman -Syu",[79,83425,83426,83429],{},[15,83427,83428],{},"Encrypting the home folder",": this is good practice and will make my computer more secure, it shouldnt be too difficult either.",[79,83431,83432,83435],{},[15,83433,83434],{},"Adding a boot partition and fixing GRUB",": the guides I worked off of didn't include this, and I think it would be very important to figure out how this works if I want to include Windows 10 for doing Windows-specific tasks on my laptop as well without having to go into the bios each time I want to switch OSs.",[79,83437,83438,83441],{},[15,83439,83440],{},"Using HDDs",": On my desktop I would like to be able to figure out how to mount my HDD that I use for mass file storage on my Windows machine. For simplicity I have kept everything on one SSD, but it would be good to figure out how to easily add additional disks at boot time.",[79,83443,83444,83447],{},[15,83445,83446],{},"NVIDIA drivers",": this does not apply to my laptop, but I would like to figure out how I can get the best drivers in Arch Linux for my NVIDIA GPU on my desktop machine. This is one thing that running Linux in a Virtual Machine really doesn't allow you to do (pass a GPU through a VM), as far as I know.",[79,83449,83450,83453],{},[15,83451,83452],{},"All other drivers",": I still have lots of questions about how to make sure that I am running things properly on my desktop PC. I feel like drivers for the laptop install were pretty automatic, but this may not be the case with additional hardware components on my desktop, such a closed-loop water cooler.",[79,83455,83456,83459],{},[15,83457,83458],{},"Additional Customization, Themes, Window Managers, etc.",": There is so much that can be done with Arch Linux in terms of GUI. I love the setup I have but it would be interesting to explore some additional options like i3, for example.",[79,83461,83462,83465],{},[15,83463,83464],{},"Cusomt kernels",": I am also interested in learning how I could swap out kernels. I have seen people add LTS kernels to Arch Linux for various reasons, and I'm interested to learn how this works.",[79,83467,83468,83471],{},[15,83469,83470],{},"Maximzing battery life",": During the install process I saw the topic discussed but didn't look into it. It would be nice to see if I could make changes to get more out of the battery in my refurbished ThinkPad.",[79,83473,83474,6208,83477],{},[15,83475,83476],{},"Other areas for improvement",[151,83478,83479],{},"add here",[79,83481,83482],{},"Install RVM",[79,83484,83485],{},"Install EMACS",[79,83487],{},{"title":464,"searchDepth":488,"depth":488,"links":83489},[83490,83534,83538,83559,83564],{"id":81019,"depth":488,"text":81020,"children":83491},[83492,83494,83495,83496,83497,83498,83500,83501,83502,83503,83504,83505,83506,83507,83508,83509,83510,83511,83512,83513,83514,83515,83516,83517,83518,83519,83520,83521,83522,83523,83524,83525,83526,83527,83528,83529,83530,83531,83532,83533],{"id":81123,"depth":500,"text":83493},"Using fdisk for partitioning",{"id":81184,"depth":500,"text":81185},{"id":81213,"depth":500,"text":81214},{"id":81246,"depth":500,"text":81247},{"id":81294,"depth":500,"text":81295},{"id":81313,"depth":500,"text":83499},"Verify our layout with lsblk",{"id":81325,"depth":500,"text":81326},{"id":81350,"depth":500,"text":81351},{"id":81363,"depth":500,"text":81364},{"id":81405,"depth":500,"text":81406},{"id":81441,"depth":500,"text":81442},{"id":81469,"depth":500,"text":81470},{"id":81494,"depth":500,"text":81495},{"id":81563,"depth":500,"text":81564},{"id":81573,"depth":500,"text":81574},{"id":81590,"depth":500,"text":81591},{"id":81614,"depth":500,"text":81615},{"id":81635,"depth":500,"text":81636},{"id":81648,"depth":500,"text":81649},{"id":81661,"depth":500,"text":81662},{"id":81694,"depth":500,"text":81695},{"id":81704,"depth":500,"text":81705},{"id":81779,"depth":500,"text":81780},{"id":81829,"depth":500,"text":81830},{"id":81842,"depth":500,"text":81843},{"id":81878,"depth":500,"text":81879},{"id":81939,"depth":500,"text":81940},{"id":81949,"depth":500,"text":81950},{"id":82047,"depth":500,"text":82048},{"id":82069,"depth":500,"text":82070},{"id":82079,"depth":500,"text":82080},{"id":82089,"depth":500,"text":82090},{"id":82099,"depth":500,"text":82100},{"id":82159,"depth":500,"text":82160},{"id":82215,"depth":500,"text":82216},{"id":82356,"depth":500,"text":82357},{"id":82420,"depth":500,"text":82421},{"id":82476,"depth":500,"text":82477},{"id":82490,"depth":500,"text":82491},{"id":82500,"depth":500,"text":82501},{"id":82507,"depth":488,"text":82508,"children":83535},[83536,83537],{"id":82528,"depth":500,"text":82529},{"id":82669,"depth":500,"text":82670},{"id":82682,"depth":488,"text":82683,"children":83539},[83540,83541,83542,83543,83544,83545,83546,83547,83549,83550,83551,83552,83553,83554,83555,83556,83557,83558],{"id":82686,"depth":500,"text":82687},{"id":82719,"depth":500,"text":81059},{"id":82747,"depth":500,"text":82748},{"id":82766,"depth":500,"text":82767},{"id":82868,"depth":500,"text":82869},{"id":27500,"depth":500,"text":27501},{"id":82920,"depth":500,"text":82921},{"id":82930,"depth":500,"text":83548},"autoenv (GitHub)",{"id":82954,"depth":500,"text":82955},{"id":12886,"depth":500,"text":11356},{"id":83103,"depth":500,"text":83104},{"id":83168,"depth":500,"text":83168},{"id":53860,"depth":500,"text":53860},{"id":83185,"depth":500,"text":83186},{"id":83226,"depth":500,"text":83227},{"id":83248,"depth":500,"text":83249},{"id":83270,"depth":500,"text":83271},{"id":83294,"depth":500,"text":83295},{"id":83301,"depth":488,"text":83302,"children":83560},[83561,83562,83563],{"id":83340,"depth":500,"text":83341},{"id":83360,"depth":500,"text":83361},{"id":83376,"depth":500,"text":83377},{"id":83398,"depth":488,"text":83399},"2017-08-03","A comprehensive guide to installing Arch Linux","/static/aur/arch.png",{"layout":48045},"/2017/08/03/arch-linux-installation-guide",{"title":81005,"description":83566},"2017/08/03/arch-linux-installation-guide",[72181],"l7wFYBuinN5MITDJN77MHPUHs20PAli0KzIvWyGhogE",{"id":83575,"title":83576,"body":83577,"comments":609,"date":83608,"description":83609,"draft":602,"extension":605,"external":606,"image":83592,"meta":83610,"navigation":609,"path":83611,"seo":83612,"stem":83613,"tags":83614,"__hash__":83616},"blog/2017/05/09/my-first-attempt-at-photogrammetry.md","Building a 3D model from 60 photographs with VisualSFM and Meshlab",{"type":8,"value":83578,"toc":83606},[83579,83588,83593,83596],[11,83580,83581,83582,83587],{},"I found an interesting ",[20,83583,83586],{"href":83584,"rel":83585},"http://wedidstuff.heavyimage.com/index.php/2013/07/12/open-source-photogrammetry-workflow/",[24],"photogrammetry tutorial"," and decided to take a shot at building a 3D model of a sun hat. Here's the result:",[11,83589,83590],{},[2718,83591],{"alt":20386,"src":83592},"/static/sunhat.png",[11,83594,83595],{},"And here is an interactive model that you can view in 3D:",[23950,83597,83601],{"margin":83598,"className":83599},"auto",[83600],"sketchfab-embed-wrapper",[23881,83602],{"width":83603,"height":83604,"src":83605,"frameBorder":9181,"allowvr":464,"allowFullScreen":609,"mozallowfullscreen":19726,"webkitallowfullscreen":19726},640,480,"https://sketchfab.com/models/ff952a9d9cae4d26a178ad74e099e96b/embed",{"title":464,"searchDepth":488,"depth":488,"links":83607},[],"2017-05-09","My first attempt at photogrammetry",{"layout":48045},"/2017/05/09/my-first-attempt-at-photogrammetry",{"title":83576,"description":83609},"2017/05/09/my-first-attempt-at-photogrammetry",[83615],"photogrammetry","QfEWaUo2T0pIAclGYgcFXQUifDRTCubKvI25kYYVfUc",{"id":83618,"title":83619,"body":83620,"comments":609,"date":83608,"description":83648,"draft":602,"extension":605,"external":606,"image":83635,"meta":83649,"navigation":609,"path":83650,"seo":83651,"stem":83652,"tags":83653,"__hash__":83657},"blog/2017/05/09/rendering-sketchup-models-with-kerkythea.md","Rendering SketchUp models with Kerkythea",{"type":8,"value":83621,"toc":83646},[83622,83631,83636,83641],[11,83623,83624,83625,83630],{},"These are some results of an architectural model I rendered with Kerkythea. ",[20,83626,83629],{"href":83627,"rel":83628},"http://www.kerkythea.net/cms/",[24],"Kerkythea"," is an open source rendering program that has a SketchUp plugin. I worked with an architect who made the original plans by hand.",[11,83632,83633],{},[2718,83634],{"alt":20386,"src":83635},"/static/sketchup/sketchup_1.jpg",[11,83637,83638],{},[2718,83639],{"alt":20386,"src":83640},"/static/sketchup/sketchup_2.jpg",[11,83642,83643],{},[2718,83644],{"alt":20386,"src":83645},"/static/sketchup/sketchup_3.jpg",{"title":464,"searchDepth":488,"depth":488,"links":83647},[],"These are some results of an architectural model I rendered with Kerkythea. Kerkythea is an open source rendering program that has a SketchUp plugin. I worked with an architect who made the original plans by hand.",{"layout":48045},"/2017/05/09/rendering-sketchup-models-with-kerkythea",{"title":83619,"description":83648},"2017/05/09/rendering-sketchup-models-with-kerkythea",[83654,83655,83656],"sketchup","kerkythea","3d-modeling","cInP-o-_HwdNry_m_XQD9OhxPGZ-IX6Z52dQFsSKImQ",{"id":83659,"title":83660,"body":83661,"comments":609,"date":90247,"description":90248,"draft":602,"extension":605,"external":606,"image":90249,"meta":90250,"navigation":609,"path":90251,"seo":90252,"stem":90253,"tags":90254,"__hash__":90256},"blog/2017/04/02/langton-ant-notebook.md","Python script for generating 2D n-state Langton's Ant animations",{"type":8,"value":83662,"toc":90245},[83663,83668,83680,83689,83694,83703,83718,83736,83745,83865,86636,86640,86647,86650,86760,86787,86796,86802,86816,87029,87040,87064,87087,87093,87096,87111,87125,87220,87229,87551,87566,87569,87660,87666,87671,87680,87686,87689,87780,87789,87795,87798,87815,87821,87824,87865,87871,87876,87879,87954,87960,87965,87974,87980,87993,87996,88029,88047,88053,88056,88076,88079,88133,88139,88144,88147,88199,88205,88231,88234,88722,88744,88750,88755,88776,88782,88787,88809,88815,88820,88841,88847,88852,88873,88879,88884,88905,88911,88916,88937,88943,88948,88969,88975,88980,89001,89007,89012,89015,89036,89042,89047,89067,89072,89077,89098,89104,89109,89130,89135,89140,89162,89168,89173,89194,89200,89205,89227,89233,89238,89259,89265,89270,89292,89298,89303,89324,89330,89335,89357,89363,89368,89389,89395,89400,89421,89427,89432,89452,89457,89462,89483,89489,89494,89516,89522,89527,89548,89554,89559,89580,89586,89591,89611,89616,89621,89642,89648,89653,89674,89680,89685,89706,89712,89717,89739,89745,89750,89771,89777,89782,89803,89809,89814,89836,89842,89847,89868,89874,89879,89900,89906,89911,89914,89918,89921,89938,89962,89981,90031,90040,90046,90049,90069,90078,90084,90087,90142,90147,90152,90207,90212,90217,90220,90222,90225,90242],[11,83664,83665],{},[2718,83666],{"alt":20386,"src":83667},"/static/LLRRRLRRRRR.png",[11,83669,83670,83671,83676,83677,643],{},"This is an old project that I would like to refactor. I'm copying the contents of ",[20,83672,83675],{"href":83673,"rel":83674},"https://github.com/briancaffey/cellular-automata/blob/master/ants.ipynb",[24],"this Jupyter notebook"," into this article with the ",[30,83678,83679],{},"jupyter nbconvert ants.ipynb --to markdown",[11,83681,83682,83683,83688],{},"This notebook explores a type of Turing Machine known as ",[20,83684,83687],{"href":83685,"rel":83686},"https://en.wikipedia.org/wiki/Turmite",[24],"termites",". The first part is a script I wrote a few years ago when I was first learning Python. If you are new to learning Python, I suggest you give it a try before reading the script; there's a lot you will learn about flow control and data structures. My script is far from perfect and every time I come back to it there is an idiom I can add and areas that can be refactored and cleaned up. It generates images of 2-dimensional n-state termites on an $a$ x $b$ rectangular grid, or it can generate multiple images (frames) of a single termite as it grows to make a video. Here's an example of a termite animatino that I made using the script below:",[23881,83690],{"width":23883,"height":46038,"src":83691,"frameBorder":9181,"gesture":83692,"allow":83693,"allowFullScreen":609},"https://www.youtube-nocookie.com/embed/Du2DorTLAo4?rel=0","media","encrypted-media",[11,83695,83696,83697,83702],{},"The type of termite explored here is a modified version of a type of cellular automata known as ",[20,83698,83701],{"href":83699,"rel":83700},"https://en.wikipedia.org/wiki/Langton%27s_ant",[24],"Langton's Ant",". Langton's Ant has a simple ruleset: an ant is placed on a 2-dimensional grid of 2-state cells (black or white) with a directional orientation. If the ant is on a black cell at $t=n$, the ant enters the cell on the immediate left at $t=n+1$ and the state of the cell it exits changes to white. If the state of the cell that the ant enters is white, the ant enters the cell immediately to the right and the cell it exits turns black. Around 11,000 steps, the ant enters a 'highway' which results in a repeated motion that moves the ant continually in one direction.",[11,83704,83705,83706,83709,83710,83713,83714,83717],{},"Instead of black and white cells, we can define $n$ number of states (colors) and assign any combination of $n$ instructions (eg. LRLLLRLLLRL). The script below generates generates an arbitrary number of ",[30,83707,83708],{},"ants",". Each number in ",[30,83711,83712],{},"range(ants)"," is converted to binary and then 1s and 0s of the corresponding binary number represent the left and right turns for each individual ant. For example: ",[30,83715,83716],{},"bin(23)"," corresponds to a 5-state ant with the following rules: RLRRR. This method avoids generating isotropes (RLRRR is the same ant as LRLLL).",[11,83719,83720,83721,47429,83724,83726,83727,83730,83731,643],{},"If ",[30,83722,83723],{},"record",[30,83725,36962],{},", one frame will be captured every ",[30,83728,83729],{},"frame_interval"," number of steps. These images can be converted into video easily with open-source programs like ",[20,83732,83735],{"href":83733,"rel":83734},"https://www.blender.org/",[24],"Blender",[11,83737,83738,83739,83744],{},"The last part of the notebook attempts to use new methods from the latest version of scikit-learn (",[20,83740,83743],{"href":83741,"rel":83742},"http://scikit-learn.org/stable/whats_new.html",[24],"0.18.1",") to cluster ants by their behavior: k-means (for clustering) and Isolation Forests (for detecting outliers).",[459,83746,83748],{"className":13136,"code":83747,"language":12886,"meta":464,"style":464},"import PIL\nfrom PIL import Image\nimport random\nimport os\nimport sys\nimport pandas as pd\nimport numpy as np\nimport scipy\nimport matplotlib.pyplot as plt\nfrom __future__ import print_function\nfrom sklearn import cluster\nimport seaborn as sns\n%matplotlib inline\n",[30,83749,83750,83756,83766,83772,83778,83784,83796,83806,83813,83823,83835,83847,83859],{"__ignoreMap":464},[151,83751,83752,83754],{"class":469,"line":470},[151,83753,16859],{"class":1869},[151,83755,44465],{"class":477},[151,83757,83758,83760,83762,83764],{"class":469,"line":488},[151,83759,16853],{"class":1869},[151,83761,44099],{"class":477},[151,83763,44102],{"class":1869},[151,83765,44105],{"class":503},[151,83767,83768,83770],{"class":469,"line":500},[151,83769,16859],{"class":1869},[151,83771,44034],{"class":503},[151,83773,83774,83776],{"class":469,"line":509},[151,83775,16859],{"class":1869},[151,83777,24070],{"class":503},[151,83779,83780,83782],{"class":469,"line":517},[151,83781,16859],{"class":1869},[151,83783,58351],{"class":503},[151,83785,83786,83788,83791,83793],{"class":469,"line":534},[151,83787,16859],{"class":1869},[151,83789,83790],{"class":503}," pandas ",[151,83792,16998],{"class":1869},[151,83794,83795],{"class":503}," pd\n",[151,83797,83798,83800,83802,83804],{"class":469,"line":1413},[151,83799,16859],{"class":1869},[151,83801,24412],{"class":503},[151,83803,16998],{"class":1869},[151,83805,24417],{"class":503},[151,83807,83808,83810],{"class":469,"line":1418},[151,83809,16859],{"class":1869},[151,83811,83812],{"class":503}," scipy\n",[151,83814,83815,83817,83819,83821],{"class":469,"line":2462},[151,83816,16859],{"class":1869},[151,83818,44073],{"class":503},[151,83820,16998],{"class":1869},[151,83822,44078],{"class":503},[151,83824,83825,83827,83830,83832],{"class":469,"line":2471},[151,83826,16853],{"class":1869},[151,83828,83829],{"class":12360}," __future__",[151,83831,44102],{"class":1869},[151,83833,83834],{"class":503}," print_function\n",[151,83836,83837,83839,83842,83844],{"class":469,"line":2480},[151,83838,16853],{"class":1869},[151,83840,83841],{"class":503}," sklearn ",[151,83843,16859],{"class":1869},[151,83845,83846],{"class":503}," cluster\n",[151,83848,83849,83851,83854,83856],{"class":469,"line":2489},[151,83850,16859],{"class":1869},[151,83852,83853],{"class":503}," seaborn ",[151,83855,16998],{"class":1869},[151,83857,83858],{"class":503}," sns\n",[151,83860,83861,83863],{"class":469,"line":2497},[151,83862,44519],{"class":1869},[151,83864,44522],{"class":503},[459,83866,83868],{"className":13136,"code":83867,"language":12886,"meta":464,"style":464},"#this script generates an image (or a series of images) for n-state 2D Langton's Ant cellular automaton.\n\n#SETTINGS\n\n#number of ants to run\nants = 65536\n#ants = 100\n#set record to True to record frames once every frame_interval steps\nrecord = False\nframe_interval = 5000\n\n#Boolean for recording final image\nrecord_final_image = False\n\n#set scale to scale the resulting images in save_image(i) function\nscale = 1\n\n#set the length and width of the square image canvas\nwidth = int(200)\nlength = int(200)\n\n#initialize ant in the center of the grid\n#grid contains length_width**2 cells\nant_pos = int((length*width)/2) + int(width/2)\n\n#boolean to check if the ant touches the border (out of bounds)\noob = False\n\n#number of steps that the ant will take on each walk\niterations = 100000\n\n#for naming the image file below\nnumber = str(iterations)\n\n#set the direction of the ant's first step: 1 --> Right; -1 --> Left. Eliminates mirror images (isotropes) from dataset\ndirection = 1\n\n#Ininitialize a blank square image\nim1 = Image.new('RGBA', (width,length),'white')\n\n#color selection\nwhite = (255,255,255,255)\nred = (255,0,0,255)\norange = (255,128,0,255)\nyellow = (255,255,0,255)\nyellow_green = (128,255,0,255)\ngreen = (0,255,0,255)\nteal = (0,255,255,255)\nlight_blue = (0,128,255,255)\nblue = (0,0,255,255)\npurple = (127,0,255,255)\nblack = (0,0,0,255)\ngrey = (150,150,150,255)\nother = (40,100,50,255)\nbrown = (130,90,44,255)\npink = (244,114,208,255)\nmauve = (118,96,138,255)\nmagenta = (216,0,115,255)\n\ncolor_choices = [red, orange, yellow, light_blue, yellow_green, blue, purple, black, grey, green, teal, light_blue, other, brown, pink, mauve, magenta]\n\n#convert an integer to binary and then convert\ndef num_to_string(num):\n    binary = bin(num)\n    moves = \"\"\n    for x in str(binary)[2:]:\n        if x == '1':\n            moves += \"R\"\n        else:\n            moves += \"L\"\n    return moves\n\n#moves list includes all 16 length moves\n#moves_list = [num_to_string(ant) for ant in range(32768,65536)]\n\n#defines the list of strings that is used for the main loop bellow\nmoves_list = [num_to_string(ant) for ant in range(ants)]\n\n#a list of the dictionaries to by passed into the pandas dataframe for later analysis\ndf_row_list = []\n\n#dataframe object for later analysis\ndf = pd.DataFrame() #index=[0]\n\n#functions for moving the postition of the ant right, left, up or down\ndef move_right():\n    global ant_pos\n    #move ant_pos one pixel to the right\n    ant_pos += 1\n    return ant_pos\ndef move_left():\n    global ant_pos\n    #move ant_pos one pixel to the left\n    ant_pos -= 1\n    return ant_pos\ndef move_up():\n    global ant_pos\n    #move ant_pos one pixel up\n    ant_pos += width\n    return ant_pos\ndef move_down():\n    global ant_pos\n    #move ant_pos one pixel down\n    ant_pos -= width\n    return ant_pos\n\ndef move(color,d):\n    global direction\n    while True:\n        #this part is a little confusing and may need to be rewritten\n        #it uses the current direction of the ant to determine the appropriate direction for the next turn\n        #breaks are used\n        if pix_list[ant_pos][2] == color and direction == width*d:\n            #set the color to the next color in the list, or loop back to the beginning of the list if the end has been reached\n            pix_list[ant_pos][2] = (color + 1) % len(pixel_colors)\n            #save the current postion of the ant\n            init = ant_pos\n            #move the ant\n            move_right()\n            #save the updated position of the ant\n            end = ant_pos\n            #calculate the new direction of the ant by taking the difference between end and init\n            direction = end - init\n            break\n\n        #same idea as above\n        elif pix_list[ant_pos][2] == color and direction == -1*width*d:\n            pix_list[ant_pos][2] = (color + 1) % len(pixel_colors)\n            init = ant_pos\n            move_left()\n            end = ant_pos\n            direction = end - init\n            break\n        #same idea as above\n        elif pix_list[ant_pos][2] == color and direction == 1*d:\n            pix_list[ant_pos][2] = (color + 1) % len(pixel_colors)\n            init = ant_pos\n            move_down()\n            end = ant_pos\n            direction = end - init\n            break\n        #same idea as above\n        elif pix_list[ant_pos][2] == color and direction == -1*d:\n            pix_list[ant_pos][2] = (color + 1) % len(pixel_colors)\n            init = ant_pos\n            move_up()\n            end = ant_pos\n            direction = end - init\n            break\n        break\n\n#captures series of pixels used for generating images\ndef get_pix_series():\n    global pix_series\n    pix_series = []\n    for x in range(len(pix_list)):\n        for y in range(len(pixel_colors)):\n            if pix_list[x][2] == y:\n                pixel = pixel_colors[y]\n                pix_series.append(pixel)\n\n#runs ant along the grid according to the moves (defined above) for the number of steps in iterations (defined above)\ndef run():\n    #variable the tracks the step number if the ant goes out of bounds\n    global oob\n    #converts moves string into a list of 0s and 1s; these numbers correspond to direction and are passed into the move() function\n    moves1 = [1 if x == 'R' else -1 for x in moves]\n    for step in range(iterations):\n        #exit the loop if the ant reaches the edge of the grid\n        if ant_pos \u003C width or ant_pos % width == 0:\n            oob = step\n            return\n        #loop through the moves\n        for index, direction in enumerate(moves1):\n            try:\n                #remember the ant position\n                not_moved = ant_pos\n                #try to move the ant position\n                move(index,direction)\n                #check to see if the position was moved\n                if ant_pos != not_moved:\n                    #set record to false in the settings to turn of frame recording\n                    if record == True:\n                        #records a new frame every frame_interval frame\n                        if step % frame_interval == 0:\n                            counter += 1\n                            print(\"Generating frame number \" + str(counter))\n                            get_pix_series()\n                            save_image(counter)\n                    break\n                else:\n                    continue\n            except:\n                #print(\"Out of bounds at step number \" + str(step))\n                oob = step\n                return\n\ndef save_image(i):\n    #give access to the image instantiated at the beginning of the script\n    global im1\n    #fill blank image canvas with pix_series pixel data\n    im1.putdata(pix_series)\n    #to rescale the image, set the scale variable in settings and call resize on im1\n    im1.resize((scale*im1.size[0],scale*im1.size[1])).save('%s.png' % (moves))\n\n#builds a dictionary to count pixels by color\ndef build_df_row():\n    colors_dict = {str(val): 0 for val, color in enumerate(pixel_colors)}\n    moves_dict = {'moves':moves}\n    last_step = {'last_step':oob}\n    row_dict = dict(colors_dict.items()+moves_dict.items()+last_step.items())\n    for x in pix_list:\n        pixel_color = str(x[2])\n        #print(pixel_color)\n        row_dict[pixel_color] += 1\n    return row_dict\n\n#uncomment below to overwrite moves_list\n#moves_list = ['LR', 'RRLR']\n\nfor _, moves in enumerate(moves_list):\n    oob = 0\n    dir_path = str(_)\n\n    #make a new directory for each new ant walk in walks based on the the walk number and navigate to that directory\n    if record == True:\n        #make a new directory to record frames for a give ant if record is set to true and that directory does not yet exist\n        if not os.path.isdir(dir_path):\n            os.makedirs(dir_path)\n        #otherwise just change into the directory\n        else:\n            os.chdir(dir_path)\n\n    #set ant at middle of grid\n    ant_pos = int((length*width)/2) + int(width/2)\n\n    #moves = len(moves)\n    pixel_colors = color_choices[:(len(moves))]\n\n    #defines an empty list of elements [x,y,0] where x amd y are the position 0 is the 0ht color in the color list (the base canvas color)\n    pix_list = []\n    for x in range(length):\n        for y in range(width):\n            a = [x,y,0]\n            pix_list.append(a)\n\n    #pix_series is a list of pixels that is passed into the put_data function to generate an image\n    pix_series = []\n\n    #counter keeps track of the frame number (if recording a series of images)\n    counter = 0\n\n    #run the ant\n    run()\n\n    #capture the final state of the grid with get_pix_series\n    get_pix_series()\n\n    #uncomment below to preview images for testing\n    #im1.putdata(pix_series)\n    #im1.resize((scale*im1.size[0],scale*im1.size[1])).show()\n\n    #build a dictionary with pixel counts\n    colors_dict = build_df_row()\n    row_df = pd.DataFrame(colors_dict, index=[0])\n    df = df.append(row_df, ignore_index=True)\n\n    if record_final_image == True:\n        os.chdir(os.path.expanduser('~/Documents/CA_1/imgs/'))\n        save_image(_)\n        os.chdir('../')\n\n    #summary\n    print(str(_), end=' ')\n\nos.chdir(os.path.expanduser('~/Documents/CA_1/'))\ndf.to_csv('ants_hist_.csv', index=False)\n",[30,83869,83870,83875,83879,83884,83888,83893,83903,83908,83913,83922,83932,83936,83941,83950,83954,83959,83968,83972,83977,83992,84007,84011,84016,84021,84057,84061,84066,84075,84079,84084,84094,84098,84103,84116,84120,84125,84134,84138,84143,84163,84167,84172,84198,84223,84248,84273,84298,84323,84348,84373,84398,84424,84449,84474,84499,84524,84551,84579,84606,84610,84620,84624,84629,84643,84656,84665,84683,84696,84706,84712,84721,84728,84732,84737,84742,84746,84751,84773,84777,84782,84791,84795,84800,84812,84816,84821,84830,84838,84843,84852,84858,84867,84873,84878,84886,84892,84901,84907,84912,84921,84927,84936,84942,84947,84955,84961,84965,84982,84989,84998,85003,85008,85013,85044,85049,85076,85081,85090,85095,85100,85105,85114,85119,85134,85139,85143,85148,85180,85204,85212,85217,85225,85237,85241,85245,85271,85295,85303,85308,85316,85328,85332,85336,85364,85388,85396,85401,85409,85421,85425,85430,85434,85439,85448,85455,85464,85481,85499,85515,85525,85530,85534,85539,85548,85553,85560,85565,85600,85613,85618,85645,85655,85660,85665,85679,85685,85690,85699,85704,85709,85714,85726,85731,85745,85750,85768,85777,85794,85799,85804,85809,85816,85821,85827,85833,85843,85849,85854,85868,85874,85882,85888,85894,85900,85938,85943,85949,85959,85988,86004,86020,86043,86055,86072,86078,86088,86096,86101,86107,86113,86118,86133,86143,86156,86161,86167,86180,86186,86196,86202,86208,86215,86221,86226,86232,86265,86270,86276,86292,86297,86303,86313,86327,86341,86356,86362,86367,86373,86382,86387,86393,86403,86408,86414,86420,86425,86431,86437,86442,86448,86454,86460,86465,86471,86481,86503,86522,86527,86541,86552,86558,86569,86574,86580,86601,86606,86617],{"__ignoreMap":464},[151,83871,83872],{"class":469,"line":470},[151,83873,83874],{"class":1527},"#this script generates an image (or a series of images) for n-state 2D Langton's Ant cellular automaton.\n",[151,83876,83877],{"class":469,"line":488},[151,83878,1090],{"emptyLinePlaceholder":609},[151,83880,83881],{"class":469,"line":500},[151,83882,83883],{"class":1527},"#SETTINGS\n",[151,83885,83886],{"class":469,"line":509},[151,83887,1090],{"emptyLinePlaceholder":609},[151,83889,83890],{"class":469,"line":517},[151,83891,83892],{"class":1527},"#number of ants to run\n",[151,83894,83895,83898,83900],{"class":469,"line":534},[151,83896,83897],{"class":503},"ants ",[151,83899,1876],{"class":1869},[151,83901,83902],{"class":477}," 65536\n",[151,83904,83905],{"class":469,"line":1413},[151,83906,83907],{"class":1527},"#ants = 100\n",[151,83909,83910],{"class":469,"line":1418},[151,83911,83912],{"class":1527},"#set record to True to record frames once every frame_interval steps\n",[151,83914,83915,83918,83920],{"class":469,"line":2462},[151,83916,83917],{"class":503},"record ",[151,83919,1876],{"class":1869},[151,83921,66889],{"class":477},[151,83923,83924,83927,83929],{"class":469,"line":2471},[151,83925,83926],{"class":503},"frame_interval ",[151,83928,1876],{"class":1869},[151,83930,83931],{"class":477}," 5000\n",[151,83933,83934],{"class":469,"line":2480},[151,83935,1090],{"emptyLinePlaceholder":609},[151,83937,83938],{"class":469,"line":2489},[151,83939,83940],{"class":1527},"#Boolean for recording final image\n",[151,83942,83943,83946,83948],{"class":469,"line":2497},[151,83944,83945],{"class":503},"record_final_image ",[151,83947,1876],{"class":1869},[151,83949,66889],{"class":477},[151,83951,83952],{"class":469,"line":3140},[151,83953,1090],{"emptyLinePlaceholder":609},[151,83955,83956],{"class":469,"line":3149},[151,83957,83958],{"class":1527},"#set scale to scale the resulting images in save_image(i) function\n",[151,83960,83961,83964,83966],{"class":469,"line":3158},[151,83962,83963],{"class":503},"scale ",[151,83965,1876],{"class":1869},[151,83967,3181],{"class":477},[151,83969,83970],{"class":469,"line":3167},[151,83971,1090],{"emptyLinePlaceholder":609},[151,83973,83974],{"class":469,"line":3175},[151,83975,83976],{"class":1527},"#set the length and width of the square image canvas\n",[151,83978,83979,83982,83984,83986,83988,83990],{"class":469,"line":3184},[151,83980,83981],{"class":503},"width ",[151,83983,1876],{"class":1869},[151,83985,16673],{"class":6205},[151,83987,12386],{"class":503},[151,83989,41624],{"class":477},[151,83991,3640],{"class":503},[151,83993,83994,83997,83999,84001,84003,84005],{"class":469,"line":3193},[151,83995,83996],{"class":503},"length ",[151,83998,1876],{"class":1869},[151,84000,16673],{"class":6205},[151,84002,12386],{"class":503},[151,84004,41624],{"class":477},[151,84006,3640],{"class":503},[151,84008,84009],{"class":469,"line":3720},[151,84010,1090],{"emptyLinePlaceholder":609},[151,84012,84013],{"class":469,"line":3729},[151,84014,84015],{"class":1527},"#initialize ant in the center of the grid\n",[151,84017,84018],{"class":469,"line":3735},[151,84019,84020],{"class":1527},"#grid contains length_width**2 cells\n",[151,84022,84023,84026,84028,84030,84033,84035,84038,84040,84042,84044,84046,84048,84051,84053,84055],{"class":469,"line":3745},[151,84024,84025],{"class":503},"ant_pos ",[151,84027,1876],{"class":1869},[151,84029,16673],{"class":6205},[151,84031,84032],{"class":503},"((length",[151,84034,23268],{"class":1869},[151,84036,84037],{"class":503},"width)",[151,84039,19883],{"class":1869},[151,84041,6619],{"class":477},[151,84043,16995],{"class":503},[151,84045,22885],{"class":1869},[151,84047,16673],{"class":6205},[151,84049,84050],{"class":503},"(width",[151,84052,19883],{"class":1869},[151,84054,6619],{"class":477},[151,84056,3640],{"class":503},[151,84058,84059],{"class":469,"line":3754},[151,84060,1090],{"emptyLinePlaceholder":609},[151,84062,84063],{"class":469,"line":3760},[151,84064,84065],{"class":1527},"#boolean to check if the ant touches the border (out of bounds)\n",[151,84067,84068,84071,84073],{"class":469,"line":3773},[151,84069,84070],{"class":503},"oob ",[151,84072,1876],{"class":1869},[151,84074,66889],{"class":477},[151,84076,84077],{"class":469,"line":3782},[151,84078,1090],{"emptyLinePlaceholder":609},[151,84080,84081],{"class":469,"line":3791},[151,84082,84083],{"class":1527},"#number of steps that the ant will take on each walk\n",[151,84085,84086,84089,84091],{"class":469,"line":3803},[151,84087,84088],{"class":503},"iterations ",[151,84090,1876],{"class":1869},[151,84092,84093],{"class":477}," 100000\n",[151,84095,84096],{"class":469,"line":3811},[151,84097,1090],{"emptyLinePlaceholder":609},[151,84099,84100],{"class":469,"line":3820},[151,84101,84102],{"class":1527},"#for naming the image file below\n",[151,84104,84105,84108,84110,84113],{"class":469,"line":7084},[151,84106,84107],{"class":503},"number ",[151,84109,1876],{"class":1869},[151,84111,84112],{"class":6205}," str",[151,84114,84115],{"class":503},"(iterations)\n",[151,84117,84118],{"class":469,"line":7148},[151,84119,1090],{"emptyLinePlaceholder":609},[151,84121,84122],{"class":469,"line":7211},[151,84123,84124],{"class":1527},"#set the direction of the ant's first step: 1 --> Right; -1 --> Left. Eliminates mirror images (isotropes) from dataset\n",[151,84126,84127,84130,84132],{"class":469,"line":7273},[151,84128,84129],{"class":503},"direction ",[151,84131,1876],{"class":1869},[151,84133,3181],{"class":477},[151,84135,84136],{"class":469,"line":7335},[151,84137,1090],{"emptyLinePlaceholder":609},[151,84139,84140],{"class":469,"line":7398},[151,84141,84142],{"class":1527},"#Ininitialize a blank square image\n",[151,84144,84145,84148,84150,84153,84156,84159,84161],{"class":469,"line":7462},[151,84146,84147],{"class":503},"im1 ",[151,84149,1876],{"class":1869},[151,84151,84152],{"class":503}," Image.new(",[151,84154,84155],{"class":481},"'RGBA'",[151,84157,84158],{"class":503},", (width,length),",[151,84160,45309],{"class":481},[151,84162,3640],{"class":503},[151,84164,84165],{"class":469,"line":7467},[151,84166,1090],{"emptyLinePlaceholder":609},[151,84168,84169],{"class":469,"line":7532},[151,84170,84171],{"class":1527},"#color selection\n",[151,84173,84174,84177,84179,84181,84184,84186,84188,84190,84192,84194,84196],{"class":469,"line":7537},[151,84175,84176],{"class":503},"white ",[151,84178,1876],{"class":1869},[151,84180,129],{"class":503},[151,84182,84183],{"class":477},"255",[151,84185,3634],{"class":503},[151,84187,84183],{"class":477},[151,84189,3634],{"class":503},[151,84191,84183],{"class":477},[151,84193,3634],{"class":503},[151,84195,84183],{"class":477},[151,84197,3640],{"class":503},[151,84199,84200,84203,84205,84207,84209,84211,84213,84215,84217,84219,84221],{"class":469,"line":7603},[151,84201,84202],{"class":503},"red ",[151,84204,1876],{"class":1869},[151,84206,129],{"class":503},[151,84208,84183],{"class":477},[151,84210,3634],{"class":503},[151,84212,9181],{"class":477},[151,84214,3634],{"class":503},[151,84216,9181],{"class":477},[151,84218,3634],{"class":503},[151,84220,84183],{"class":477},[151,84222,3640],{"class":503},[151,84224,84225,84228,84230,84232,84234,84236,84238,84240,84242,84244,84246],{"class":469,"line":7608},[151,84226,84227],{"class":503},"orange ",[151,84229,1876],{"class":1869},[151,84231,129],{"class":503},[151,84233,84183],{"class":477},[151,84235,3634],{"class":503},[151,84237,9488],{"class":477},[151,84239,3634],{"class":503},[151,84241,9181],{"class":477},[151,84243,3634],{"class":503},[151,84245,84183],{"class":477},[151,84247,3640],{"class":503},[151,84249,84250,84253,84255,84257,84259,84261,84263,84265,84267,84269,84271],{"class":469,"line":7673},[151,84251,84252],{"class":503},"yellow ",[151,84254,1876],{"class":1869},[151,84256,129],{"class":503},[151,84258,84183],{"class":477},[151,84260,3634],{"class":503},[151,84262,84183],{"class":477},[151,84264,3634],{"class":503},[151,84266,9181],{"class":477},[151,84268,3634],{"class":503},[151,84270,84183],{"class":477},[151,84272,3640],{"class":503},[151,84274,84275,84278,84280,84282,84284,84286,84288,84290,84292,84294,84296],{"class":469,"line":7678},[151,84276,84277],{"class":503},"yellow_green ",[151,84279,1876],{"class":1869},[151,84281,129],{"class":503},[151,84283,9488],{"class":477},[151,84285,3634],{"class":503},[151,84287,84183],{"class":477},[151,84289,3634],{"class":503},[151,84291,9181],{"class":477},[151,84293,3634],{"class":503},[151,84295,84183],{"class":477},[151,84297,3640],{"class":503},[151,84299,84300,84303,84305,84307,84309,84311,84313,84315,84317,84319,84321],{"class":469,"line":7708},[151,84301,84302],{"class":503},"green ",[151,84304,1876],{"class":1869},[151,84306,129],{"class":503},[151,84308,9181],{"class":477},[151,84310,3634],{"class":503},[151,84312,84183],{"class":477},[151,84314,3634],{"class":503},[151,84316,9181],{"class":477},[151,84318,3634],{"class":503},[151,84320,84183],{"class":477},[151,84322,3640],{"class":503},[151,84324,84325,84328,84330,84332,84334,84336,84338,84340,84342,84344,84346],{"class":469,"line":7713},[151,84326,84327],{"class":503},"teal ",[151,84329,1876],{"class":1869},[151,84331,129],{"class":503},[151,84333,9181],{"class":477},[151,84335,3634],{"class":503},[151,84337,84183],{"class":477},[151,84339,3634],{"class":503},[151,84341,84183],{"class":477},[151,84343,3634],{"class":503},[151,84345,84183],{"class":477},[151,84347,3640],{"class":503},[151,84349,84350,84353,84355,84357,84359,84361,84363,84365,84367,84369,84371],{"class":469,"line":7746},[151,84351,84352],{"class":503},"light_blue ",[151,84354,1876],{"class":1869},[151,84356,129],{"class":503},[151,84358,9181],{"class":477},[151,84360,3634],{"class":503},[151,84362,9488],{"class":477},[151,84364,3634],{"class":503},[151,84366,84183],{"class":477},[151,84368,3634],{"class":503},[151,84370,84183],{"class":477},[151,84372,3640],{"class":503},[151,84374,84375,84378,84380,84382,84384,84386,84388,84390,84392,84394,84396],{"class":469,"line":7751},[151,84376,84377],{"class":503},"blue ",[151,84379,1876],{"class":1869},[151,84381,129],{"class":503},[151,84383,9181],{"class":477},[151,84385,3634],{"class":503},[151,84387,9181],{"class":477},[151,84389,3634],{"class":503},[151,84391,84183],{"class":477},[151,84393,3634],{"class":503},[151,84395,84183],{"class":477},[151,84397,3640],{"class":503},[151,84399,84400,84403,84405,84407,84410,84412,84414,84416,84418,84420,84422],{"class":469,"line":7816},[151,84401,84402],{"class":503},"purple ",[151,84404,1876],{"class":1869},[151,84406,129],{"class":503},[151,84408,84409],{"class":477},"127",[151,84411,3634],{"class":503},[151,84413,9181],{"class":477},[151,84415,3634],{"class":503},[151,84417,84183],{"class":477},[151,84419,3634],{"class":503},[151,84421,84183],{"class":477},[151,84423,3640],{"class":503},[151,84425,84426,84429,84431,84433,84435,84437,84439,84441,84443,84445,84447],{"class":469,"line":7821},[151,84427,84428],{"class":503},"black ",[151,84430,1876],{"class":1869},[151,84432,129],{"class":503},[151,84434,9181],{"class":477},[151,84436,3634],{"class":503},[151,84438,9181],{"class":477},[151,84440,3634],{"class":503},[151,84442,9181],{"class":477},[151,84444,3634],{"class":503},[151,84446,84183],{"class":477},[151,84448,3640],{"class":503},[151,84450,84451,84454,84456,84458,84460,84462,84464,84466,84468,84470,84472],{"class":469,"line":7847},[151,84452,84453],{"class":503},"grey ",[151,84455,1876],{"class":1869},[151,84457,129],{"class":503},[151,84459,45949],{"class":477},[151,84461,3634],{"class":503},[151,84463,45949],{"class":477},[151,84465,3634],{"class":503},[151,84467,45949],{"class":477},[151,84469,3634],{"class":503},[151,84471,84183],{"class":477},[151,84473,3640],{"class":503},[151,84475,84476,84479,84481,84483,84485,84487,84489,84491,84493,84495,84497],{"class":469,"line":7852},[151,84477,84478],{"class":503},"other ",[151,84480,1876],{"class":1869},[151,84482,129],{"class":503},[151,84484,44365],{"class":477},[151,84486,3634],{"class":503},[151,84488,71821],{"class":477},[151,84490,3634],{"class":503},[151,84492,73146],{"class":477},[151,84494,3634],{"class":503},[151,84496,84183],{"class":477},[151,84498,3640],{"class":503},[151,84500,84501,84504,84506,84508,84510,84512,84514,84516,84518,84520,84522],{"class":469,"line":7887},[151,84502,84503],{"class":503},"brown ",[151,84505,1876],{"class":1869},[151,84507,129],{"class":503},[151,84509,73296],{"class":477},[151,84511,3634],{"class":503},[151,84513,65941],{"class":477},[151,84515,3634],{"class":503},[151,84517,41885],{"class":477},[151,84519,3634],{"class":503},[151,84521,84183],{"class":477},[151,84523,3640],{"class":503},[151,84525,84526,84529,84531,84533,84536,84538,84540,84542,84545,84547,84549],{"class":469,"line":7892},[151,84527,84528],{"class":503},"pink ",[151,84530,1876],{"class":1869},[151,84532,129],{"class":503},[151,84534,84535],{"class":477},"244",[151,84537,3634],{"class":503},[151,84539,41675],{"class":477},[151,84541,3634],{"class":503},[151,84543,84544],{"class":477},"208",[151,84546,3634],{"class":503},[151,84548,84183],{"class":477},[151,84550,3640],{"class":503},[151,84552,84553,84556,84558,84560,84563,84565,84568,84570,84573,84575,84577],{"class":469,"line":7924},[151,84554,84555],{"class":503},"mauve ",[151,84557,1876],{"class":1869},[151,84559,129],{"class":503},[151,84561,84562],{"class":477},"118",[151,84564,3634],{"class":503},[151,84566,84567],{"class":477},"96",[151,84569,3634],{"class":503},[151,84571,84572],{"class":477},"138",[151,84574,3634],{"class":503},[151,84576,84183],{"class":477},[151,84578,3640],{"class":503},[151,84580,84581,84584,84586,84588,84591,84593,84595,84597,84600,84602,84604],{"class":469,"line":7929},[151,84582,84583],{"class":503},"magenta ",[151,84585,1876],{"class":1869},[151,84587,129],{"class":503},[151,84589,84590],{"class":477},"216",[151,84592,3634],{"class":503},[151,84594,9181],{"class":477},[151,84596,3634],{"class":503},[151,84598,84599],{"class":477},"115",[151,84601,3634],{"class":503},[151,84603,84183],{"class":477},[151,84605,3640],{"class":503},[151,84607,84608],{"class":469,"line":7991},[151,84609,1090],{"emptyLinePlaceholder":609},[151,84611,84612,84615,84617],{"class":469,"line":7996},[151,84613,84614],{"class":503},"color_choices ",[151,84616,1876],{"class":1869},[151,84618,84619],{"class":503}," [red, orange, yellow, light_blue, yellow_green, blue, purple, black, grey, green, teal, light_blue, other, brown, pink, mauve, magenta]\n",[151,84621,84622],{"class":469,"line":8078},[151,84623,1090],{"emptyLinePlaceholder":609},[151,84625,84626],{"class":469,"line":8140},[151,84627,84628],{"class":1527},"#convert an integer to binary and then convert\n",[151,84630,84631,84633,84636,84638,84641],{"class":469,"line":8145},[151,84632,16925],{"class":12347},[151,84634,84635],{"class":473}," num_to_string",[151,84637,12386],{"class":503},[151,84639,84640],{"class":15232},"num",[151,84642,15264],{"class":503},[151,84644,84645,84648,84650,84653],{"class":469,"line":8259},[151,84646,84647],{"class":503},"    binary ",[151,84649,1876],{"class":1869},[151,84651,84652],{"class":2226}," bin",[151,84654,84655],{"class":503},"(num)\n",[151,84657,84658,84661,84663],{"class":469,"line":8264},[151,84659,84660],{"class":503},"    moves ",[151,84662,1876],{"class":1869},[151,84664,38981],{"class":481},[151,84666,84667,84669,84671,84673,84675,84678,84680],{"class":469,"line":8613},[151,84668,16411],{"class":1869},[151,84670,44552],{"class":503},[151,84672,16417],{"class":1869},[151,84674,84112],{"class":6205},[151,84676,84677],{"class":503},"(binary)[",[151,84679,6619],{"class":477},[151,84681,84682],{"class":503},":]:\n",[151,84684,84685,84687,84689,84691,84694],{"class":469,"line":8678},[151,84686,23357],{"class":1869},[151,84688,44552],{"class":503},[151,84690,17223],{"class":1869},[151,84692,84693],{"class":481}," '1'",[151,84695,14372],{"class":503},[151,84697,84698,84701,84703],{"class":469,"line":8742},[151,84699,84700],{"class":503},"            moves ",[151,84702,24780],{"class":1869},[151,84704,84705],{"class":481}," \"R\"\n",[151,84707,84708,84710],{"class":469,"line":8806},[151,84709,23395],{"class":1869},[151,84711,14372],{"class":503},[151,84713,84714,84716,84718],{"class":469,"line":8870},[151,84715,84700],{"class":503},[151,84717,24780],{"class":1869},[151,84719,84720],{"class":481}," \"L\"\n",[151,84722,84723,84725],{"class":469,"line":8875},[151,84724,17496],{"class":1869},[151,84726,84727],{"class":503}," moves\n",[151,84729,84730],{"class":469,"line":8881},[151,84731,1090],{"emptyLinePlaceholder":609},[151,84733,84734],{"class":469,"line":8886},[151,84735,84736],{"class":1527},"#moves list includes all 16 length moves\n",[151,84738,84739],{"class":469,"line":8892},[151,84740,84741],{"class":1527},"#moves_list = [num_to_string(ant) for ant in range(32768,65536)]\n",[151,84743,84744],{"class":469,"line":8963},[151,84745,1090],{"emptyLinePlaceholder":609},[151,84747,84748],{"class":469,"line":8969},[151,84749,84750],{"class":1527},"#defines the list of strings that is used for the main loop bellow\n",[151,84752,84753,84756,84758,84761,84763,84766,84768,84770],{"class":469,"line":15001},[151,84754,84755],{"class":503},"moves_list ",[151,84757,1876],{"class":1869},[151,84759,84760],{"class":503}," [num_to_string(ant) ",[151,84762,16732],{"class":1869},[151,84764,84765],{"class":503}," ant ",[151,84767,16417],{"class":1869},[151,84769,2793],{"class":2226},[151,84771,84772],{"class":503},"(ants)]\n",[151,84774,84775],{"class":469,"line":15009},[151,84776,1090],{"emptyLinePlaceholder":609},[151,84778,84779],{"class":469,"line":15019},[151,84780,84781],{"class":1527},"#a list of the dictionaries to by passed into the pandas dataframe for later analysis\n",[151,84783,84784,84787,84789],{"class":469,"line":15027},[151,84785,84786],{"class":503},"df_row_list ",[151,84788,1876],{"class":1869},[151,84790,16606],{"class":503},[151,84792,84793],{"class":469,"line":15037},[151,84794,1090],{"emptyLinePlaceholder":609},[151,84796,84797],{"class":469,"line":15045},[151,84798,84799],{"class":1527},"#dataframe object for later analysis\n",[151,84801,84802,84804,84806,84809],{"class":469,"line":15055},[151,84803,70720],{"class":503},[151,84805,1876],{"class":1869},[151,84807,84808],{"class":503}," pd.DataFrame() ",[151,84810,84811],{"class":1527},"#index=[0]\n",[151,84813,84814],{"class":469,"line":15060},[151,84815,1090],{"emptyLinePlaceholder":609},[151,84817,84818],{"class":469,"line":15068},[151,84819,84820],{"class":1527},"#functions for moving the postition of the ant right, left, up or down\n",[151,84822,84823,84825,84828],{"class":469,"line":15076},[151,84824,16925],{"class":12347},[151,84826,84827],{"class":473}," move_right",[151,84829,16931],{"class":503},[151,84831,84832,84835],{"class":469,"line":15085},[151,84833,84834],{"class":1869},"    global",[151,84836,84837],{"class":503}," ant_pos\n",[151,84839,84840],{"class":469,"line":15095},[151,84841,84842],{"class":1527},"    #move ant_pos one pixel to the right\n",[151,84844,84845,84848,84850],{"class":469,"line":15105},[151,84846,84847],{"class":503},"    ant_pos ",[151,84849,24780],{"class":1869},[151,84851,3181],{"class":477},[151,84853,84854,84856],{"class":469,"line":15110},[151,84855,17496],{"class":1869},[151,84857,84837],{"class":503},[151,84859,84860,84862,84865],{"class":469,"line":15118},[151,84861,16925],{"class":12347},[151,84863,84864],{"class":473}," move_left",[151,84866,16931],{"class":503},[151,84868,84869,84871],{"class":469,"line":15128},[151,84870,84834],{"class":1869},[151,84872,84837],{"class":503},[151,84874,84875],{"class":469,"line":15139},[151,84876,84877],{"class":1527},"    #move ant_pos one pixel to the left\n",[151,84879,84880,84882,84884],{"class":469,"line":31954},[151,84881,84847],{"class":503},[151,84883,78694],{"class":1869},[151,84885,3181],{"class":477},[151,84887,84888,84890],{"class":469,"line":31960},[151,84889,17496],{"class":1869},[151,84891,84837],{"class":503},[151,84893,84894,84896,84899],{"class":469,"line":31965},[151,84895,16925],{"class":12347},[151,84897,84898],{"class":473}," move_up",[151,84900,16931],{"class":503},[151,84902,84903,84905],{"class":469,"line":31971},[151,84904,84834],{"class":1869},[151,84906,84837],{"class":503},[151,84908,84909],{"class":469,"line":31983},[151,84910,84911],{"class":1527},"    #move ant_pos one pixel up\n",[151,84913,84914,84916,84918],{"class":469,"line":31994},[151,84915,84847],{"class":503},[151,84917,24780],{"class":1869},[151,84919,84920],{"class":503}," width\n",[151,84922,84923,84925],{"class":469,"line":32007},[151,84924,17496],{"class":1869},[151,84926,84837],{"class":503},[151,84928,84929,84931,84934],{"class":469,"line":32018},[151,84930,16925],{"class":12347},[151,84932,84933],{"class":473}," move_down",[151,84935,16931],{"class":503},[151,84937,84938,84940],{"class":469,"line":32026},[151,84939,84834],{"class":1869},[151,84941,84837],{"class":503},[151,84943,84944],{"class":469,"line":32031},[151,84945,84946],{"class":1527},"    #move ant_pos one pixel down\n",[151,84948,84949,84951,84953],{"class":469,"line":32036},[151,84950,84847],{"class":503},[151,84952,78694],{"class":1869},[151,84954,84920],{"class":503},[151,84956,84957,84959],{"class":469,"line":32042},[151,84958,17496],{"class":1869},[151,84960,84837],{"class":503},[151,84962,84963],{"class":469,"line":32054},[151,84964,1090],{"emptyLinePlaceholder":609},[151,84966,84967,84969,84972,84974,84976,84978,84980],{"class":469,"line":32067},[151,84968,16925],{"class":12347},[151,84970,84971],{"class":473}," move",[151,84973,12386],{"class":503},[151,84975,79362],{"class":15232},[151,84977,3634],{"class":503},[151,84979,78271],{"class":15232},[151,84981,15264],{"class":503},[151,84983,84984,84986],{"class":469,"line":32086},[151,84985,84834],{"class":1869},[151,84987,84988],{"class":503}," direction\n",[151,84990,84991,84994,84996],{"class":469,"line":32097},[151,84992,84993],{"class":1869},"    while",[151,84995,68564],{"class":477},[151,84997,14372],{"class":503},[151,84999,85000],{"class":469,"line":25585},[151,85001,85002],{"class":1527},"        #this part is a little confusing and may need to be rewritten\n",[151,85004,85005],{"class":469,"line":32112},[151,85006,85007],{"class":1527},"        #it uses the current direction of the ant to determine the appropriate direction for the next turn\n",[151,85009,85010],{"class":469,"line":32117},[151,85011,85012],{"class":1527},"        #breaks are used\n",[151,85014,85015,85017,85020,85022,85024,85026,85029,85031,85034,85036,85039,85041],{"class":469,"line":32123},[151,85016,23357],{"class":1869},[151,85018,85019],{"class":503}," pix_list[ant_pos][",[151,85021,6619],{"class":477},[151,85023,16654],{"class":503},[151,85025,17223],{"class":1869},[151,85027,85028],{"class":503}," color ",[151,85030,40499],{"class":1869},[151,85032,85033],{"class":503}," direction ",[151,85035,17223],{"class":1869},[151,85037,85038],{"class":503}," width",[151,85040,23268],{"class":1869},[151,85042,85043],{"class":503},"d:\n",[151,85045,85046],{"class":469,"line":32151},[151,85047,85048],{"class":1527},"            #set the color to the next color in the list, or loop back to the beginning of the list if the end has been reached\n",[151,85050,85051,85054,85056,85058,85060,85063,85065,85067,85069,85071,85073],{"class":469,"line":32156},[151,85052,85053],{"class":503},"            pix_list[ant_pos][",[151,85055,6619],{"class":477},[151,85057,16654],{"class":503},[151,85059,1876],{"class":1869},[151,85061,85062],{"class":503}," (color ",[151,85064,22885],{"class":1869},[151,85066,12448],{"class":477},[151,85068,16995],{"class":503},[151,85070,44519],{"class":1869},[151,85072,45035],{"class":2226},[151,85074,85075],{"class":503},"(pixel_colors)\n",[151,85077,85078],{"class":469,"line":32162},[151,85079,85080],{"class":1527},"            #save the current postion of the ant\n",[151,85082,85083,85086,85088],{"class":469,"line":32168},[151,85084,85085],{"class":503},"            init ",[151,85087,1876],{"class":1869},[151,85089,84837],{"class":503},[151,85091,85092],{"class":469,"line":32180},[151,85093,85094],{"class":1527},"            #move the ant\n",[151,85096,85097],{"class":469,"line":32192},[151,85098,85099],{"class":503},"            move_right()\n",[151,85101,85102],{"class":469,"line":32207},[151,85103,85104],{"class":1527},"            #save the updated position of the ant\n",[151,85106,85107,85110,85112],{"class":469,"line":32217},[151,85108,85109],{"class":503},"            end ",[151,85111,1876],{"class":1869},[151,85113,84837],{"class":503},[151,85115,85116],{"class":469,"line":32226},[151,85117,85118],{"class":1527},"            #calculate the new direction of the ant by taking the difference between end and init\n",[151,85120,85121,85124,85126,85129,85131],{"class":469,"line":32231},[151,85122,85123],{"class":503},"            direction ",[151,85125,1876],{"class":1869},[151,85127,85128],{"class":503}," end ",[151,85130,12445],{"class":1869},[151,85132,85133],{"class":503}," init\n",[151,85135,85136],{"class":469,"line":32236},[151,85137,85138],{"class":1869},"            break\n",[151,85140,85141],{"class":469,"line":32244},[151,85142,1090],{"emptyLinePlaceholder":609},[151,85144,85145],{"class":469,"line":32249},[151,85146,85147],{"class":1527},"        #same idea as above\n",[151,85149,85150,85152,85154,85156,85158,85160,85162,85164,85166,85168,85170,85172,85174,85176,85178],{"class":469,"line":32255},[151,85151,39233],{"class":1869},[151,85153,85019],{"class":503},[151,85155,6619],{"class":477},[151,85157,16654],{"class":503},[151,85159,17223],{"class":1869},[151,85161,85028],{"class":503},[151,85163,40499],{"class":1869},[151,85165,85033],{"class":503},[151,85167,17223],{"class":1869},[151,85169,9949],{"class":1869},[151,85171,6760],{"class":477},[151,85173,23268],{"class":1869},[151,85175,44275],{"class":503},[151,85177,23268],{"class":1869},[151,85179,85043],{"class":503},[151,85181,85182,85184,85186,85188,85190,85192,85194,85196,85198,85200,85202],{"class":469,"line":32272},[151,85183,85053],{"class":503},[151,85185,6619],{"class":477},[151,85187,16654],{"class":503},[151,85189,1876],{"class":1869},[151,85191,85062],{"class":503},[151,85193,22885],{"class":1869},[151,85195,12448],{"class":477},[151,85197,16995],{"class":503},[151,85199,44519],{"class":1869},[151,85201,45035],{"class":2226},[151,85203,85075],{"class":503},[151,85205,85206,85208,85210],{"class":469,"line":32277},[151,85207,85085],{"class":503},[151,85209,1876],{"class":1869},[151,85211,84837],{"class":503},[151,85213,85214],{"class":469,"line":32283},[151,85215,85216],{"class":503},"            move_left()\n",[151,85218,85219,85221,85223],{"class":469,"line":32295},[151,85220,85109],{"class":503},[151,85222,1876],{"class":1869},[151,85224,84837],{"class":503},[151,85226,85227,85229,85231,85233,85235],{"class":469,"line":32307},[151,85228,85123],{"class":503},[151,85230,1876],{"class":1869},[151,85232,85128],{"class":503},[151,85234,12445],{"class":1869},[151,85236,85133],{"class":503},[151,85238,85239],{"class":469,"line":32320},[151,85240,85138],{"class":1869},[151,85242,85243],{"class":469,"line":32330},[151,85244,85147],{"class":1527},[151,85246,85247,85249,85251,85253,85255,85257,85259,85261,85263,85265,85267,85269],{"class":469,"line":32352},[151,85248,39233],{"class":1869},[151,85250,85019],{"class":503},[151,85252,6619],{"class":477},[151,85254,16654],{"class":503},[151,85256,17223],{"class":1869},[151,85258,85028],{"class":503},[151,85260,40499],{"class":1869},[151,85262,85033],{"class":503},[151,85264,17223],{"class":1869},[151,85266,12448],{"class":477},[151,85268,23268],{"class":1869},[151,85270,85043],{"class":503},[151,85272,85273,85275,85277,85279,85281,85283,85285,85287,85289,85291,85293],{"class":469,"line":32366},[151,85274,85053],{"class":503},[151,85276,6619],{"class":477},[151,85278,16654],{"class":503},[151,85280,1876],{"class":1869},[151,85282,85062],{"class":503},[151,85284,22885],{"class":1869},[151,85286,12448],{"class":477},[151,85288,16995],{"class":503},[151,85290,44519],{"class":1869},[151,85292,45035],{"class":2226},[151,85294,85075],{"class":503},[151,85296,85297,85299,85301],{"class":469,"line":32371},[151,85298,85085],{"class":503},[151,85300,1876],{"class":1869},[151,85302,84837],{"class":503},[151,85304,85305],{"class":469,"line":32376},[151,85306,85307],{"class":503},"            move_down()\n",[151,85309,85310,85312,85314],{"class":469,"line":32389},[151,85311,85109],{"class":503},[151,85313,1876],{"class":1869},[151,85315,84837],{"class":503},[151,85317,85318,85320,85322,85324,85326],{"class":469,"line":32394},[151,85319,85123],{"class":503},[151,85321,1876],{"class":1869},[151,85323,85128],{"class":503},[151,85325,12445],{"class":1869},[151,85327,85133],{"class":503},[151,85329,85330],{"class":469,"line":32400},[151,85331,85138],{"class":1869},[151,85333,85334],{"class":469,"line":32406},[151,85335,85147],{"class":1527},[151,85337,85338,85340,85342,85344,85346,85348,85350,85352,85354,85356,85358,85360,85362],{"class":469,"line":32412},[151,85339,39233],{"class":1869},[151,85341,85019],{"class":503},[151,85343,6619],{"class":477},[151,85345,16654],{"class":503},[151,85347,17223],{"class":1869},[151,85349,85028],{"class":503},[151,85351,40499],{"class":1869},[151,85353,85033],{"class":503},[151,85355,17223],{"class":1869},[151,85357,9949],{"class":1869},[151,85359,6760],{"class":477},[151,85361,23268],{"class":1869},[151,85363,85043],{"class":503},[151,85365,85366,85368,85370,85372,85374,85376,85378,85380,85382,85384,85386],{"class":469,"line":32418},[151,85367,85053],{"class":503},[151,85369,6619],{"class":477},[151,85371,16654],{"class":503},[151,85373,1876],{"class":1869},[151,85375,85062],{"class":503},[151,85377,22885],{"class":1869},[151,85379,12448],{"class":477},[151,85381,16995],{"class":503},[151,85383,44519],{"class":1869},[151,85385,45035],{"class":2226},[151,85387,85075],{"class":503},[151,85389,85390,85392,85394],{"class":469,"line":32433},[151,85391,85085],{"class":503},[151,85393,1876],{"class":1869},[151,85395,84837],{"class":503},[151,85397,85398],{"class":469,"line":32443},[151,85399,85400],{"class":503},"            move_up()\n",[151,85402,85403,85405,85407],{"class":469,"line":32454},[151,85404,85109],{"class":503},[151,85406,1876],{"class":1869},[151,85408,84837],{"class":503},[151,85410,85411,85413,85415,85417,85419],{"class":469,"line":32459},[151,85412,85123],{"class":503},[151,85414,1876],{"class":1869},[151,85416,85128],{"class":503},[151,85418,12445],{"class":1869},[151,85420,85133],{"class":503},[151,85422,85423],{"class":469,"line":32464},[151,85424,85138],{"class":1869},[151,85426,85427],{"class":469,"line":32480},[151,85428,85429],{"class":1869},"        break\n",[151,85431,85432],{"class":469,"line":32485},[151,85433,1090],{"emptyLinePlaceholder":609},[151,85435,85436],{"class":469,"line":32491},[151,85437,85438],{"class":1527},"#captures series of pixels used for generating images\n",[151,85440,85441,85443,85446],{"class":469,"line":32503},[151,85442,16925],{"class":12347},[151,85444,85445],{"class":473}," get_pix_series",[151,85447,16931],{"class":503},[151,85449,85450,85452],{"class":469,"line":32519},[151,85451,84834],{"class":1869},[151,85453,85454],{"class":503}," pix_series\n",[151,85456,85457,85460,85462],{"class":469,"line":32538},[151,85458,85459],{"class":503},"    pix_series ",[151,85461,1876],{"class":1869},[151,85463,16606],{"class":503},[151,85465,85466,85468,85470,85472,85474,85476,85478],{"class":469,"line":32549},[151,85467,16411],{"class":1869},[151,85469,44552],{"class":503},[151,85471,16417],{"class":1869},[151,85473,2793],{"class":2226},[151,85475,12386],{"class":503},[151,85477,65875],{"class":2226},[151,85479,85480],{"class":503},"(pix_list)):\n",[151,85482,85483,85485,85488,85490,85492,85494,85496],{"class":469,"line":32560},[151,85484,16616],{"class":1869},[151,85486,85487],{"class":503}," y ",[151,85489,16417],{"class":1869},[151,85491,2793],{"class":2226},[151,85493,12386],{"class":503},[151,85495,65875],{"class":2226},[151,85497,85498],{"class":503},"(pixel_colors)):\n",[151,85500,85501,85503,85506,85508,85510,85512],{"class":469,"line":32573},[151,85502,40442],{"class":1869},[151,85504,85505],{"class":503}," pix_list[x][",[151,85507,6619],{"class":477},[151,85509,16654],{"class":503},[151,85511,17223],{"class":1869},[151,85513,85514],{"class":503}," y:\n",[151,85516,85517,85520,85522],{"class":469,"line":32578},[151,85518,85519],{"class":503},"                pixel ",[151,85521,1876],{"class":1869},[151,85523,85524],{"class":503}," pixel_colors[y]\n",[151,85526,85527],{"class":469,"line":32586},[151,85528,85529],{"class":503},"                pix_series.append(pixel)\n",[151,85531,85532],{"class":469,"line":32591},[151,85533,1090],{"emptyLinePlaceholder":609},[151,85535,85536],{"class":469,"line":32597},[151,85537,85538],{"class":1527},"#runs ant along the grid according to the moves (defined above) for the number of steps in iterations (defined above)\n",[151,85540,85541,85543,85546],{"class":469,"line":32612},[151,85542,16925],{"class":12347},[151,85544,85545],{"class":473}," run",[151,85547,16931],{"class":503},[151,85549,85550],{"class":469,"line":32617},[151,85551,85552],{"class":1527},"    #variable the tracks the step number if the ant goes out of bounds\n",[151,85554,85555,85557],{"class":469,"line":32622},[151,85556,84834],{"class":1869},[151,85558,85559],{"class":503}," oob\n",[151,85561,85562],{"class":469,"line":32628},[151,85563,85564],{"class":1527},"    #converts moves string into a list of 0s and 1s; these numbers correspond to direction and are passed into the move() function\n",[151,85566,85567,85570,85572,85574,85576,85578,85580,85582,85585,85587,85589,85591,85593,85595,85597],{"class":469,"line":32634},[151,85568,85569],{"class":503},"    moves1 ",[151,85571,1876],{"class":1869},[151,85573,6604],{"class":503},[151,85575,6760],{"class":477},[151,85577,3435],{"class":1869},[151,85579,44552],{"class":503},[151,85581,17223],{"class":1869},[151,85583,85584],{"class":481}," 'R'",[151,85586,17229],{"class":1869},[151,85588,9949],{"class":1869},[151,85590,6760],{"class":477},[151,85592,2235],{"class":1869},[151,85594,44552],{"class":503},[151,85596,16417],{"class":1869},[151,85598,85599],{"class":503}," moves]\n",[151,85601,85602,85604,85606,85608,85610],{"class":469,"line":32640},[151,85603,16411],{"class":1869},[151,85605,24620],{"class":503},[151,85607,16417],{"class":1869},[151,85609,2793],{"class":2226},[151,85611,85612],{"class":503},"(iterations):\n",[151,85614,85615],{"class":469,"line":32652},[151,85616,85617],{"class":1527},"        #exit the loop if the ant reaches the edge of the grid\n",[151,85619,85620,85622,85625,85627,85630,85633,85635,85637,85639,85641,85643],{"class":469,"line":32664},[151,85621,23357],{"class":1869},[151,85623,85624],{"class":503}," ant_pos ",[151,85626,3613],{"class":1869},[151,85628,85629],{"class":503}," width ",[151,85631,85632],{"class":1869},"or",[151,85634,85624],{"class":503},[151,85636,44519],{"class":1869},[151,85638,85629],{"class":503},[151,85640,17223],{"class":1869},[151,85642,57890],{"class":477},[151,85644,14372],{"class":503},[151,85646,85647,85650,85652],{"class":469,"line":32679},[151,85648,85649],{"class":503},"            oob ",[151,85651,1876],{"class":1869},[151,85653,85654],{"class":503}," step\n",[151,85656,85657],{"class":469,"line":32691},[151,85658,85659],{"class":1869},"            return\n",[151,85661,85662],{"class":469,"line":32699},[151,85663,85664],{"class":1527},"        #loop through the moves\n",[151,85666,85667,85669,85672,85674,85676],{"class":469,"line":32704},[151,85668,16616],{"class":1869},[151,85670,85671],{"class":503}," index, direction ",[151,85673,16417],{"class":1869},[151,85675,17042],{"class":2226},[151,85677,85678],{"class":503},"(moves1):\n",[151,85680,85681,85683],{"class":469,"line":32709},[151,85682,39004],{"class":1869},[151,85684,14372],{"class":503},[151,85686,85687],{"class":469,"line":32715},[151,85688,85689],{"class":1527},"                #remember the ant position\n",[151,85691,85692,85695,85697],{"class":469,"line":32727},[151,85693,85694],{"class":503},"                not_moved ",[151,85696,1876],{"class":1869},[151,85698,84837],{"class":503},[151,85700,85701],{"class":469,"line":32738},[151,85702,85703],{"class":1527},"                #try to move the ant position\n",[151,85705,85706],{"class":469,"line":32752},[151,85707,85708],{"class":503},"                move(index,direction)\n",[151,85710,85711],{"class":469,"line":32761},[151,85712,85713],{"class":1527},"                #check to see if the position was moved\n",[151,85715,85716,85719,85721,85723],{"class":469,"line":32767},[151,85717,85718],{"class":1869},"                if",[151,85720,85624],{"class":503},[151,85722,58602],{"class":1869},[151,85724,85725],{"class":503}," not_moved:\n",[151,85727,85728],{"class":469,"line":32772},[151,85729,85730],{"class":1527},"                    #set record to false in the settings to turn of frame recording\n",[151,85732,85733,85736,85739,85741,85743],{"class":469,"line":32777},[151,85734,85735],{"class":1869},"                    if",[151,85737,85738],{"class":503}," record ",[151,85740,17223],{"class":1869},[151,85742,68564],{"class":477},[151,85744,14372],{"class":503},[151,85746,85747],{"class":469,"line":32782},[151,85748,85749],{"class":1527},"                        #records a new frame every frame_interval frame\n",[151,85751,85752,85755,85757,85759,85762,85764,85766],{"class":469,"line":32790},[151,85753,85754],{"class":1869},"                        if",[151,85756,24620],{"class":503},[151,85758,44519],{"class":1869},[151,85760,85761],{"class":503}," frame_interval ",[151,85763,17223],{"class":1869},[151,85765,57890],{"class":477},[151,85767,14372],{"class":503},[151,85769,85770,85773,85775],{"class":469,"line":32795},[151,85771,85772],{"class":503},"                            counter ",[151,85774,24780],{"class":1869},[151,85776,3181],{"class":477},[151,85778,85779,85782,85784,85787,85789,85791],{"class":469,"line":32801},[151,85780,85781],{"class":2226},"                            print",[151,85783,12386],{"class":503},[151,85785,85786],{"class":481},"\"Generating frame number \"",[151,85788,23378],{"class":1869},[151,85790,84112],{"class":6205},[151,85792,85793],{"class":503},"(counter))\n",[151,85795,85796],{"class":469,"line":32815},[151,85797,85798],{"class":503},"                            get_pix_series()\n",[151,85800,85801],{"class":469,"line":32826},[151,85802,85803],{"class":503},"                            save_image(counter)\n",[151,85805,85806],{"class":469,"line":32847},[151,85807,85808],{"class":1869},"                    break\n",[151,85810,85811,85814],{"class":469,"line":32852},[151,85812,85813],{"class":1869},"                else",[151,85815,14372],{"class":503},[151,85817,85818],{"class":469,"line":32865},[151,85819,85820],{"class":1869},"                    continue\n",[151,85822,85823,85825],{"class":469,"line":32870},[151,85824,39193],{"class":1869},[151,85826,14372],{"class":503},[151,85828,85830],{"class":469,"line":85829},194,[151,85831,85832],{"class":1527},"                #print(\"Out of bounds at step number \" + str(step))\n",[151,85834,85836,85839,85841],{"class":469,"line":85835},195,[151,85837,85838],{"class":503},"                oob ",[151,85840,1876],{"class":1869},[151,85842,85654],{"class":503},[151,85844,85846],{"class":469,"line":85845},196,[151,85847,85848],{"class":1869},"                return\n",[151,85850,85852],{"class":469,"line":85851},197,[151,85853,1090],{"emptyLinePlaceholder":609},[151,85855,85857,85859,85862,85864,85866],{"class":469,"line":85856},198,[151,85858,16925],{"class":12347},[151,85860,85861],{"class":473}," save_image",[151,85863,12386],{"class":503},[151,85865,77563],{"class":15232},[151,85867,15264],{"class":503},[151,85869,85871],{"class":469,"line":85870},199,[151,85872,85873],{"class":1527},"    #give access to the image instantiated at the beginning of the script\n",[151,85875,85877,85879],{"class":469,"line":85876},200,[151,85878,84834],{"class":1869},[151,85880,85881],{"class":503}," im1\n",[151,85883,85885],{"class":469,"line":85884},201,[151,85886,85887],{"class":1527},"    #fill blank image canvas with pix_series pixel data\n",[151,85889,85891],{"class":469,"line":85890},202,[151,85892,85893],{"class":503},"    im1.putdata(pix_series)\n",[151,85895,85897],{"class":469,"line":85896},203,[151,85898,85899],{"class":1527},"    #to rescale the image, set the scale variable in settings and call resize on im1\n",[151,85901,85903,85906,85908,85911,85913,85916,85918,85920,85922,85925,85927,85930,85933,85935],{"class":469,"line":85902},204,[151,85904,85905],{"class":503},"    im1.resize((scale",[151,85907,23268],{"class":1869},[151,85909,85910],{"class":503},"im1.size[",[151,85912,9181],{"class":477},[151,85914,85915],{"class":503},"],scale",[151,85917,23268],{"class":1869},[151,85919,85910],{"class":503},[151,85921,6760],{"class":477},[151,85923,85924],{"class":503},"])).save(",[151,85926,13223],{"class":481},[151,85928,85929],{"class":477},"%s",[151,85931,85932],{"class":481},".png'",[151,85934,71811],{"class":1869},[151,85936,85937],{"class":503}," (moves))\n",[151,85939,85941],{"class":469,"line":85940},205,[151,85942,1090],{"emptyLinePlaceholder":609},[151,85944,85946],{"class":469,"line":85945},206,[151,85947,85948],{"class":1527},"#builds a dictionary to count pixels by color\n",[151,85950,85952,85954,85957],{"class":469,"line":85951},207,[151,85953,16925],{"class":12347},[151,85955,85956],{"class":473}," build_df_row",[151,85958,16931],{"class":503},[151,85960,85962,85965,85967,85969,85971,85974,85976,85978,85981,85983,85985],{"class":469,"line":85961},208,[151,85963,85964],{"class":503},"    colors_dict ",[151,85966,1876],{"class":1869},[151,85968,52023],{"class":503},[151,85970,15343],{"class":6205},[151,85972,85973],{"class":503},"(val): ",[151,85975,9181],{"class":477},[151,85977,2235],{"class":1869},[151,85979,85980],{"class":503}," val, color ",[151,85982,16417],{"class":1869},[151,85984,17042],{"class":2226},[151,85986,85987],{"class":503},"(pixel_colors)}\n",[151,85989,85991,85994,85996,85998,86001],{"class":469,"line":85990},209,[151,85992,85993],{"class":503},"    moves_dict ",[151,85995,1876],{"class":1869},[151,85997,52023],{"class":503},[151,85999,86000],{"class":481},"'moves'",[151,86002,86003],{"class":503},":moves}\n",[151,86005,86007,86010,86012,86014,86017],{"class":469,"line":86006},210,[151,86008,86009],{"class":503},"    last_step ",[151,86011,1876],{"class":1869},[151,86013,52023],{"class":503},[151,86015,86016],{"class":481},"'last_step'",[151,86018,86019],{"class":503},":oob}\n",[151,86021,86023,86026,86028,86030,86033,86035,86038,86040],{"class":469,"line":86022},211,[151,86024,86025],{"class":503},"    row_dict ",[151,86027,1876],{"class":1869},[151,86029,51817],{"class":6205},[151,86031,86032],{"class":503},"(colors_dict.items()",[151,86034,22885],{"class":1869},[151,86036,86037],{"class":503},"moves_dict.items()",[151,86039,22885],{"class":1869},[151,86041,86042],{"class":503},"last_step.items())\n",[151,86044,86046,86048,86050,86052],{"class":469,"line":86045},212,[151,86047,16411],{"class":1869},[151,86049,44552],{"class":503},[151,86051,16417],{"class":1869},[151,86053,86054],{"class":503}," pix_list:\n",[151,86056,86058,86061,86063,86065,86068,86070],{"class":469,"line":86057},213,[151,86059,86060],{"class":503},"        pixel_color ",[151,86062,1876],{"class":1869},[151,86064,84112],{"class":6205},[151,86066,86067],{"class":503},"(x[",[151,86069,6619],{"class":477},[151,86071,38820],{"class":503},[151,86073,86075],{"class":469,"line":86074},214,[151,86076,86077],{"class":1527},"        #print(pixel_color)\n",[151,86079,86081,86084,86086],{"class":469,"line":86080},215,[151,86082,86083],{"class":503},"        row_dict[pixel_color] ",[151,86085,24780],{"class":1869},[151,86087,3181],{"class":477},[151,86089,86091,86093],{"class":469,"line":86090},216,[151,86092,17496],{"class":1869},[151,86094,86095],{"class":503}," row_dict\n",[151,86097,86099],{"class":469,"line":86098},217,[151,86100,1090],{"emptyLinePlaceholder":609},[151,86102,86104],{"class":469,"line":86103},218,[151,86105,86106],{"class":1527},"#uncomment below to overwrite moves_list\n",[151,86108,86110],{"class":469,"line":86109},219,[151,86111,86112],{"class":1527},"#moves_list = ['LR', 'RRLR']\n",[151,86114,86116],{"class":469,"line":86115},220,[151,86117,1090],{"emptyLinePlaceholder":609},[151,86119,86121,86123,86126,86128,86130],{"class":469,"line":86120},221,[151,86122,16732],{"class":1869},[151,86124,86125],{"class":503}," _, moves ",[151,86127,16417],{"class":1869},[151,86129,17042],{"class":2226},[151,86131,86132],{"class":503},"(moves_list):\n",[151,86134,86136,86139,86141],{"class":469,"line":86135},222,[151,86137,86138],{"class":503},"    oob ",[151,86140,1876],{"class":1869},[151,86142,57871],{"class":477},[151,86144,86146,86149,86151,86153],{"class":469,"line":86145},223,[151,86147,86148],{"class":503},"    dir_path ",[151,86150,1876],{"class":1869},[151,86152,84112],{"class":6205},[151,86154,86155],{"class":503},"(_)\n",[151,86157,86159],{"class":469,"line":86158},224,[151,86160,1090],{"emptyLinePlaceholder":609},[151,86162,86164],{"class":469,"line":86163},225,[151,86165,86166],{"class":1527},"    #make a new directory for each new ant walk in walks based on the the walk number and navigate to that directory\n",[151,86168,86170,86172,86174,86176,86178],{"class":469,"line":86169},226,[151,86171,23327],{"class":1869},[151,86173,85738],{"class":503},[151,86175,17223],{"class":1869},[151,86177,68564],{"class":477},[151,86179,14372],{"class":503},[151,86181,86183],{"class":469,"line":86182},227,[151,86184,86185],{"class":1527},"        #make a new directory to record frames for a give ant if record is set to true and that directory does not yet exist\n",[151,86187,86189,86191,86193],{"class":469,"line":86188},228,[151,86190,23357],{"class":1869},[151,86192,4191],{"class":1869},[151,86194,86195],{"class":503}," os.path.isdir(dir_path):\n",[151,86197,86199],{"class":469,"line":86198},229,[151,86200,86201],{"class":503},"            os.makedirs(dir_path)\n",[151,86203,86205],{"class":469,"line":86204},230,[151,86206,86207],{"class":1527},"        #otherwise just change into the directory\n",[151,86209,86211,86213],{"class":469,"line":86210},231,[151,86212,23395],{"class":1869},[151,86214,14372],{"class":503},[151,86216,86218],{"class":469,"line":86217},232,[151,86219,86220],{"class":503},"            os.chdir(dir_path)\n",[151,86222,86224],{"class":469,"line":86223},233,[151,86225,1090],{"emptyLinePlaceholder":609},[151,86227,86229],{"class":469,"line":86228},234,[151,86230,86231],{"class":1527},"    #set ant at middle of grid\n",[151,86233,86235,86237,86239,86241,86243,86245,86247,86249,86251,86253,86255,86257,86259,86261,86263],{"class":469,"line":86234},235,[151,86236,84847],{"class":503},[151,86238,1876],{"class":1869},[151,86240,16673],{"class":6205},[151,86242,84032],{"class":503},[151,86244,23268],{"class":1869},[151,86246,84037],{"class":503},[151,86248,19883],{"class":1869},[151,86250,6619],{"class":477},[151,86252,16995],{"class":503},[151,86254,22885],{"class":1869},[151,86256,16673],{"class":6205},[151,86258,84050],{"class":503},[151,86260,19883],{"class":1869},[151,86262,6619],{"class":477},[151,86264,3640],{"class":503},[151,86266,86268],{"class":469,"line":86267},236,[151,86269,1090],{"emptyLinePlaceholder":609},[151,86271,86273],{"class":469,"line":86272},237,[151,86274,86275],{"class":1527},"    #moves = len(moves)\n",[151,86277,86279,86282,86284,86287,86289],{"class":469,"line":86278},238,[151,86280,86281],{"class":503},"    pixel_colors ",[151,86283,1876],{"class":1869},[151,86285,86286],{"class":503}," color_choices[:(",[151,86288,65875],{"class":2226},[151,86290,86291],{"class":503},"(moves))]\n",[151,86293,86295],{"class":469,"line":86294},239,[151,86296,1090],{"emptyLinePlaceholder":609},[151,86298,86300],{"class":469,"line":86299},240,[151,86301,86302],{"class":1527},"    #defines an empty list of elements [x,y,0] where x amd y are the position 0 is the 0ht color in the color list (the base canvas color)\n",[151,86304,86306,86309,86311],{"class":469,"line":86305},241,[151,86307,86308],{"class":503},"    pix_list ",[151,86310,1876],{"class":1869},[151,86312,16606],{"class":503},[151,86314,86316,86318,86320,86322,86324],{"class":469,"line":86315},242,[151,86317,16411],{"class":1869},[151,86319,44552],{"class":503},[151,86321,16417],{"class":1869},[151,86323,2793],{"class":2226},[151,86325,86326],{"class":503},"(length):\n",[151,86328,86330,86332,86334,86336,86338],{"class":469,"line":86329},243,[151,86331,16616],{"class":1869},[151,86333,85487],{"class":503},[151,86335,16417],{"class":1869},[151,86337,2793],{"class":2226},[151,86339,86340],{"class":503},"(width):\n",[151,86342,86344,86347,86349,86352,86354],{"class":469,"line":86343},244,[151,86345,86346],{"class":503},"            a ",[151,86348,1876],{"class":1869},[151,86350,86351],{"class":503}," [x,y,",[151,86353,9181],{"class":477},[151,86355,3691],{"class":503},[151,86357,86359],{"class":469,"line":86358},245,[151,86360,86361],{"class":503},"            pix_list.append(a)\n",[151,86363,86365],{"class":469,"line":86364},246,[151,86366,1090],{"emptyLinePlaceholder":609},[151,86368,86370],{"class":469,"line":86369},247,[151,86371,86372],{"class":1527},"    #pix_series is a list of pixels that is passed into the put_data function to generate an image\n",[151,86374,86376,86378,86380],{"class":469,"line":86375},248,[151,86377,85459],{"class":503},[151,86379,1876],{"class":1869},[151,86381,16606],{"class":503},[151,86383,86385],{"class":469,"line":86384},249,[151,86386,1090],{"emptyLinePlaceholder":609},[151,86388,86390],{"class":469,"line":86389},250,[151,86391,86392],{"class":1527},"    #counter keeps track of the frame number (if recording a series of images)\n",[151,86394,86396,86399,86401],{"class":469,"line":86395},251,[151,86397,86398],{"class":503},"    counter ",[151,86400,1876],{"class":1869},[151,86402,57871],{"class":477},[151,86404,86406],{"class":469,"line":86405},252,[151,86407,1090],{"emptyLinePlaceholder":609},[151,86409,86411],{"class":469,"line":86410},253,[151,86412,86413],{"class":1527},"    #run the ant\n",[151,86415,86417],{"class":469,"line":86416},254,[151,86418,86419],{"class":503},"    run()\n",[151,86421,86423],{"class":469,"line":86422},255,[151,86424,1090],{"emptyLinePlaceholder":609},[151,86426,86428],{"class":469,"line":86427},256,[151,86429,86430],{"class":1527},"    #capture the final state of the grid with get_pix_series\n",[151,86432,86434],{"class":469,"line":86433},257,[151,86435,86436],{"class":503},"    get_pix_series()\n",[151,86438,86440],{"class":469,"line":86439},258,[151,86441,1090],{"emptyLinePlaceholder":609},[151,86443,86445],{"class":469,"line":86444},259,[151,86446,86447],{"class":1527},"    #uncomment below to preview images for testing\n",[151,86449,86451],{"class":469,"line":86450},260,[151,86452,86453],{"class":1527},"    #im1.putdata(pix_series)\n",[151,86455,86457],{"class":469,"line":86456},261,[151,86458,86459],{"class":1527},"    #im1.resize((scale*im1.size[0],scale*im1.size[1])).show()\n",[151,86461,86463],{"class":469,"line":86462},262,[151,86464,1090],{"emptyLinePlaceholder":609},[151,86466,86468],{"class":469,"line":86467},263,[151,86469,86470],{"class":1527},"    #build a dictionary with pixel counts\n",[151,86472,86474,86476,86478],{"class":469,"line":86473},264,[151,86475,85964],{"class":503},[151,86477,1876],{"class":1869},[151,86479,86480],{"class":503}," build_df_row()\n",[151,86482,86484,86487,86489,86492,86495,86497,86499,86501],{"class":469,"line":86483},265,[151,86485,86486],{"class":503},"    row_df ",[151,86488,1876],{"class":1869},[151,86490,86491],{"class":503}," pd.DataFrame(colors_dict, ",[151,86493,86494],{"class":15210},"index",[151,86496,1876],{"class":1869},[151,86498,6698],{"class":503},[151,86500,9181],{"class":477},[151,86502,38820],{"class":503},[151,86504,86506,86508,86510,86513,86516,86518,86520],{"class":469,"line":86505},266,[151,86507,65019],{"class":503},[151,86509,1876],{"class":1869},[151,86511,86512],{"class":503}," df.append(row_df, ",[151,86514,86515],{"class":15210},"ignore_index",[151,86517,1876],{"class":1869},[151,86519,36962],{"class":477},[151,86521,3640],{"class":503},[151,86523,86525],{"class":469,"line":86524},267,[151,86526,1090],{"emptyLinePlaceholder":609},[151,86528,86530,86532,86535,86537,86539],{"class":469,"line":86529},268,[151,86531,23327],{"class":1869},[151,86533,86534],{"class":503}," record_final_image ",[151,86536,17223],{"class":1869},[151,86538,68564],{"class":477},[151,86540,14372],{"class":503},[151,86542,86544,86547,86550],{"class":469,"line":86543},269,[151,86545,86546],{"class":503},"        os.chdir(os.path.expanduser(",[151,86548,86549],{"class":481},"'~/Documents/CA_1/imgs/'",[151,86551,12451],{"class":503},[151,86553,86555],{"class":469,"line":86554},270,[151,86556,86557],{"class":503},"        save_image(_)\n",[151,86559,86561,86564,86567],{"class":469,"line":86560},271,[151,86562,86563],{"class":503},"        os.chdir(",[151,86565,86566],{"class":481},"'../'",[151,86568,3640],{"class":503},[151,86570,86572],{"class":469,"line":86571},272,[151,86573,1090],{"emptyLinePlaceholder":609},[151,86575,86577],{"class":469,"line":86576},273,[151,86578,86579],{"class":1527},"    #summary\n",[151,86581,86583,86585,86587,86589,86592,86594,86596,86599],{"class":469,"line":86582},274,[151,86584,24285],{"class":2226},[151,86586,12386],{"class":503},[151,86588,15343],{"class":6205},[151,86590,86591],{"class":503},"(_), ",[151,86593,24306],{"class":15210},[151,86595,1876],{"class":1869},[151,86597,86598],{"class":481},"' '",[151,86600,3640],{"class":503},[151,86602,86604],{"class":469,"line":86603},275,[151,86605,1090],{"emptyLinePlaceholder":609},[151,86607,86609,86612,86615],{"class":469,"line":86608},276,[151,86610,86611],{"class":503},"os.chdir(os.path.expanduser(",[151,86613,86614],{"class":481},"'~/Documents/CA_1/'",[151,86616,12451],{"class":503},[151,86618,86620,86623,86626,86628,86630,86632,86634],{"class":469,"line":86619},277,[151,86621,86622],{"class":503},"df.to_csv(",[151,86624,86625],{"class":481},"'ants_hist_.csv'",[151,86627,106],{"class":503},[151,86629,86494],{"class":15210},[151,86631,1876],{"class":1869},[151,86633,39461],{"class":477},[151,86635,3640],{"class":503},[14063,86637,86639],{"id":86638},"clustering","Clustering",[11,86641,86642,86643,86646],{},"We now have a csv file where each row is a 16-state termite and the columns labeled 0 through 15 count the sum of pixels in each state (the different colors). With ",[30,86644,86645],{},"last_step"," we also track the last step reached in the event that the ant runs into the edge of the grid. This will be helpful in clustering ants that form highways in different groups from those that complete 100000 steps inside the 200 x 200 grid.",[11,86648,86649],{},"First let's read the csv into a pandas DataFrame and look at some of the data.",[459,86651,86652],{"className":13136,"code":83747,"language":12886,"meta":464,"style":464},[30,86653,86654,86660,86670,86676,86682,86688,86698,86708,86714,86724,86734,86744,86754],{"__ignoreMap":464},[151,86655,86656,86658],{"class":469,"line":470},[151,86657,16859],{"class":1869},[151,86659,44465],{"class":477},[151,86661,86662,86664,86666,86668],{"class":469,"line":488},[151,86663,16853],{"class":1869},[151,86665,44099],{"class":477},[151,86667,44102],{"class":1869},[151,86669,44105],{"class":503},[151,86671,86672,86674],{"class":469,"line":500},[151,86673,16859],{"class":1869},[151,86675,44034],{"class":503},[151,86677,86678,86680],{"class":469,"line":509},[151,86679,16859],{"class":1869},[151,86681,24070],{"class":503},[151,86683,86684,86686],{"class":469,"line":517},[151,86685,16859],{"class":1869},[151,86687,58351],{"class":503},[151,86689,86690,86692,86694,86696],{"class":469,"line":534},[151,86691,16859],{"class":1869},[151,86693,83790],{"class":503},[151,86695,16998],{"class":1869},[151,86697,83795],{"class":503},[151,86699,86700,86702,86704,86706],{"class":469,"line":1413},[151,86701,16859],{"class":1869},[151,86703,24412],{"class":503},[151,86705,16998],{"class":1869},[151,86707,24417],{"class":503},[151,86709,86710,86712],{"class":469,"line":1418},[151,86711,16859],{"class":1869},[151,86713,83812],{"class":503},[151,86715,86716,86718,86720,86722],{"class":469,"line":2462},[151,86717,16859],{"class":1869},[151,86719,44073],{"class":503},[151,86721,16998],{"class":1869},[151,86723,44078],{"class":503},[151,86725,86726,86728,86730,86732],{"class":469,"line":2471},[151,86727,16853],{"class":1869},[151,86729,83829],{"class":12360},[151,86731,44102],{"class":1869},[151,86733,83834],{"class":503},[151,86735,86736,86738,86740,86742],{"class":469,"line":2480},[151,86737,16853],{"class":1869},[151,86739,83841],{"class":503},[151,86741,16859],{"class":1869},[151,86743,83846],{"class":503},[151,86745,86746,86748,86750,86752],{"class":469,"line":2489},[151,86747,16859],{"class":1869},[151,86749,83853],{"class":503},[151,86751,16998],{"class":1869},[151,86753,83858],{"class":503},[151,86755,86756,86758],{"class":469,"line":2497},[151,86757,44519],{"class":1869},[151,86759,44522],{"class":503},[459,86761,86763],{"className":13136,"code":86762,"language":12886,"meta":464,"style":464},"os.chdir(os.path.expanduser('~/Documents/CA_1/'))\ndf1 = pd.read_csv('ants_hist_.csv')\n",[30,86764,86765,86773],{"__ignoreMap":464},[151,86766,86767,86769,86771],{"class":469,"line":470},[151,86768,86611],{"class":503},[151,86770,86614],{"class":481},[151,86772,12451],{"class":503},[151,86774,86775,86778,86780,86783,86785],{"class":469,"line":488},[151,86776,86777],{"class":503},"df1 ",[151,86779,1876],{"class":1869},[151,86781,86782],{"class":503}," pd.read_csv(",[151,86784,86625],{"class":481},[151,86786,3640],{"class":503},[459,86788,86790],{"className":13136,"code":86789,"language":12886,"meta":464,"style":464},"df1.shape\n",[30,86791,86792],{"__ignoreMap":464},[151,86793,86794],{"class":469,"line":470},[151,86795,86789],{"class":503},[459,86797,86800],{"className":86798,"code":86799,"language":997},[995],"(32768, 18)\n",[30,86801,86799],{"__ignoreMap":464},[459,86803,86805],{"className":13136,"code":86804,"language":12886,"meta":464,"style":464},"df1.sample(3)\n",[30,86806,86807],{"__ignoreMap":464},[151,86808,86809,86812,86814],{"class":469,"line":470},[151,86810,86811],{"class":503},"df1.sample(",[151,86813,6557],{"class":477},[151,86815,3640],{"class":503},[23950,86817,86818],{},[1131,86819,35598,86822,35598,86868],{"border":470,"className":86820},[86821],"dataframe",[1134,86823,86824,86825,35598],{},"\n    ",[1137,86826,86828,86829,86828,86831,86828,86833,86828,86835,86828,86837,86828,86839,86828,86841,86828,86843,86828,86845,86828,86847,86828,86849,86828,86851,86828,86853,86828,86855,86828,86857,86828,86859,86828,86861,86828,86863,86828,86865,86824],{"style":86827},"text-align: right;","\n      ",[1140,86830],{},[1140,86832,9181],{},[1140,86834,6760],{},[1140,86836,12423],{},[1140,86838,42377],{},[1140,86840,42360],{},[1140,86842,42327],{},[1140,86844,67140],{},[1140,86846,42310],{},[1140,86848,6619],{},[1140,86850,6557],{},[1140,86852,9187],{},[1140,86854,24380],{},[1140,86856,25038],{},[1140,86858,25043],{},[1140,86860,24369],{},[1140,86862,7918],{},[1140,86864,86645],{},[1140,86866,86867],{},"moves",[1153,86869,86824,86870,86824,86925,86824,86981,35598],{},[1137,86871,86828,86872,86828,86875,86828,86878,86828,86881,86828,86884,86828,86887,86828,86889,86828,86892,86828,86895,86828,86897,86828,86900,86828,86903,86828,86906,86828,86909,86828,86912,86828,86914,86828,86917,86828,86920,86828,86922,86824],{},[1140,86873,86874],{},"25892",[1158,86876,86877],{},"37741",[1158,86879,86880],{},"212",[1158,86882,86883],{},"135",[1158,86885,86886],{},"148",[1158,86888,86883],{},[1158,86890,86891],{},"152",[1158,86893,86894],{},"119",[1158,86896,84599],{},[1158,86898,86899],{},"176",[1158,86901,86902],{},"168",[1158,86904,86905],{},"129",[1158,86907,86908],{},"182",[1158,86910,86911],{},"155",[1158,86913,73296],{},[1158,86915,86916],{},"162",[1158,86918,86919],{},"141",[1158,86921,9181],{},[1158,86923,86924],{},"RRRLLRLRLLRLLRLL",[1137,86926,86828,86927,86828,86930,86828,86933,86828,86935,86828,86938,86828,86941,86828,86944,86828,86947,86828,86950,86828,86953,86828,86956,86828,86959,86828,86962,86828,86965,86828,86968,86828,86971,86828,86973,86828,86976,86828,86978,86824],{},[1140,86928,86929],{},"27264",[1158,86931,86932],{},"35861",[1158,86934,45374],{},[1158,86936,86937],{},"444",[1158,86939,86940],{},"421",[1158,86942,86943],{},"497",[1158,86945,86946],{},"395",[1158,86948,86949],{},"327",[1158,86951,86952],{},"222",[1158,86954,86955],{},"345",[1158,86957,86958],{},"143",[1158,86960,86961],{},"126",[1158,86963,86964],{},"218",[1158,86966,86967],{},"241",[1158,86969,86970],{},"187",[1158,86972,86883],{},[1158,86974,86975],{},"321",[1158,86977,9181],{},[1158,86979,86980],{},"RRRLRLRLRLLLLLLL",[1137,86982,86828,86983,86828,86986,86828,86989,86828,86991,86828,86994,86828,86996,86828,86998,86828,87000,86828,87003,86828,87006,86828,87009,86828,87011,86828,87014,86828,87016,86828,87018,86828,87020,86828,87022,86828,87024,86828,87026,86824],{},[1140,86984,86985],{},"10181",[1158,86987,86988],{},"37202",[1158,86990,59584],{},[1158,86992,86993],{},"177",[1158,86995,73296],{},[1158,86997,45949],{},[1158,86999,86886],{},[1158,87001,87002],{},"124",[1158,87004,87005],{},"156",[1158,87007,87008],{},"191",[1158,87010,45338],{},[1158,87012,87013],{},"234",[1158,87015,45338],{},[1158,87017,86916],{},[1158,87019,41624],{},[1158,87021,86952],{},[1158,87023,73343],{},[1158,87025,9181],{},[1158,87027,87028],{},"RLRLLRRRRRLLLRLR",[11,87030,87031,87032,87035,87036,87039],{},"There are 32768 unique instructions for 16-state termites ",[30,87033,87034],{},"(2^16)/2 = 32768",". Let's check to see how many of these are duplicates. We want to select only the state-counts and then call ",[30,87037,87038],{},".drop_duplicates"," on that DataFrame.",[459,87041,87043],{"className":13136,"code":87042,"language":12886,"meta":464,"style":464},"df2 = df1.iloc[:,0:16]\n",[30,87044,87045],{"__ignoreMap":464},[151,87046,87047,87050,87052,87055,87057,87059,87062],{"class":469,"line":470},[151,87048,87049],{"class":503},"df2 ",[151,87051,1876],{"class":1869},[151,87053,87054],{"class":503}," df1.iloc[:,",[151,87056,9181],{"class":477},[151,87058,208],{"class":503},[151,87060,87061],{"class":477},"16",[151,87063,3691],{"class":503},[459,87065,87067],{"className":13136,"code":87066,"language":12886,"meta":464,"style":464},"df2.shape[0] - df2.drop_duplicates().shape[0]\n",[30,87068,87069],{"__ignoreMap":464},[151,87070,87071,87074,87076,87078,87080,87083,87085],{"class":469,"line":470},[151,87072,87073],{"class":503},"df2.shape[",[151,87075,9181],{"class":477},[151,87077,16654],{"class":503},[151,87079,12445],{"class":1869},[151,87081,87082],{"class":503}," df2.drop_duplicates().shape[",[151,87084,9181],{"class":477},[151,87086,3691],{"class":503},[459,87088,87091],{"className":87089,"code":87090,"language":997},[995],"1566\n",[30,87092,87090],{"__ignoreMap":464},[11,87094,87095],{},"1566 of the 16-state termites. It might be helpful to remove these termites from the DataFrame before we cluster them.",[459,87097,87099],{"className":13136,"code":87098,"language":12886,"meta":464,"style":464},"unique_termites_index = df2.drop_duplicates().index\n",[30,87100,87101],{"__ignoreMap":464},[151,87102,87103,87106,87108],{"class":469,"line":470},[151,87104,87105],{"class":503},"unique_termites_index ",[151,87107,1876],{"class":1869},[151,87109,87110],{"class":503}," df2.drop_duplicates().index\n",[459,87112,87114],{"className":13136,"code":87113,"language":12886,"meta":464,"style":464},"df = df1.loc[unique_termites_index,:]\n",[30,87115,87116],{"__ignoreMap":464},[151,87117,87118,87120,87122],{"class":469,"line":470},[151,87119,70720],{"class":503},[151,87121,1876],{"class":1869},[151,87123,87124],{"class":503}," df1.loc[unique_termites_index,:]\n",[459,87126,87128],{"className":13136,"code":87127,"language":12886,"meta":464,"style":464},"df['steps_taken'] = [100000 if x==0 else x for x in df.last_step]\ndf['file_names'] = [x+'.png' for x in df.moves]\ndf['move_len'] = [len(x) for x in df.moves]\n",[30,87129,87130,87167,87194],{"__ignoreMap":464},[151,87131,87132,87134,87137,87139,87141,87143,87146,87148,87150,87152,87154,87156,87158,87160,87162,87164],{"class":469,"line":470},[151,87133,70736],{"class":503},[151,87135,87136],{"class":481},"'steps_taken'",[151,87138,16654],{"class":503},[151,87140,1876],{"class":1869},[151,87142,6604],{"class":503},[151,87144,87145],{"class":477},"100000",[151,87147,3435],{"class":1869},[151,87149,27729],{"class":503},[151,87151,17223],{"class":1869},[151,87153,9181],{"class":477},[151,87155,17229],{"class":1869},[151,87157,44552],{"class":503},[151,87159,16732],{"class":1869},[151,87161,44552],{"class":503},[151,87163,16417],{"class":1869},[151,87165,87166],{"class":503}," df.last_step]\n",[151,87168,87169,87171,87174,87176,87178,87181,87183,87185,87187,87189,87191],{"class":469,"line":488},[151,87170,70736],{"class":503},[151,87172,87173],{"class":481},"'file_names'",[151,87175,16654],{"class":503},[151,87177,1876],{"class":1869},[151,87179,87180],{"class":503}," [x",[151,87182,22885],{"class":1869},[151,87184,44573],{"class":481},[151,87186,2235],{"class":1869},[151,87188,44552],{"class":503},[151,87190,16417],{"class":1869},[151,87192,87193],{"class":503}," df.moves]\n",[151,87195,87196,87198,87201,87203,87205,87207,87209,87212,87214,87216,87218],{"class":469,"line":500},[151,87197,70736],{"class":503},[151,87199,87200],{"class":481},"'move_len'",[151,87202,16654],{"class":503},[151,87204,1876],{"class":1869},[151,87206,6604],{"class":503},[151,87208,65875],{"class":2226},[151,87210,87211],{"class":503},"(x) ",[151,87213,16732],{"class":1869},[151,87215,44552],{"class":503},[151,87217,16417],{"class":1869},[151,87219,87193],{"class":503},[459,87221,87223],{"className":13136,"code":87222,"language":12886,"meta":464,"style":464},"df.head()\n",[30,87224,87225],{"__ignoreMap":464},[151,87226,87227],{"class":469,"line":470},[151,87228,87222],{"class":503},[23950,87230,87231,87548],{},[1131,87232,35598,87234,35598,87285],{"border":470,"className":87233},[86821],[1134,87235,86824,87236,35598],{},[1137,87237,86828,87238,86828,87240,86828,87242,86828,87244,86828,87246,86828,87248,86828,87250,86828,87252,86828,87254,86828,87256,86828,87258,86828,87260,86828,87262,86828,87264,86828,87266,86828,87268,86828,87270,86828,87272,86828,87274,86828,87276,86828,87279,86828,87282,86824],{"style":86827},[1140,87239],{},[1140,87241,9181],{},[1140,87243,6760],{},[1140,87245,12423],{},[1140,87247,42377],{},[1140,87249,42360],{},[1140,87251,42327],{},[1140,87253,67140],{},[1140,87255,42310],{},[1140,87257,6619],{},[1140,87259,6557],{},[1140,87261,27455],{},[1140,87263,24380],{},[1140,87265,25038],{},[1140,87267,25043],{},[1140,87269,24369],{},[1140,87271,7918],{},[1140,87273,86645],{},[1140,87275,86867],{},[1140,87277,87278],{},"steps_taken",[1140,87280,87281],{},"file_names",[1140,87283,87284],{},"move_len",[1153,87286,86824,87287,86824,87346,86824,87395,86824,87444,86824,87499,35598],{},[1137,87288,86828,87289,86828,87291,86828,87294,86828,87297,86828,87299,86828,87301,86828,87304,86828,87307,86828,87310,86828,87313,86828,87316,86828,87319,86828,87321,86828,87324,86828,87326,86828,87328,86828,87331,86828,87334,86828,87336,86828,87339,86828,87341,86828,87344,86824],{},[1140,87290,9181],{},[1158,87292,87293],{},"36381",[1158,87295,87296],{},"584",[1158,87298,41967],{},[1158,87300,9302],{},[1158,87302,87303],{},"41",[1158,87305,87306],{},"85",[1158,87308,87309],{},"179",[1158,87311,87312],{},"174",[1158,87314,87315],{},"1864",[1158,87317,87318],{},"325",[1158,87320,27455],{},[1158,87322,87323],{},"45",[1158,87325,87323],{},[1158,87327,9302],{},[1158,87329,87330],{},"31",[1158,87332,87333],{},"37",[1158,87335,9181],{},[1158,87337,87338],{},"RLLLLLLLLLLLLLLL",[1158,87340,87145],{},[1158,87342,87343],{},"RLLLLLLLLLLLLLLL.png",[1158,87345,87061],{},[1137,87347,86828,87348,86828,87350,86828,87353,86828,87355,86828,87357,86828,87359,86828,87361,86828,87363,86828,87365,86828,87367,86828,87369,86828,87371,86828,87373,86828,87375,86828,87377,86828,87379,86828,87381,86828,87383,86828,87385,86828,87388,86828,87390,86828,87393,86824],{},[1140,87349,6760],{},[1158,87351,87352],{},"39857",[1158,87354,41885],{},[1158,87356,9187],{},[1158,87358,9181],{},[1158,87360,25038],{},[1158,87362,9181],{},[1158,87364,9181],{},[1158,87366,9187],{},[1158,87368,61454],{},[1158,87370,42293],{},[1158,87372,27455],{},[1158,87374,24369],{},[1158,87376,6557],{},[1158,87378,42377],{},[1158,87380,25038],{},[1158,87382,9187],{},[1158,87384,9181],{},[1158,87386,87387],{},"RLLLLLLLLLLLLLLR",[1158,87389,87145],{},[1158,87391,87392],{},"RLLLLLLLLLLLLLLR.png",[1158,87394,87061],{},[1137,87396,86828,87397,86828,87399,86828,87402,86828,87404,86828,87406,86828,87408,86828,87410,86828,87412,86828,87414,86828,87416,86828,87418,86828,87420,86828,87422,86828,87424,86828,87426,86828,87428,86828,87430,86828,87432,86828,87434,86828,87437,86828,87439,86828,87442,86824],{},[1140,87398,6619],{},[1158,87400,87401],{},"39804",[1158,87403,41852],{},[1158,87405,24380],{},[1158,87407,42360],{},[1158,87409,25043],{},[1158,87411,7918],{},[1158,87413,25043],{},[1158,87415,9181],{},[1158,87417,7728],{},[1158,87419,42360],{},[1158,87421,27455],{},[1158,87423,42377],{},[1158,87425,7918],{},[1158,87427,25043],{},[1158,87429,9097],{},[1158,87431,24380],{},[1158,87433,9181],{},[1158,87435,87436],{},"RLLLLLLLLLLLLLRL",[1158,87438,87145],{},[1158,87440,87441],{},"RLLLLLLLLLLLLLRL.png",[1158,87443,87061],{},[1137,87445,86828,87446,86828,87448,86828,87451,86828,87454,86828,87456,86828,87459,86828,87462,86828,87465,86828,87467,86828,87469,86828,87472,86828,87474,86828,87476,86828,87478,86828,87480,86828,87483,86828,87485,86828,87487,86828,87489,86828,87492,86828,87494,86828,87497,86824],{},[1140,87447,6557],{},[1158,87449,87450],{},"37223",[1158,87452,87453],{},"346",[1158,87455,73169],{},[1158,87457,87458],{},"97",[1158,87460,87461],{},"89",[1158,87463,87464],{},"122",[1158,87466,71821],{},[1158,87468,87002],{},[1158,87470,87471],{},"1150",[1158,87473,84590],{},[1158,87475,27455],{},[1158,87477,87306],{},[1158,87479,81272],{},[1158,87481,87482],{},"61",[1158,87484,39825],{},[1158,87486,41835],{},[1158,87488,9181],{},[1158,87490,87491],{},"RLLLLLLLLLLLLLRR",[1158,87493,87145],{},[1158,87495,87496],{},"RLLLLLLLLLLLLLRR.png",[1158,87498,87061],{},[1137,87500,86828,87501,86828,87503,86828,87506,86828,87508,86828,87510,86828,87512,86828,87514,86828,87516,86828,87518,86828,87520,86828,87522,86828,87524,86828,87526,86828,87528,86828,87530,86828,87532,86828,87534,86828,87536,86828,87538,86828,87541,86828,87543,86828,87546,86824],{},[1140,87502,9187],{},[1158,87504,87505],{},"39678",[1158,87507,58546],{},[1158,87509,42115],{},[1158,87511,87061],{},[1158,87513,42293],{},[1158,87515,24369],{},[1158,87517,6760],{},[1158,87519,6619],{},[1158,87521,7691],{},[1158,87523,42293],{},[1158,87525,27455],{},[1158,87527,87330],{},[1158,87529,67140],{},[1158,87531,42310],{},[1158,87533,6596],{},[1158,87535,42377],{},[1158,87537,9181],{},[1158,87539,87540],{},"RLLLLLLLLLLLLRLL",[1158,87542,87145],{},[1158,87544,87545],{},"RLLLLLLLLLLLLRLL.png",[1158,87547,87061],{},[11,87549,87550],{},"5 rows × 21 columns",[459,87552,87554],{"className":13136,"code":87553,"language":12886,"meta":464,"style":464},"df.index = df.file_names\n",[30,87555,87556],{"__ignoreMap":464},[151,87557,87558,87561,87563],{"class":469,"line":470},[151,87559,87560],{"class":503},"df.index ",[151,87562,1876],{"class":1869},[151,87564,87565],{"class":503}," df.file_names\n",[11,87567,87568],{},"Here's a quick look at the distribution of the base canvas color (red in the images below) over all of the unique termites.",[459,87570,87572],{"className":13136,"code":87571,"language":12886,"meta":464,"style":464},"x = '0'\nsns.set_style('whitegrid')\nplt.figure(figsize=(12,4))\ndf[(df[x]>0)][x].hist(bins=250)\nplt.xlabel('Count of Cells in state 0')\nplt.ylabel('Count')\nplt.title('Histogram Showing Termite Count by number of cells in state 0')\n",[30,87573,87574,87584,87594,87612,87633,87642,87651],{"__ignoreMap":464},[151,87575,87576,87579,87581],{"class":469,"line":470},[151,87577,87578],{"class":503},"x ",[151,87580,1876],{"class":1869},[151,87582,87583],{"class":481}," '0'\n",[151,87585,87586,87589,87592],{"class":469,"line":488},[151,87587,87588],{"class":503},"sns.set_style(",[151,87590,87591],{"class":481},"'whitegrid'",[151,87593,3640],{"class":503},[151,87595,87596,87598,87600,87602,87604,87606,87608,87610],{"class":469,"line":500},[151,87597,44355],{"class":503},[151,87599,44358],{"class":15210},[151,87601,1876],{"class":1869},[151,87603,12386],{"class":503},[151,87605,42360],{"class":477},[151,87607,3634],{"class":503},[151,87609,9187],{"class":477},[151,87611,12451],{"class":503},[151,87613,87614,87617,87619,87621,87624,87627,87629,87631],{"class":469,"line":509},[151,87615,87616],{"class":503},"df[(df[x]",[151,87618,3663],{"class":1869},[151,87620,9181],{"class":477},[151,87622,87623],{"class":503},")][x].hist(",[151,87625,87626],{"class":15210},"bins",[151,87628,1876],{"class":1869},[151,87630,73447],{"class":477},[151,87632,3640],{"class":503},[151,87634,87635,87637,87640],{"class":469,"line":517},[151,87636,65133],{"class":503},[151,87638,87639],{"class":481},"'Count of Cells in state 0'",[151,87641,3640],{"class":503},[151,87643,87644,87646,87649],{"class":469,"line":534},[151,87645,65143],{"class":503},[151,87647,87648],{"class":481},"'Count'",[151,87650,3640],{"class":503},[151,87652,87653,87655,87658],{"class":469,"line":1413},[151,87654,65123],{"class":503},[151,87656,87657],{"class":481},"'Histogram Showing Termite Count by number of cells in state 0'",[151,87659,3640],{"class":503},[459,87661,87664],{"className":87662,"code":87663,"language":997},[995],"\u003Cmatplotlib.text.Text at 0x116dee10>\n",[30,87665,87663],{"__ignoreMap":464},[11,87667,87668],{},[2718,87669],{"alt":20386,"src":87670},"/static/ants_files/ants_20_1.png",[459,87672,87674],{"className":13136,"code":87673,"language":12886,"meta":464,"style":464},"df.shape\n",[30,87675,87676],{"__ignoreMap":464},[151,87677,87678],{"class":469,"line":470},[151,87679,87673],{"class":503},[459,87681,87684],{"className":87682,"code":87683,"language":997},[995],"(31202, 21)\n",[30,87685,87683],{"__ignoreMap":464},[11,87687,87688],{},"Now we can prepare a DataFrame that we will feed in to the clustering model. We will take only the pixel counts and the total number of steps taken.",[459,87690,87692],{"className":13136,"code":87691,"language":12886,"meta":464,"style":464},"X = df[df.move_len==16].iloc[:,[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,18]]\n",[30,87693,87694],{"__ignoreMap":464},[151,87695,87696,87699,87701,87704,87706,87708,87711,87713,87715,87717,87719,87721,87723,87725,87727,87729,87731,87733,87735,87737,87739,87741,87743,87745,87747,87749,87751,87753,87755,87757,87759,87761,87763,87765,87767,87769,87771,87773,87775,87777],{"class":469,"line":470},[151,87697,87698],{"class":503},"X ",[151,87700,1876],{"class":1869},[151,87702,87703],{"class":503}," df[df.move_len",[151,87705,17223],{"class":1869},[151,87707,87061],{"class":477},[151,87709,87710],{"class":503},"].iloc[:,[",[151,87712,9181],{"class":477},[151,87714,3634],{"class":503},[151,87716,6760],{"class":477},[151,87718,3634],{"class":503},[151,87720,6619],{"class":477},[151,87722,3634],{"class":503},[151,87724,6557],{"class":477},[151,87726,3634],{"class":503},[151,87728,9187],{"class":477},[151,87730,3634],{"class":503},[151,87732,24380],{"class":477},[151,87734,3634],{"class":503},[151,87736,25038],{"class":477},[151,87738,3634],{"class":503},[151,87740,25043],{"class":477},[151,87742,3634],{"class":503},[151,87744,24369],{"class":477},[151,87746,3634],{"class":503},[151,87748,7918],{"class":477},[151,87750,3634],{"class":503},[151,87752,12423],{"class":477},[151,87754,3634],{"class":503},[151,87756,42377],{"class":477},[151,87758,3634],{"class":503},[151,87760,42360],{"class":477},[151,87762,3634],{"class":503},[151,87764,42327],{"class":477},[151,87766,3634],{"class":503},[151,87768,67140],{"class":477},[151,87770,3634],{"class":503},[151,87772,42310],{"class":477},[151,87774,3634],{"class":503},[151,87776,7696],{"class":477},[151,87778,87779],{"class":503},"]]\n",[459,87781,87783],{"className":13136,"code":87782,"language":12886,"meta":464,"style":464},"X.columns\n",[30,87784,87785],{"__ignoreMap":464},[151,87786,87787],{"class":469,"line":470},[151,87788,87782],{"class":503},[459,87790,87793],{"className":87791,"code":87792,"language":997},[995],"Index([u'0', u'1', u'10', u'11', u'12', u'13', u'14', u'15', u'2', u'3', u'4',\n       u'5', u'6', u'7', u'8', u'9', u'steps_taken'],\n      dtype='object')\n",[30,87794,87792],{"__ignoreMap":464},[11,87796,87797],{},"Most of the termites completed all 100000 steps within the grid boundries.",[459,87799,87801],{"className":13136,"code":87800,"language":12886,"meta":464,"style":464},"X[X.steps_taken==100000].steps_taken.count()\n",[30,87802,87803],{"__ignoreMap":464},[151,87804,87805,87808,87810,87812],{"class":469,"line":470},[151,87806,87807],{"class":503},"X[X.steps_taken",[151,87809,17223],{"class":1869},[151,87811,87145],{"class":477},[151,87813,87814],{"class":503},"].steps_taken.count()\n",[459,87816,87819],{"className":87817,"code":87818,"language":997},[995],"29955\n",[30,87820,87818],{"__ignoreMap":464},[11,87822,87823],{},"Here'a a histogram of the steps taken by ants that took less than 100000 steps.",[459,87825,87827],{"className":13136,"code":87826,"language":12886,"meta":464,"style":464},"X[X.steps_taken\u003C100000].steps_taken.hist()\nplt.title('Histogram of steps_taken for termites that stayed in bounds')\nplt.xlabel('steps_taken')\nplt.ylabel('Count')\n",[30,87828,87829,87840,87849,87857],{"__ignoreMap":464},[151,87830,87831,87833,87835,87837],{"class":469,"line":470},[151,87832,87807],{"class":503},[151,87834,3613],{"class":1869},[151,87836,87145],{"class":477},[151,87838,87839],{"class":503},"].steps_taken.hist()\n",[151,87841,87842,87844,87847],{"class":469,"line":488},[151,87843,65123],{"class":503},[151,87845,87846],{"class":481},"'Histogram of steps_taken for termites that stayed in bounds'",[151,87848,3640],{"class":503},[151,87850,87851,87853,87855],{"class":469,"line":500},[151,87852,65133],{"class":503},[151,87854,87136],{"class":481},[151,87856,3640],{"class":503},[151,87858,87859,87861,87863],{"class":469,"line":509},[151,87860,65143],{"class":503},[151,87862,87648],{"class":481},[151,87864,3640],{"class":503},[459,87866,87869],{"className":87867,"code":87868,"language":997},[995],"\u003Cmatplotlib.text.Text at 0x18aa3a58>\n",[30,87870,87868],{"__ignoreMap":464},[11,87872,87873],{},[2718,87874],{"alt":20386,"src":87875},"/static/ants_files/ants_28_1.png",[11,87877,87878],{},"Here's another look at the distribution of cells in state 3 over all termites:",[459,87880,87882],{"className":13136,"code":87881,"language":12886,"meta":464,"style":464},"x = '3' #other intereting states: 1, 7, 11, 15\nsns.set_style('whitegrid')\n#plt.figure(figsize=(12,8))\ndf[(df[x]>0)][x].hist(bins=100)\nplt.xlabel('Count of Cells in state 3')\nplt.ylabel('Count (termites)')\nplt.title('Histogram Showing Termite Count by number of cells in state 3')\n",[30,87883,87884,87896,87904,87909,87927,87936,87945],{"__ignoreMap":464},[151,87885,87886,87888,87890,87893],{"class":469,"line":470},[151,87887,87578],{"class":503},[151,87889,1876],{"class":1869},[151,87891,87892],{"class":481}," '3'",[151,87894,87895],{"class":1527}," #other intereting states: 1, 7, 11, 15\n",[151,87897,87898,87900,87902],{"class":469,"line":488},[151,87899,87588],{"class":503},[151,87901,87591],{"class":481},[151,87903,3640],{"class":503},[151,87905,87906],{"class":469,"line":500},[151,87907,87908],{"class":1527},"#plt.figure(figsize=(12,8))\n",[151,87910,87911,87913,87915,87917,87919,87921,87923,87925],{"class":469,"line":509},[151,87912,87616],{"class":503},[151,87914,3663],{"class":1869},[151,87916,9181],{"class":477},[151,87918,87623],{"class":503},[151,87920,87626],{"class":15210},[151,87922,1876],{"class":1869},[151,87924,71821],{"class":477},[151,87926,3640],{"class":503},[151,87928,87929,87931,87934],{"class":469,"line":517},[151,87930,65133],{"class":503},[151,87932,87933],{"class":481},"'Count of Cells in state 3'",[151,87935,3640],{"class":503},[151,87937,87938,87940,87943],{"class":469,"line":534},[151,87939,65143],{"class":503},[151,87941,87942],{"class":481},"'Count (termites)'",[151,87944,3640],{"class":503},[151,87946,87947,87949,87952],{"class":469,"line":1413},[151,87948,65123],{"class":503},[151,87950,87951],{"class":481},"'Histogram Showing Termite Count by number of cells in state 3'",[151,87953,3640],{"class":503},[459,87955,87958],{"className":87956,"code":87957,"language":997},[995],"\u003Cmatplotlib.text.Text at 0x15ba6630>\n",[30,87959,87957],{"__ignoreMap":464},[11,87961,87962],{},[2718,87963],{"alt":20386,"src":87964},"/static/ants_files/ants_30_1.png",[459,87966,87968],{"className":13136,"code":87967,"language":12886,"meta":464,"style":464},"X.shape\n",[30,87969,87970],{"__ignoreMap":464},[151,87971,87972],{"class":469,"line":470},[151,87973,87967],{"class":503},[459,87975,87978],{"className":87976,"code":87977,"language":997},[995],"(31202, 17)\n",[30,87979,87977],{"__ignoreMap":464},[11,87981,87982,87983,87986,87987,87992],{},"To cluster the different termites, we can use an unsupervised learning method called clustering. It is \"unsupervised\" because I don't explicitly tell the model what types termites should be grouped together. Instead, we will tell the model ",[51,87984,87985],{},"how many different clusters there are."," Of course, I really don't know how many clusters there should be. I do know from looking at the results that there seem to be many different types of behavior, patterns, sizes and other characteristics. We significantly reduce the complexity of clustering task by training the model on the count of pixels by what state the are in. I'm sure that the model won't be able to pick up on all of the nuances that humans can detect by looking at the images, but I have a feeling that it should be able to do a fairly good job. After we take a look at the individual clusters, we can try to find an optimal number of clusters by minimizing the total number of outliers of all the clusters. Here's ",[20,87988,87991],{"href":87989,"rel":87990},"http://papers.nips.cc/paper/5306-on-integrated-clustering-and-outlier-detection.pdf",[24],"an interesting paper"," on integrated clustering and outlier detection.",[11,87994,87995],{},"Here's how we set up the clustering model. For the numebr of clusters, let's start with 75.",[459,87997,87999],{"className":13136,"code":87998,"language":12886,"meta":464,"style":464},"k_means = cluster.KMeans(n_clusters=75, random_state=1)\n",[30,88000,88001],{"__ignoreMap":464},[151,88002,88003,88006,88008,88011,88014,88016,88019,88021,88023,88025,88027],{"class":469,"line":470},[151,88004,88005],{"class":503},"k_means ",[151,88007,1876],{"class":1869},[151,88009,88010],{"class":503}," cluster.KMeans(",[151,88012,88013],{"class":15210},"n_clusters",[151,88015,1876],{"class":1869},[151,88017,88018],{"class":477},"75",[151,88020,106],{"class":503},[151,88022,71775],{"class":15210},[151,88024,1876],{"class":1869},[151,88026,6760],{"class":477},[151,88028,3640],{"class":503},[459,88030,88032],{"className":13136,"code":88031,"language":12886,"meta":464,"style":464},"k_means.fit(X, y=None)\n",[30,88033,88034],{"__ignoreMap":464},[151,88035,88036,88039,88041,88043,88045],{"class":469,"line":470},[151,88037,88038],{"class":503},"k_means.fit(X, ",[151,88040,25286],{"class":15210},[151,88042,1876],{"class":1869},[151,88044,15437],{"class":477},[151,88046,3640],{"class":503},[459,88048,88051],{"className":88049,"code":88050,"language":997},[995],"KMeans(algorithm='auto', copy_x=True, init='k-means++', max_iter=300,\n    n_clusters=75, n_init=10, n_jobs=1, precompute_distances='auto',\n    random_state=1, tol=0.0001, verbose=0)\n",[30,88052,88050],{"__ignoreMap":464},[11,88054,88055],{},"Then we add the cluster number to each termite:",[459,88057,88059],{"className":13136,"code":88058,"language":12886,"meta":464,"style":464},"X['clusters'] = k_means.labels_\n",[30,88060,88061],{"__ignoreMap":464},[151,88062,88063,88066,88069,88071,88073],{"class":469,"line":470},[151,88064,88065],{"class":503},"X[",[151,88067,88068],{"class":481},"'clusters'",[151,88070,16654],{"class":503},[151,88072,1876],{"class":1869},[151,88074,88075],{"class":503}," k_means.labels_\n",[11,88077,88078],{},"Here's the breakdown of clusters by number of termites in each cluster:",[459,88080,88082],{"className":13136,"code":88081,"language":12886,"meta":464,"style":464},"plt.figure(figsize=(12,4))\nplt.bar(X.clusters.value_counts().index, X.clusters.value_counts())\nplt.xlabel('Cluster Number')\nplt.ylabel('Count')\nplt.title('Termite count by cluster')\n",[30,88083,88084,88102,88107,88116,88124],{"__ignoreMap":464},[151,88085,88086,88088,88090,88092,88094,88096,88098,88100],{"class":469,"line":470},[151,88087,44355],{"class":503},[151,88089,44358],{"class":15210},[151,88091,1876],{"class":1869},[151,88093,12386],{"class":503},[151,88095,42360],{"class":477},[151,88097,3634],{"class":503},[151,88099,9187],{"class":477},[151,88101,12451],{"class":503},[151,88103,88104],{"class":469,"line":488},[151,88105,88106],{"class":503},"plt.bar(X.clusters.value_counts().index, X.clusters.value_counts())\n",[151,88108,88109,88111,88114],{"class":469,"line":500},[151,88110,65133],{"class":503},[151,88112,88113],{"class":481},"'Cluster Number'",[151,88115,3640],{"class":503},[151,88117,88118,88120,88122],{"class":469,"line":509},[151,88119,65143],{"class":503},[151,88121,87648],{"class":481},[151,88123,3640],{"class":503},[151,88125,88126,88128,88131],{"class":469,"line":517},[151,88127,65123],{"class":503},[151,88129,88130],{"class":481},"'Termite count by cluster'",[151,88132,3640],{"class":503},[459,88134,88137],{"className":88135,"code":88136,"language":997},[995],"\u003Cmatplotlib.text.Text at 0x15c4c358>\n",[30,88138,88136],{"__ignoreMap":464},[11,88140,88141],{},[2718,88142],{"alt":20386,"src":88143},"/static/ants_files/ants_38_1.png",[11,88145,88146],{},"And here is a list of the data shown above:",[459,88148,88150],{"className":13136,"code":88149,"language":12886,"meta":464,"style":464},"for x, y in zip(X.clusters.value_counts().index, X.clusters.value_counts()): print(' || cluster_num: ' + str(x) , 'count: ' + str(y), end='  ')\n",[30,88151,88152],{"__ignoreMap":464},[151,88153,88154,88156,88159,88161,88163,88166,88168,88170,88173,88175,88177,88180,88183,88185,88187,88190,88192,88194,88197],{"class":469,"line":470},[151,88155,16732],{"class":1869},[151,88157,88158],{"class":503}," x, y ",[151,88160,16417],{"class":1869},[151,88162,44908],{"class":2226},[151,88164,88165],{"class":503},"(X.clusters.value_counts().index, X.clusters.value_counts()): ",[151,88167,18513],{"class":2226},[151,88169,12386],{"class":503},[151,88171,88172],{"class":481},"' || cluster_num: '",[151,88174,23378],{"class":1869},[151,88176,84112],{"class":6205},[151,88178,88179],{"class":503},"(x) , ",[151,88181,88182],{"class":481},"'count: '",[151,88184,23378],{"class":1869},[151,88186,84112],{"class":6205},[151,88188,88189],{"class":503},"(y), ",[151,88191,24306],{"class":15210},[151,88193,1876],{"class":1869},[151,88195,88196],{"class":481},"'  '",[151,88198,3640],{"class":503},[459,88200,88203],{"className":88201,"code":88202,"language":997},[995]," || cluster_num: 16 count: 3293   || cluster_num: 71 count: 3036   || cluster_num: 46 count: 2993   || cluster_num: 0 count: 2752   || cluster_num: 23 count: 2667   || cluster_num: 52 count: 2540   || cluster_num: 50 count: 2185   || cluster_num: 5 count: 1412   || cluster_num: 19 count: 987   || cluster_num: 45 count: 816   || cluster_num: 44 count: 791   || cluster_num: 54 count: 686   || cluster_num: 42 count: 625   || cluster_num: 64 count: 603   || cluster_num: 21 count: 534   || cluster_num: 37 count: 518   || cluster_num: 38 count: 345   || cluster_num: 73 count: 315   || cluster_num: 57 count: 314   || cluster_num: 4 count: 302   || cluster_num: 22 count: 291   || cluster_num: 70 count: 285   || cluster_num: 8 count: 225   || cluster_num: 2 count: 224   || cluster_num: 9 count: 218   || cluster_num: 26 count: 182   || cluster_num: 49 count: 178   || cluster_num: 65 count: 138   || cluster_num: 29 count: 137   || cluster_num: 63 count: 123   || cluster_num: 25 count: 122   || cluster_num: 33 count: 105   || cluster_num: 1 count: 74   || cluster_num: 66 count: 71   || cluster_num: 27 count: 70   || cluster_num: 62 count: 66   || cluster_num: 69 count: 58   || cluster_num: 35 count: 56   || cluster_num: 11 count: 55   || cluster_num: 68 count: 54   || cluster_num: 14 count: 51   || cluster_num: 32 count: 49   || cluster_num: 30 count: 45   || cluster_num: 55 count: 38   || cluster_num: 6 count: 35   || cluster_num: 13 count: 34   || cluster_num: 43 count: 30   || cluster_num: 60 count: 30   || cluster_num: 58 count: 30   || cluster_num: 24 count: 29   || cluster_num: 39 count: 26   || cluster_num: 51 count: 25   || cluster_num: 3 count: 24   || cluster_num: 28 count: 24   || cluster_num: 15 count: 23   || cluster_num: 72 count: 23   || cluster_num: 17 count: 22   || cluster_num: 10 count: 22   || cluster_num: 34 count: 21   || cluster_num: 74 count: 20   || cluster_num: 36 count: 20   || cluster_num: 61 count: 19   || cluster_num: 31 count: 17   || cluster_num: 20 count: 13   || cluster_num: 18 count: 12   || cluster_num: 67 count: 12   || cluster_num: 12 count: 11   || cluster_num: 47 count: 11   || cluster_num: 59 count: 10   || cluster_num: 53 count: 8   || cluster_num: 7 count: 6   || cluster_num: 41 count: 6   || cluster_num: 40 count: 6   || cluster_num: 48 count: 3   || cluster_num: 56 count: 1\n",[30,88204,88202],{"__ignoreMap":464},[459,88206,88208],{"className":13136,"code":88207,"language":12886,"meta":464,"style":464},"cluster_dict = {x: y for x, y in zip(X.clusters.value_counts().index, X.clusters.value_counts())}\n",[30,88209,88210],{"__ignoreMap":464},[151,88211,88212,88215,88217,88220,88222,88224,88226,88228],{"class":469,"line":470},[151,88213,88214],{"class":503},"cluster_dict ",[151,88216,1876],{"class":1869},[151,88218,88219],{"class":503}," {x: y ",[151,88221,16732],{"class":1869},[151,88223,88158],{"class":503},[151,88225,16417],{"class":1869},[151,88227,44908],{"class":2226},[151,88229,88230],{"class":503},"(X.clusters.value_counts().index, X.clusters.value_counts())}\n",[11,88232,88233],{},"Now let's have a look at some of the termite clusters. We can use matplotlib and PIL to display multiple images using subplots.",[459,88235,88237],{"className":13136,"code":88236,"language":12886,"meta":464,"style":464},"#variables to manage the arrangement and spacing of cluster images\ntotal = 0\nrows = 0\nim_length = 0\n\n#set variables for cluster images based on cluster size\ndef set_spacing(files):\n    global total\n    global rows\n    global im_length\n    #columns = 6\n    total = len(files)\n    extras = len(files) % 6\n    if extras > 0:\n        total += (6 - extras)\n    rows = total/6.\n    im_length = rows*(20/9.)\n\n#use matplotlib to show images loaded with PIL\ndef show_images(cluster_num, samples = 0,  files_bool=False, files=None):\n    if files_bool==True:\n        files1 = np.random.choice(files.index, min(files.shape[0], samples), replace=False)\n    if (samples == 0) & (files_bool==False):\n        files1 = X[X.clusters==cluster_num].index\n    if (samples > 0) & (files_bool==False):\n        files1 = np.random.choice(X[X.clusters==cluster_num].index, min(cluster_dict[cluster_num],samples), replace=False)\n    set_spacing(files1)\n    plt.figure(figsize = (14,im_length))\n    os.chdir(os.path.expanduser('~/Documents/CA_1/imgs/'))\n    for num, x in enumerate(files1):\n        img = PIL.Image.open(x)\n        plt.subplot(rows,6,num+1)\n        plt.title(x.split('.')[0])\n        plt.axis('off')\n        plt.imshow(img)\n    print('Cluster #' + str(X.ix[x].clusters) + ' -- Cluster Total: ' + str(cluster_dict[X.ix[x].clusters]))\n",[30,88238,88239,88244,88253,88262,88271,88275,88280,88294,88301,88308,88315,88320,88332,88349,88362,88378,88393,88415,88419,88424,88464,88477,88507,88531,88545,88567,88594,88599,88615,88624,88638,88650,88666,88680,88689,88694],{"__ignoreMap":464},[151,88240,88241],{"class":469,"line":470},[151,88242,88243],{"class":1527},"#variables to manage the arrangement and spacing of cluster images\n",[151,88245,88246,88249,88251],{"class":469,"line":488},[151,88247,88248],{"class":503},"total ",[151,88250,1876],{"class":1869},[151,88252,57871],{"class":477},[151,88254,88255,88258,88260],{"class":469,"line":500},[151,88256,88257],{"class":503},"rows ",[151,88259,1876],{"class":1869},[151,88261,57871],{"class":477},[151,88263,88264,88267,88269],{"class":469,"line":509},[151,88265,88266],{"class":503},"im_length ",[151,88268,1876],{"class":1869},[151,88270,57871],{"class":477},[151,88272,88273],{"class":469,"line":517},[151,88274,1090],{"emptyLinePlaceholder":609},[151,88276,88277],{"class":469,"line":534},[151,88278,88279],{"class":1527},"#set variables for cluster images based on cluster size\n",[151,88281,88282,88284,88287,88289,88292],{"class":469,"line":1413},[151,88283,16925],{"class":12347},[151,88285,88286],{"class":473}," set_spacing",[151,88288,12386],{"class":503},[151,88290,88291],{"class":15232},"files",[151,88293,15264],{"class":503},[151,88295,88296,88298],{"class":469,"line":1418},[151,88297,84834],{"class":1869},[151,88299,88300],{"class":503}," total\n",[151,88302,88303,88305],{"class":469,"line":2462},[151,88304,84834],{"class":1869},[151,88306,88307],{"class":503}," rows\n",[151,88309,88310,88312],{"class":469,"line":2471},[151,88311,84834],{"class":1869},[151,88313,88314],{"class":503}," im_length\n",[151,88316,88317],{"class":469,"line":2480},[151,88318,88319],{"class":1527},"    #columns = 6\n",[151,88321,88322,88325,88327,88329],{"class":469,"line":2489},[151,88323,88324],{"class":503},"    total ",[151,88326,1876],{"class":1869},[151,88328,45035],{"class":2226},[151,88330,88331],{"class":503},"(files)\n",[151,88333,88334,88337,88339,88341,88344,88346],{"class":469,"line":2497},[151,88335,88336],{"class":503},"    extras ",[151,88338,1876],{"class":1869},[151,88340,45035],{"class":2226},[151,88342,88343],{"class":503},"(files) ",[151,88345,44519],{"class":1869},[151,88347,88348],{"class":477}," 6\n",[151,88350,88351,88353,88356,88358,88360],{"class":469,"line":3140},[151,88352,23327],{"class":1869},[151,88354,88355],{"class":503}," extras ",[151,88357,3663],{"class":1869},[151,88359,57890],{"class":477},[151,88361,14372],{"class":503},[151,88363,88364,88367,88369,88371,88373,88375],{"class":469,"line":3149},[151,88365,88366],{"class":503},"        total ",[151,88368,24780],{"class":1869},[151,88370,129],{"class":503},[151,88372,25038],{"class":477},[151,88374,9949],{"class":1869},[151,88376,88377],{"class":503}," extras)\n",[151,88379,88380,88382,88384,88387,88389,88391],{"class":469,"line":3158},[151,88381,64811],{"class":503},[151,88383,1876],{"class":1869},[151,88385,88386],{"class":503}," total",[151,88388,19883],{"class":1869},[151,88390,25038],{"class":477},[151,88392,6565],{"class":503},[151,88394,88395,88398,88400,88402,88404,88406,88408,88410,88412],{"class":469,"line":3167},[151,88396,88397],{"class":503},"    im_length ",[151,88399,1876],{"class":1869},[151,88401,78206],{"class":503},[151,88403,23268],{"class":1869},[151,88405,12386],{"class":503},[151,88407,9097],{"class":477},[151,88409,19883],{"class":1869},[151,88411,7918],{"class":477},[151,88413,88414],{"class":503},".)\n",[151,88416,88417],{"class":469,"line":3175},[151,88418,1090],{"emptyLinePlaceholder":609},[151,88420,88421],{"class":469,"line":3184},[151,88422,88423],{"class":1527},"#use matplotlib to show images loaded with PIL\n",[151,88425,88426,88428,88431,88433,88436,88438,88441,88443,88445,88447,88450,88452,88454,88456,88458,88460,88462],{"class":469,"line":3193},[151,88427,16925],{"class":12347},[151,88429,88430],{"class":473}," show_images",[151,88432,12386],{"class":503},[151,88434,88435],{"class":15232},"cluster_num",[151,88437,106],{"class":503},[151,88439,88440],{"class":15232},"samples",[151,88442,19865],{"class":1869},[151,88444,57890],{"class":477},[151,88446,25131],{"class":503},[151,88448,88449],{"class":15232},"files_bool",[151,88451,1876],{"class":1869},[151,88453,39461],{"class":477},[151,88455,106],{"class":503},[151,88457,88291],{"class":15232},[151,88459,1876],{"class":1869},[151,88461,15437],{"class":477},[151,88463,15264],{"class":503},[151,88465,88466,88468,88471,88473,88475],{"class":469,"line":3720},[151,88467,23327],{"class":1869},[151,88469,88470],{"class":503}," files_bool",[151,88472,17223],{"class":1869},[151,88474,36962],{"class":477},[151,88476,14372],{"class":503},[151,88478,88479,88482,88484,88487,88490,88493,88495,88498,88501,88503,88505],{"class":469,"line":3729},[151,88480,88481],{"class":503},"        files1 ",[151,88483,1876],{"class":1869},[151,88485,88486],{"class":503}," np.random.choice(files.index, ",[151,88488,88489],{"class":2226},"min",[151,88491,88492],{"class":503},"(files.shape[",[151,88494,9181],{"class":477},[151,88496,88497],{"class":503},"], samples), ",[151,88499,88500],{"class":15210},"replace",[151,88502,1876],{"class":1869},[151,88504,39461],{"class":477},[151,88506,3640],{"class":503},[151,88508,88509,88511,88514,88516,88518,88520,88522,88525,88527,88529],{"class":469,"line":3735},[151,88510,23327],{"class":1869},[151,88512,88513],{"class":503}," (samples ",[151,88515,17223],{"class":1869},[151,88517,57890],{"class":477},[151,88519,16995],{"class":503},[151,88521,54214],{"class":1869},[151,88523,88524],{"class":503}," (files_bool",[151,88526,17223],{"class":1869},[151,88528,39461],{"class":477},[151,88530,15264],{"class":503},[151,88532,88533,88535,88537,88540,88542],{"class":469,"line":3745},[151,88534,88481],{"class":503},[151,88536,1876],{"class":1869},[151,88538,88539],{"class":503}," X[X.clusters",[151,88541,17223],{"class":1869},[151,88543,88544],{"class":503},"cluster_num].index\n",[151,88546,88547,88549,88551,88553,88555,88557,88559,88561,88563,88565],{"class":469,"line":3754},[151,88548,23327],{"class":1869},[151,88550,88513],{"class":503},[151,88552,3663],{"class":1869},[151,88554,57890],{"class":477},[151,88556,16995],{"class":503},[151,88558,54214],{"class":1869},[151,88560,88524],{"class":503},[151,88562,17223],{"class":1869},[151,88564,39461],{"class":477},[151,88566,15264],{"class":503},[151,88568,88569,88571,88573,88576,88578,88581,88583,88586,88588,88590,88592],{"class":469,"line":3760},[151,88570,88481],{"class":503},[151,88572,1876],{"class":1869},[151,88574,88575],{"class":503}," np.random.choice(X[X.clusters",[151,88577,17223],{"class":1869},[151,88579,88580],{"class":503},"cluster_num].index, ",[151,88582,88489],{"class":2226},[151,88584,88585],{"class":503},"(cluster_dict[cluster_num],samples), ",[151,88587,88500],{"class":15210},[151,88589,1876],{"class":1869},[151,88591,39461],{"class":477},[151,88593,3640],{"class":503},[151,88595,88596],{"class":469,"line":3773},[151,88597,88598],{"class":503},"    set_spacing(files1)\n",[151,88600,88601,88604,88606,88608,88610,88612],{"class":469,"line":3782},[151,88602,88603],{"class":503},"    plt.figure(",[151,88605,44358],{"class":15210},[151,88607,19865],{"class":1869},[151,88609,129],{"class":503},[151,88611,67140],{"class":477},[151,88613,88614],{"class":503},",im_length))\n",[151,88616,88617,88620,88622],{"class":469,"line":3791},[151,88618,88619],{"class":503},"    os.chdir(os.path.expanduser(",[151,88621,86549],{"class":481},[151,88623,12451],{"class":503},[151,88625,88626,88628,88631,88633,88635],{"class":469,"line":3803},[151,88627,16411],{"class":1869},[151,88629,88630],{"class":503}," num, x ",[151,88632,16417],{"class":1869},[151,88634,17042],{"class":2226},[151,88636,88637],{"class":503},"(files1):\n",[151,88639,88640,88643,88645,88647],{"class":469,"line":3811},[151,88641,88642],{"class":503},"        img ",[151,88644,1876],{"class":1869},[151,88646,44099],{"class":477},[151,88648,88649],{"class":503},".Image.open(x)\n",[151,88651,88652,88655,88657,88660,88662,88664],{"class":469,"line":3820},[151,88653,88654],{"class":503},"        plt.subplot(rows,",[151,88656,25038],{"class":477},[151,88658,88659],{"class":503},",num",[151,88661,22885],{"class":1869},[151,88663,6760],{"class":477},[151,88665,3640],{"class":503},[151,88667,88668,88671,88674,88676,88678],{"class":469,"line":7084},[151,88669,88670],{"class":503},"        plt.title(x.split(",[151,88672,88673],{"class":481},"'.'",[151,88675,40832],{"class":503},[151,88677,9181],{"class":477},[151,88679,38820],{"class":503},[151,88681,88682,88685,88687],{"class":469,"line":7148},[151,88683,88684],{"class":503},"        plt.axis(",[151,88686,44929],{"class":481},[151,88688,3640],{"class":503},[151,88690,88691],{"class":469,"line":7211},[151,88692,88693],{"class":503},"        plt.imshow(img)\n",[151,88695,88696,88698,88700,88703,88705,88707,88710,88712,88715,88717,88719],{"class":469,"line":7273},[151,88697,24285],{"class":2226},[151,88699,12386],{"class":503},[151,88701,88702],{"class":481},"'Cluster #'",[151,88704,23378],{"class":1869},[151,88706,84112],{"class":6205},[151,88708,88709],{"class":503},"(X.ix[x].clusters) ",[151,88711,22885],{"class":1869},[151,88713,88714],{"class":481}," ' -- Cluster Total: '",[151,88716,23378],{"class":1869},[151,88718,84112],{"class":6205},[151,88720,88721],{"class":503},"(cluster_dict[X.ix[x].clusters]))\n",[459,88723,88725],{"className":13136,"code":88724,"language":12886,"meta":464,"style":464},"show_images(16, samples=12)\n",[30,88726,88727],{"__ignoreMap":464},[151,88728,88729,88732,88734,88736,88738,88740,88742],{"class":469,"line":470},[151,88730,88731],{"class":503},"show_images(",[151,88733,87061],{"class":477},[151,88735,106],{"class":503},[151,88737,88440],{"class":15210},[151,88739,1876],{"class":1869},[151,88741,42360],{"class":477},[151,88743,3640],{"class":503},[459,88745,88748],{"className":88746,"code":88747,"language":997},[995],"Cluster #16 -- Cluster Total: 3293\n",[30,88749,88747],{"__ignoreMap":464},[11,88751,88752],{},[2718,88753],{"alt":20386,"src":88754},"/static/ants_files/ants_44_1.png",[459,88756,88758],{"className":13136,"code":88757,"language":12886,"meta":464,"style":464},"show_images(46, samples=12)\n",[30,88759,88760],{"__ignoreMap":464},[151,88761,88762,88764,88766,88768,88770,88772,88774],{"class":469,"line":470},[151,88763,88731],{"class":503},[151,88765,6591],{"class":477},[151,88767,106],{"class":503},[151,88769,88440],{"class":15210},[151,88771,1876],{"class":1869},[151,88773,42360],{"class":477},[151,88775,3640],{"class":503},[459,88777,88780],{"className":88778,"code":88779,"language":997},[995],"Cluster #46 -- Cluster Total: 2993\n",[30,88781,88779],{"__ignoreMap":464},[11,88783,88784],{},[2718,88785],{"alt":20386,"src":88786},"/static/ants_files/ants_45_1.png",[459,88788,88790],{"className":13136,"code":88789,"language":12886,"meta":464,"style":464},"show_images(51, samples=12)\n",[30,88791,88792],{"__ignoreMap":464},[151,88793,88794,88796,88799,88801,88803,88805,88807],{"class":469,"line":470},[151,88795,88731],{"class":503},[151,88797,88798],{"class":477},"51",[151,88800,106],{"class":503},[151,88802,88440],{"class":15210},[151,88804,1876],{"class":1869},[151,88806,42360],{"class":477},[151,88808,3640],{"class":503},[459,88810,88813],{"className":88811,"code":88812,"language":997},[995],"Cluster #51 -- Cluster Total: 25\n",[30,88814,88812],{"__ignoreMap":464},[11,88816,88817],{},[2718,88818],{"alt":20386,"src":88819},"/static/ants_files/ants_46_1.png",[459,88821,88823],{"className":13136,"code":88822,"language":12886,"meta":464,"style":464},"show_images(18, samples=12)\n",[30,88824,88825],{"__ignoreMap":464},[151,88826,88827,88829,88831,88833,88835,88837,88839],{"class":469,"line":470},[151,88828,88731],{"class":503},[151,88830,7696],{"class":477},[151,88832,106],{"class":503},[151,88834,88440],{"class":15210},[151,88836,1876],{"class":1869},[151,88838,42360],{"class":477},[151,88840,3640],{"class":503},[459,88842,88845],{"className":88843,"code":88844,"language":997},[995],"Cluster #18 -- Cluster Total: 12\n",[30,88846,88844],{"__ignoreMap":464},[11,88848,88849],{},[2718,88850],{"alt":20386,"src":88851},"/static/ants_files/ants_47_1.png",[459,88853,88855],{"className":13136,"code":88854,"language":12886,"meta":464,"style":464},"show_images(4, samples=12)\n",[30,88856,88857],{"__ignoreMap":464},[151,88858,88859,88861,88863,88865,88867,88869,88871],{"class":469,"line":470},[151,88860,88731],{"class":503},[151,88862,9187],{"class":477},[151,88864,106],{"class":503},[151,88866,88440],{"class":15210},[151,88868,1876],{"class":1869},[151,88870,42360],{"class":477},[151,88872,3640],{"class":503},[459,88874,88877],{"className":88875,"code":88876,"language":997},[995],"Cluster #4 -- Cluster Total: 302\n",[30,88878,88876],{"__ignoreMap":464},[11,88880,88881],{},[2718,88882],{"alt":20386,"src":88883},"/static/ants_files/ants_48_1.png",[459,88885,88887],{"className":13136,"code":88886,"language":12886,"meta":464,"style":464},"show_images(5, samples=12)\n",[30,88888,88889],{"__ignoreMap":464},[151,88890,88891,88893,88895,88897,88899,88901,88903],{"class":469,"line":470},[151,88892,88731],{"class":503},[151,88894,24380],{"class":477},[151,88896,106],{"class":503},[151,88898,88440],{"class":15210},[151,88900,1876],{"class":1869},[151,88902,42360],{"class":477},[151,88904,3640],{"class":503},[459,88906,88909],{"className":88907,"code":88908,"language":997},[995],"Cluster #5 -- Cluster Total: 1412\n",[30,88910,88908],{"__ignoreMap":464},[11,88912,88913],{},[2718,88914],{"alt":20386,"src":88915},"/static/ants_files/ants_49_1.png",[459,88917,88919],{"className":13136,"code":88918,"language":12886,"meta":464,"style":464},"show_images(6, samples=12)\n",[30,88920,88921],{"__ignoreMap":464},[151,88922,88923,88925,88927,88929,88931,88933,88935],{"class":469,"line":470},[151,88924,88731],{"class":503},[151,88926,25038],{"class":477},[151,88928,106],{"class":503},[151,88930,88440],{"class":15210},[151,88932,1876],{"class":1869},[151,88934,42360],{"class":477},[151,88936,3640],{"class":503},[459,88938,88941],{"className":88939,"code":88940,"language":997},[995],"Cluster #6 -- Cluster Total: 35\n",[30,88942,88940],{"__ignoreMap":464},[11,88944,88945],{},[2718,88946],{"alt":20386,"src":88947},"/static/ants_files/ants_50_1.png",[459,88949,88951],{"className":13136,"code":88950,"language":12886,"meta":464,"style":464},"show_images(9, samples=12)\n",[30,88952,88953],{"__ignoreMap":464},[151,88954,88955,88957,88959,88961,88963,88965,88967],{"class":469,"line":470},[151,88956,88731],{"class":503},[151,88958,7918],{"class":477},[151,88960,106],{"class":503},[151,88962,88440],{"class":15210},[151,88964,1876],{"class":1869},[151,88966,42360],{"class":477},[151,88968,3640],{"class":503},[459,88970,88973],{"className":88971,"code":88972,"language":997},[995],"Cluster #9 -- Cluster Total: 218\n",[30,88974,88972],{"__ignoreMap":464},[11,88976,88977],{},[2718,88978],{"alt":20386,"src":88979},"/static/ants_files/ants_51_1.png",[459,88981,88983],{"className":13136,"code":88982,"language":12886,"meta":464,"style":464},"show_images(12, samples=12)\n",[30,88984,88985],{"__ignoreMap":464},[151,88986,88987,88989,88991,88993,88995,88997,88999],{"class":469,"line":470},[151,88988,88731],{"class":503},[151,88990,42360],{"class":477},[151,88992,106],{"class":503},[151,88994,88440],{"class":15210},[151,88996,1876],{"class":1869},[151,88998,42360],{"class":477},[151,89000,3640],{"class":503},[459,89002,89005],{"className":89003,"code":89004,"language":997},[995],"Cluster #12 -- Cluster Total: 11\n",[30,89006,89004],{"__ignoreMap":464},[11,89008,89009],{},[2718,89010],{"alt":20386,"src":89011},"/static/ants_files/ants_52_1.png",[11,89013,89014],{},"The next four cluster samples are the largest clusters:",[459,89016,89018],{"className":13136,"code":89017,"language":12886,"meta":464,"style":464},"show_images(30, samples=12)\n",[30,89019,89020],{"__ignoreMap":464},[151,89021,89022,89024,89026,89028,89030,89032,89034],{"class":469,"line":470},[151,89023,88731],{"class":503},[151,89025,42017],{"class":477},[151,89027,106],{"class":503},[151,89029,88440],{"class":15210},[151,89031,1876],{"class":1869},[151,89033,42360],{"class":477},[151,89035,3640],{"class":503},[459,89037,89040],{"className":89038,"code":89039,"language":997},[995],"Cluster #30 -- Cluster Total: 45\n",[30,89041,89039],{"__ignoreMap":464},[11,89043,89044],{},[2718,89045],{"alt":20386,"src":89046},"/static/ants_files/ants_54_1.png",[459,89048,89049],{"className":13136,"code":89017,"language":12886,"meta":464,"style":464},[30,89050,89051],{"__ignoreMap":464},[151,89052,89053,89055,89057,89059,89061,89063,89065],{"class":469,"line":470},[151,89054,88731],{"class":503},[151,89056,42017],{"class":477},[151,89058,106],{"class":503},[151,89060,88440],{"class":15210},[151,89062,1876],{"class":1869},[151,89064,42360],{"class":477},[151,89066,3640],{"class":503},[459,89068,89070],{"className":89069,"code":89039,"language":997},[995],[30,89071,89039],{"__ignoreMap":464},[11,89073,89074],{},[2718,89075],{"alt":20386,"src":89076},"/static/ants_files/ants_55_1.png",[459,89078,89080],{"className":13136,"code":89079,"language":12886,"meta":464,"style":464},"show_images(0, samples=12)\n",[30,89081,89082],{"__ignoreMap":464},[151,89083,89084,89086,89088,89090,89092,89094,89096],{"class":469,"line":470},[151,89085,88731],{"class":503},[151,89087,9181],{"class":477},[151,89089,106],{"class":503},[151,89091,88440],{"class":15210},[151,89093,1876],{"class":1869},[151,89095,42360],{"class":477},[151,89097,3640],{"class":503},[459,89099,89102],{"className":89100,"code":89101,"language":997},[995],"Cluster #0 -- Cluster Total: 2752\n",[30,89103,89101],{"__ignoreMap":464},[11,89105,89106],{},[2718,89107],{"alt":20386,"src":89108},"/static/ants_files/ants_56_1.png",[459,89110,89112],{"className":13136,"code":89111,"language":12886,"meta":464,"style":464},"show_images(30, samples=15)\n",[30,89113,89114],{"__ignoreMap":464},[151,89115,89116,89118,89120,89122,89124,89126,89128],{"class":469,"line":470},[151,89117,88731],{"class":503},[151,89119,42017],{"class":477},[151,89121,106],{"class":503},[151,89123,88440],{"class":15210},[151,89125,1876],{"class":1869},[151,89127,42310],{"class":477},[151,89129,3640],{"class":503},[459,89131,89133],{"className":89132,"code":89039,"language":997},[995],[30,89134,89039],{"__ignoreMap":464},[11,89136,89137],{},[2718,89138],{"alt":20386,"src":89139},"/static/ants_files/ants_57_1.png",[459,89141,89143],{"className":13136,"code":89142,"language":12886,"meta":464,"style":464},"show_images(63, samples=12)\n",[30,89144,89145],{"__ignoreMap":464},[151,89146,89147,89149,89152,89154,89156,89158,89160],{"class":469,"line":470},[151,89148,88731],{"class":503},[151,89150,89151],{"class":477},"63",[151,89153,106],{"class":503},[151,89155,88440],{"class":15210},[151,89157,1876],{"class":1869},[151,89159,42360],{"class":477},[151,89161,3640],{"class":503},[459,89163,89166],{"className":89164,"code":89165,"language":997},[995],"Cluster #63 -- Cluster Total: 123\n",[30,89167,89165],{"__ignoreMap":464},[11,89169,89170],{},[2718,89171],{"alt":20386,"src":89172},"/static/ants_files/ants_58_1.png",[459,89174,89176],{"className":13136,"code":89175,"language":12886,"meta":464,"style":464},"show_images(60, samples=12)\n",[30,89177,89178],{"__ignoreMap":464},[151,89179,89180,89182,89184,89186,89188,89190,89192],{"class":469,"line":470},[151,89181,88731],{"class":503},[151,89183,39825],{"class":477},[151,89185,106],{"class":503},[151,89187,88440],{"class":15210},[151,89189,1876],{"class":1869},[151,89191,42360],{"class":477},[151,89193,3640],{"class":503},[459,89195,89198],{"className":89196,"code":89197,"language":997},[995],"Cluster #60 -- Cluster Total: 30\n",[30,89199,89197],{"__ignoreMap":464},[11,89201,89202],{},[2718,89203],{"alt":20386,"src":89204},"/static/ants_files/ants_59_1.png",[459,89206,89208],{"className":13136,"code":89207,"language":12886,"meta":464,"style":464},"show_images(59, samples=12)\n",[30,89209,89210],{"__ignoreMap":464},[151,89211,89212,89214,89217,89219,89221,89223,89225],{"class":469,"line":470},[151,89213,88731],{"class":503},[151,89215,89216],{"class":477},"59",[151,89218,106],{"class":503},[151,89220,88440],{"class":15210},[151,89222,1876],{"class":1869},[151,89224,42360],{"class":477},[151,89226,3640],{"class":503},[459,89228,89231],{"className":89229,"code":89230,"language":997},[995],"Cluster #59 -- Cluster Total: 10\n",[30,89232,89230],{"__ignoreMap":464},[11,89234,89235],{},[2718,89236],{"alt":20386,"src":89237},"/static/ants_files/ants_60_1.png",[459,89239,89241],{"className":13136,"code":89240,"language":12886,"meta":464,"style":464},"show_images(57, samples=12)\n",[30,89242,89243],{"__ignoreMap":464},[151,89244,89245,89247,89249,89251,89253,89255,89257],{"class":469,"line":470},[151,89246,88731],{"class":503},[151,89248,58453],{"class":477},[151,89250,106],{"class":503},[151,89252,88440],{"class":15210},[151,89254,1876],{"class":1869},[151,89256,42360],{"class":477},[151,89258,3640],{"class":503},[459,89260,89263],{"className":89261,"code":89262,"language":997},[995],"Cluster #57 -- Cluster Total: 314\n",[30,89264,89262],{"__ignoreMap":464},[11,89266,89267],{},[2718,89268],{"alt":20386,"src":89269},"/static/ants_files/ants_61_1.png",[459,89271,89273],{"className":13136,"code":89272,"language":12886,"meta":464,"style":464},"show_images(56, samples=12)\n",[30,89274,89275],{"__ignoreMap":464},[151,89276,89277,89279,89282,89284,89286,89288,89290],{"class":469,"line":470},[151,89278,88731],{"class":503},[151,89280,89281],{"class":477},"56",[151,89283,106],{"class":503},[151,89285,88440],{"class":15210},[151,89287,1876],{"class":1869},[151,89289,42360],{"class":477},[151,89291,3640],{"class":503},[459,89293,89296],{"className":89294,"code":89295,"language":997},[995],"Cluster #56 -- Cluster Total: 1\n",[30,89297,89295],{"__ignoreMap":464},[11,89299,89300],{},[2718,89301],{"alt":20386,"src":89302},"/static/ants_files/ants_62_1.png",[459,89304,89306],{"className":13136,"code":89305,"language":12886,"meta":464,"style":464},"show_images(55, samples=12)\n",[30,89307,89308],{"__ignoreMap":464},[151,89309,89310,89312,89314,89316,89318,89320,89322],{"class":469,"line":470},[151,89311,88731],{"class":503},[151,89313,41835],{"class":477},[151,89315,106],{"class":503},[151,89317,88440],{"class":15210},[151,89319,1876],{"class":1869},[151,89321,42360],{"class":477},[151,89323,3640],{"class":503},[459,89325,89328],{"className":89326,"code":89327,"language":997},[995],"Cluster #55 -- Cluster Total: 38\n",[30,89329,89327],{"__ignoreMap":464},[11,89331,89332],{},[2718,89333],{"alt":20386,"src":89334},"/static/ants_files/ants_63_1.png",[459,89336,89338],{"className":13136,"code":89337,"language":12886,"meta":464,"style":464},"show_images(54, samples=12)\n",[30,89339,89340],{"__ignoreMap":464},[151,89341,89342,89344,89347,89349,89351,89353,89355],{"class":469,"line":470},[151,89343,88731],{"class":503},[151,89345,89346],{"class":477},"54",[151,89348,106],{"class":503},[151,89350,88440],{"class":15210},[151,89352,1876],{"class":1869},[151,89354,42360],{"class":477},[151,89356,3640],{"class":503},[459,89358,89361],{"className":89359,"code":89360,"language":997},[995],"Cluster #54 -- Cluster Total: 686\n",[30,89362,89360],{"__ignoreMap":464},[11,89364,89365],{},[2718,89366],{"alt":20386,"src":89367},"/static/ants_files/ants_64_1.png",[459,89369,89371],{"className":13136,"code":89370,"language":12886,"meta":464,"style":464},"show_images(53, samples=12)\n",[30,89372,89373],{"__ignoreMap":464},[151,89374,89375,89377,89379,89381,89383,89385,89387],{"class":469,"line":470},[151,89376,88731],{"class":503},[151,89378,9110],{"class":477},[151,89380,106],{"class":503},[151,89382,88440],{"class":15210},[151,89384,1876],{"class":1869},[151,89386,42360],{"class":477},[151,89388,3640],{"class":503},[459,89390,89393],{"className":89391,"code":89392,"language":997},[995],"Cluster #53 -- Cluster Total: 8\n",[30,89394,89392],{"__ignoreMap":464},[11,89396,89397],{},[2718,89398],{"alt":20386,"src":89399},"/static/ants_files/ants_65_1.png",[459,89401,89403],{"className":13136,"code":89402,"language":12886,"meta":464,"style":464},"show_images(52, samples=12)\n",[30,89404,89405],{"__ignoreMap":464},[151,89406,89407,89409,89411,89413,89415,89417,89419],{"class":469,"line":470},[151,89408,88731],{"class":503},[151,89410,45428],{"class":477},[151,89412,106],{"class":503},[151,89414,88440],{"class":15210},[151,89416,1876],{"class":1869},[151,89418,42360],{"class":477},[151,89420,3640],{"class":503},[459,89422,89425],{"className":89423,"code":89424,"language":997},[995],"Cluster #52 -- Cluster Total: 2540\n",[30,89426,89424],{"__ignoreMap":464},[11,89428,89429],{},[2718,89430],{"alt":20386,"src":89431},"/static/ants_files/ants_66_1.png",[459,89433,89434],{"className":13136,"code":88789,"language":12886,"meta":464,"style":464},[30,89435,89436],{"__ignoreMap":464},[151,89437,89438,89440,89442,89444,89446,89448,89450],{"class":469,"line":470},[151,89439,88731],{"class":503},[151,89441,88798],{"class":477},[151,89443,106],{"class":503},[151,89445,88440],{"class":15210},[151,89447,1876],{"class":1869},[151,89449,42360],{"class":477},[151,89451,3640],{"class":503},[459,89453,89455],{"className":89454,"code":88812,"language":997},[995],[30,89456,88812],{"__ignoreMap":464},[11,89458,89459],{},[2718,89460],{"alt":20386,"src":89461},"/static/ants_files/ants_67_1.png",[459,89463,89465],{"className":13136,"code":89464,"language":12886,"meta":464,"style":464},"show_images(50, samples=12)\n",[30,89466,89467],{"__ignoreMap":464},[151,89468,89469,89471,89473,89475,89477,89479,89481],{"class":469,"line":470},[151,89470,88731],{"class":503},[151,89472,73146],{"class":477},[151,89474,106],{"class":503},[151,89476,88440],{"class":15210},[151,89478,1876],{"class":1869},[151,89480,42360],{"class":477},[151,89482,3640],{"class":503},[459,89484,89487],{"className":89485,"code":89486,"language":997},[995],"Cluster #50 -- Cluster Total: 2185\n",[30,89488,89486],{"__ignoreMap":464},[11,89490,89491],{},[2718,89492],{"alt":20386,"src":89493},"/static/ants_files/ants_68_1.png",[459,89495,89497],{"className":13136,"code":89496,"language":12886,"meta":464,"style":464},"show_images(49, samples=12)\n",[30,89498,89499],{"__ignoreMap":464},[151,89500,89501,89503,89506,89508,89510,89512,89514],{"class":469,"line":470},[151,89502,88731],{"class":503},[151,89504,89505],{"class":477},"49",[151,89507,106],{"class":503},[151,89509,88440],{"class":15210},[151,89511,1876],{"class":1869},[151,89513,42360],{"class":477},[151,89515,3640],{"class":503},[459,89517,89520],{"className":89518,"code":89519,"language":997},[995],"Cluster #49 -- Cluster Total: 178\n",[30,89521,89519],{"__ignoreMap":464},[11,89523,89524],{},[2718,89525],{"alt":20386,"src":89526},"/static/ants_files/ants_69_1.png",[459,89528,89530],{"className":13136,"code":89529,"language":12886,"meta":464,"style":464},"show_images(48, samples=12)\n",[30,89531,89532],{"__ignoreMap":464},[151,89533,89534,89536,89538,89540,89542,89544,89546],{"class":469,"line":470},[151,89535,88731],{"class":503},[151,89537,41852],{"class":477},[151,89539,106],{"class":503},[151,89541,88440],{"class":15210},[151,89543,1876],{"class":1869},[151,89545,42360],{"class":477},[151,89547,3640],{"class":503},[459,89549,89552],{"className":89550,"code":89551,"language":997},[995],"Cluster #48 -- Cluster Total: 3\n",[30,89553,89551],{"__ignoreMap":464},[11,89555,89556],{},[2718,89557],{"alt":20386,"src":89558},"/static/ants_files/ants_70_1.png",[459,89560,89562],{"className":13136,"code":89561,"language":12886,"meta":464,"style":464},"show_images(47, samples=12)\n",[30,89563,89564],{"__ignoreMap":464},[151,89565,89566,89568,89570,89572,89574,89576,89578],{"class":469,"line":470},[151,89567,88731],{"class":503},[151,89569,7691],{"class":477},[151,89571,106],{"class":503},[151,89573,88440],{"class":15210},[151,89575,1876],{"class":1869},[151,89577,42360],{"class":477},[151,89579,3640],{"class":503},[459,89581,89584],{"className":89582,"code":89583,"language":997},[995],"Cluster #47 -- Cluster Total: 11\n",[30,89585,89583],{"__ignoreMap":464},[11,89587,89588],{},[2718,89589],{"alt":20386,"src":89590},"/static/ants_files/ants_71_1.png",[459,89592,89593],{"className":13136,"code":88757,"language":12886,"meta":464,"style":464},[30,89594,89595],{"__ignoreMap":464},[151,89596,89597,89599,89601,89603,89605,89607,89609],{"class":469,"line":470},[151,89598,88731],{"class":503},[151,89600,6591],{"class":477},[151,89602,106],{"class":503},[151,89604,88440],{"class":15210},[151,89606,1876],{"class":1869},[151,89608,42360],{"class":477},[151,89610,3640],{"class":503},[459,89612,89614],{"className":89613,"code":88779,"language":997},[995],[30,89615,88779],{"__ignoreMap":464},[11,89617,89618],{},[2718,89619],{"alt":20386,"src":89620},"/static/ants_files/ants_72_1.png",[459,89622,89624],{"className":13136,"code":89623,"language":12886,"meta":464,"style":464},"show_images(45, samples=12)\n",[30,89625,89626],{"__ignoreMap":464},[151,89627,89628,89630,89632,89634,89636,89638,89640],{"class":469,"line":470},[151,89629,88731],{"class":503},[151,89631,87323],{"class":477},[151,89633,106],{"class":503},[151,89635,88440],{"class":15210},[151,89637,1876],{"class":1869},[151,89639,42360],{"class":477},[151,89641,3640],{"class":503},[459,89643,89646],{"className":89644,"code":89645,"language":997},[995],"Cluster #45 -- Cluster Total: 816\n",[30,89647,89645],{"__ignoreMap":464},[11,89649,89650],{},[2718,89651],{"alt":20386,"src":89652},"/static/ants_files/ants_73_1.png",[459,89654,89656],{"className":13136,"code":89655,"language":12886,"meta":464,"style":464},"show_images(44, samples=12)\n",[30,89657,89658],{"__ignoreMap":464},[151,89659,89660,89662,89664,89666,89668,89670,89672],{"class":469,"line":470},[151,89661,88731],{"class":503},[151,89663,41885],{"class":477},[151,89665,106],{"class":503},[151,89667,88440],{"class":15210},[151,89669,1876],{"class":1869},[151,89671,42360],{"class":477},[151,89673,3640],{"class":503},[459,89675,89678],{"className":89676,"code":89677,"language":997},[995],"Cluster #44 -- Cluster Total: 791\n",[30,89679,89677],{"__ignoreMap":464},[11,89681,89682],{},[2718,89683],{"alt":20386,"src":89684},"/static/ants_files/ants_74_1.png",[459,89686,89688],{"className":13136,"code":89687,"language":12886,"meta":464,"style":464},"show_images(43, samples=12)\n",[30,89689,89690],{"__ignoreMap":464},[151,89691,89692,89694,89696,89698,89700,89702,89704],{"class":469,"line":470},[151,89693,88731],{"class":503},[151,89695,41934],{"class":477},[151,89697,106],{"class":503},[151,89699,88440],{"class":15210},[151,89701,1876],{"class":1869},[151,89703,42360],{"class":477},[151,89705,3640],{"class":503},[459,89707,89710],{"className":89708,"code":89709,"language":997},[995],"Cluster #43 -- Cluster Total: 30\n",[30,89711,89709],{"__ignoreMap":464},[11,89713,89714],{},[2718,89715],{"alt":20386,"src":89716},"/static/ants_files/ants_75_1.png",[459,89718,89720],{"className":13136,"code":89719,"language":12886,"meta":464,"style":464},"show_images(42, samples=12)\n",[30,89721,89722],{"__ignoreMap":464},[151,89723,89724,89726,89729,89731,89733,89735,89737],{"class":469,"line":470},[151,89725,88731],{"class":503},[151,89727,89728],{"class":477},"42",[151,89730,106],{"class":503},[151,89732,88440],{"class":15210},[151,89734,1876],{"class":1869},[151,89736,42360],{"class":477},[151,89738,3640],{"class":503},[459,89740,89743],{"className":89741,"code":89742,"language":997},[995],"Cluster #42 -- Cluster Total: 625\n",[30,89744,89742],{"__ignoreMap":464},[11,89746,89747],{},[2718,89748],{"alt":20386,"src":89749},"/static/ants_files/ants_76_1.png",[459,89751,89753],{"className":13136,"code":89752,"language":12886,"meta":464,"style":464},"show_images(41, samples=12)\n",[30,89754,89755],{"__ignoreMap":464},[151,89756,89757,89759,89761,89763,89765,89767,89769],{"class":469,"line":470},[151,89758,88731],{"class":503},[151,89760,87303],{"class":477},[151,89762,106],{"class":503},[151,89764,88440],{"class":15210},[151,89766,1876],{"class":1869},[151,89768,42360],{"class":477},[151,89770,3640],{"class":503},[459,89772,89775],{"className":89773,"code":89774,"language":997},[995],"Cluster #41 -- Cluster Total: 6\n",[30,89776,89774],{"__ignoreMap":464},[11,89778,89779],{},[2718,89780],{"alt":20386,"src":89781},"/static/ants_files/ants_77_1.png",[459,89783,89785],{"className":13136,"code":89784,"language":12886,"meta":464,"style":464},"show_images(40, samples=12)\n",[30,89786,89787],{"__ignoreMap":464},[151,89788,89789,89791,89793,89795,89797,89799,89801],{"class":469,"line":470},[151,89790,88731],{"class":503},[151,89792,44365],{"class":477},[151,89794,106],{"class":503},[151,89796,88440],{"class":15210},[151,89798,1876],{"class":1869},[151,89800,42360],{"class":477},[151,89802,3640],{"class":503},[459,89804,89807],{"className":89805,"code":89806,"language":997},[995],"Cluster #40 -- Cluster Total: 6\n",[30,89808,89806],{"__ignoreMap":464},[11,89810,89811],{},[2718,89812],{"alt":20386,"src":89813},"/static/ants_files/ants_78_1.png",[459,89815,89817],{"className":13136,"code":89816,"language":12886,"meta":464,"style":464},"show_images(39, samples=12)\n",[30,89818,89819],{"__ignoreMap":464},[151,89820,89821,89823,89826,89828,89830,89832,89834],{"class":469,"line":470},[151,89822,88731],{"class":503},[151,89824,89825],{"class":477},"39",[151,89827,106],{"class":503},[151,89829,88440],{"class":15210},[151,89831,1876],{"class":1869},[151,89833,42360],{"class":477},[151,89835,3640],{"class":503},[459,89837,89840],{"className":89838,"code":89839,"language":997},[995],"Cluster #39 -- Cluster Total: 26\n",[30,89841,89839],{"__ignoreMap":464},[11,89843,89844],{},[2718,89845],{"alt":20386,"src":89846},"/static/ants_files/ants_79_1.png",[459,89848,89850],{"className":13136,"code":89849,"language":12886,"meta":464,"style":464},"show_images(38, samples=12)\n",[30,89851,89852],{"__ignoreMap":464},[151,89853,89854,89856,89858,89860,89862,89864,89866],{"class":469,"line":470},[151,89855,88731],{"class":503},[151,89857,7907],{"class":477},[151,89859,106],{"class":503},[151,89861,88440],{"class":15210},[151,89863,1876],{"class":1869},[151,89865,42360],{"class":477},[151,89867,3640],{"class":503},[459,89869,89872],{"className":89870,"code":89871,"language":997},[995],"Cluster #38 -- Cluster Total: 345\n",[30,89873,89871],{"__ignoreMap":464},[11,89875,89876],{},[2718,89877],{"alt":20386,"src":89878},"/static/ants_files/ants_80_1.png",[459,89880,89882],{"className":13136,"code":89881,"language":12886,"meta":464,"style":464},"show_images(37, samples=12)\n",[30,89883,89884],{"__ignoreMap":464},[151,89885,89886,89888,89890,89892,89894,89896,89898],{"class":469,"line":470},[151,89887,88731],{"class":503},[151,89889,87333],{"class":477},[151,89891,106],{"class":503},[151,89893,88440],{"class":15210},[151,89895,1876],{"class":1869},[151,89897,42360],{"class":477},[151,89899,3640],{"class":503},[459,89901,89904],{"className":89902,"code":89903,"language":997},[995],"Cluster #37 -- Cluster Total: 518\n",[30,89905,89903],{"__ignoreMap":464},[11,89907,89908],{},[2718,89909],{"alt":20386,"src":89910},"/static/ants_files/ants_81_1.png",[11,89912,89913],{},"The vast majority of termites seem to form nondescript blobs after 100000 steps. There are perhaps many thousands of termites that didn't yet reach a . Setting the clusters parameter to 75 is probably too high. Many of the groups have similar behaviour. There were several cluster groups that formed 'highways'. It may make more sense to filter out these termites and cluster termites that didn't form highways.",[14063,89915,89917],{"id":89916},"outlier-detection","Outlier Detection",[11,89919,89920],{},"It could also be interesting to see how many outliers are present in each cluster for various values of k in the k-means algorithm. This may help us choose a more fitting number of clusters by which the termites can be grouped. here's how we could do that:",[459,89922,89924],{"className":13136,"code":89923,"language":12886,"meta":464,"style":464},"from sklearn.ensemble import IsolationForest\n",[30,89925,89926],{"__ignoreMap":464},[151,89927,89928,89930,89933,89935],{"class":469,"line":470},[151,89929,16853],{"class":1869},[151,89931,89932],{"class":503}," sklearn.ensemble ",[151,89934,16859],{"class":1869},[151,89936,89937],{"class":503}," IsolationForest\n",[459,89939,89941],{"className":13136,"code":89940,"language":12886,"meta":464,"style":464},"X_ = X.loc[X.clusters==37, :] #16\n",[30,89942,89943],{"__ignoreMap":464},[151,89944,89945,89948,89950,89953,89955,89957,89959],{"class":469,"line":470},[151,89946,89947],{"class":503},"X_ ",[151,89949,1876],{"class":1869},[151,89951,89952],{"class":503}," X.loc[X.clusters",[151,89954,17223],{"class":1869},[151,89956,87333],{"class":477},[151,89958,24654],{"class":503},[151,89960,89961],{"class":1527},"#16\n",[459,89963,89965],{"className":13136,"code":89964,"language":12886,"meta":464,"style":464},"X_ = X.loc[X.clusters==37, :]\n",[30,89966,89967],{"__ignoreMap":464},[151,89968,89969,89971,89973,89975,89977,89979],{"class":469,"line":470},[151,89970,89947],{"class":503},[151,89972,1876],{"class":1869},[151,89974,89952],{"class":503},[151,89976,17223],{"class":1869},[151,89978,87333],{"class":477},[151,89980,25087],{"class":503},[459,89982,89984],{"className":13136,"code":89983,"language":12886,"meta":464,"style":464},"clf = IsolationForest(max_samples=100, random_state=rng)\nclf.fit(X_)\n\ny_pred_train = clf.predict(X_)\n\n",[30,89985,89986,90012,90017,90021],{"__ignoreMap":464},[151,89987,89988,89991,89993,89996,89999,90001,90003,90005,90007,90009],{"class":469,"line":470},[151,89989,89990],{"class":503},"clf ",[151,89992,1876],{"class":1869},[151,89994,89995],{"class":503}," IsolationForest(",[151,89997,89998],{"class":15210},"max_samples",[151,90000,1876],{"class":1869},[151,90002,71821],{"class":477},[151,90004,106],{"class":503},[151,90006,71775],{"class":15210},[151,90008,1876],{"class":1869},[151,90010,90011],{"class":503},"rng)\n",[151,90013,90014],{"class":469,"line":488},[151,90015,90016],{"class":503},"clf.fit(X_)\n",[151,90018,90019],{"class":469,"line":500},[151,90020,1090],{"emptyLinePlaceholder":609},[151,90022,90023,90026,90028],{"class":469,"line":509},[151,90024,90025],{"class":503},"y_pred_train ",[151,90027,1876],{"class":1869},[151,90029,90030],{"class":503}," clf.predict(X_)\n",[459,90032,90034],{"className":13136,"code":90033,"language":12886,"meta":464,"style":464},"y_pred_train.mean()\n",[30,90035,90036],{"__ignoreMap":464},[151,90037,90038],{"class":469,"line":470},[151,90039,90033],{"class":503},[459,90041,90044],{"className":90042,"code":90043,"language":997},[995],"0.79922779922779918\n",[30,90045,90043],{"__ignoreMap":464},[11,90047,90048],{},"The following values gives us the average of the predicted values (1 for inlier, -1 for outlier), so this value doesn't correspond to a percentage accuracy. The accuracy is about 89% (the model determined that 89% of termites in cluster 37 are inliers and the remaining 11% are outliers.",[459,90050,90052],{"className":13136,"code":90051,"language":12886,"meta":464,"style":464},"X_['anom'] = y_pred_train\n",[30,90053,90054],{"__ignoreMap":464},[151,90055,90056,90059,90062,90064,90066],{"class":469,"line":470},[151,90057,90058],{"class":503},"X_[",[151,90060,90061],{"class":481},"'anom'",[151,90063,16654],{"class":503},[151,90065,1876],{"class":1869},[151,90067,90068],{"class":503}," y_pred_train\n",[459,90070,90072],{"className":13136,"code":90071,"language":12886,"meta":464,"style":464},"X_.anom.value_counts()\n",[30,90073,90074],{"__ignoreMap":464},[151,90075,90076],{"class":469,"line":470},[151,90077,90071],{"class":503},[459,90079,90082],{"className":90080,"code":90081,"language":997},[995]," 1    466\n-1     52\nName: anom, dtype: int64\n",[30,90083,90081],{"__ignoreMap":464},[11,90085,90086],{},"Let's compare some of the inliers with the outliers:",[459,90088,90090],{"className":13136,"code":90089,"language":12886,"meta":464,"style":464},"files_normal = X_[X_.anom==(1)]\nshow_images(0, samples = 24, files_bool=True, files=files_normal)\n",[30,90091,90092,90110],{"__ignoreMap":464},[151,90093,90094,90097,90099,90102,90104,90106,90108],{"class":469,"line":470},[151,90095,90096],{"class":503},"files_normal ",[151,90098,1876],{"class":1869},[151,90100,90101],{"class":503}," X_[X_.anom",[151,90103,17223],{"class":1869},[151,90105,12386],{"class":503},[151,90107,6760],{"class":477},[151,90109,44576],{"class":503},[151,90111,90112,90114,90116,90118,90120,90122,90125,90127,90129,90131,90133,90135,90137,90139],{"class":469,"line":488},[151,90113,88731],{"class":503},[151,90115,9181],{"class":477},[151,90117,106],{"class":503},[151,90119,88440],{"class":15210},[151,90121,19865],{"class":1869},[151,90123,90124],{"class":477}," 24",[151,90126,106],{"class":503},[151,90128,88449],{"class":15210},[151,90130,1876],{"class":1869},[151,90132,36962],{"class":477},[151,90134,106],{"class":503},[151,90136,88291],{"class":15210},[151,90138,1876],{"class":1869},[151,90140,90141],{"class":503},"files_normal)\n",[459,90143,90145],{"className":90144,"code":89903,"language":997},[995],[30,90146,89903],{"__ignoreMap":464},[11,90148,90149],{},[2718,90150],{"alt":20386,"src":90151},"/static/ants_files/ants_92_1.png",[459,90153,90155],{"className":13136,"code":90154,"language":12886,"meta":464,"style":464},"files_abnormal = X_[X_.anom==(-1)]\nshow_images(0, samples = 24, files_bool=True, files=files_abnormal)\n",[30,90156,90157,90176],{"__ignoreMap":464},[151,90158,90159,90162,90164,90166,90168,90170,90172,90174],{"class":469,"line":470},[151,90160,90161],{"class":503},"files_abnormal ",[151,90163,1876],{"class":1869},[151,90165,90101],{"class":503},[151,90167,17223],{"class":1869},[151,90169,12386],{"class":503},[151,90171,12445],{"class":1869},[151,90173,6760],{"class":477},[151,90175,44576],{"class":503},[151,90177,90178,90180,90182,90184,90186,90188,90190,90192,90194,90196,90198,90200,90202,90204],{"class":469,"line":488},[151,90179,88731],{"class":503},[151,90181,9181],{"class":477},[151,90183,106],{"class":503},[151,90185,88440],{"class":15210},[151,90187,19865],{"class":1869},[151,90189,90124],{"class":477},[151,90191,106],{"class":503},[151,90193,88449],{"class":15210},[151,90195,1876],{"class":1869},[151,90197,36962],{"class":477},[151,90199,106],{"class":503},[151,90201,88291],{"class":15210},[151,90203,1876],{"class":1869},[151,90205,90206],{"class":503},"files_abnormal)\n",[459,90208,90210],{"className":90209,"code":89903,"language":997},[995],[30,90211,89903],{"__ignoreMap":464},[11,90213,90214],{},[2718,90215],{"alt":20386,"src":90216},"/static/ants_files/ants_93_1.png",[11,90218,90219],{},"This sample of outliers seems to have slightly different characteristics compared with the inlier sample. This can be seen in the patches of solid colors (pink, purple, teal, grey).",[14063,90221,14265],{"id":30030},[11,90223,90224],{},"Using k-means and Isolation Forests with this set of over 30,000 termites offers a quick and easy way to sort out major trends that these deterministic systems display. As you can see in the cluster samples above, the classification is far from perfect. Some near-identical termites are in different clusters. It would be interesting to tweak some aspects of this experiment in the future:",[76,90226,90227,90230,90233,90236,90239],{},[79,90228,90229],{},"Larger number of states (>16)",[79,90231,90232],{},"More steps (>100000) / bigger grid",[79,90234,90235],{},"Random \"noise\" on the grid at step 0",[79,90237,90238],{},"Variation on the rules",[79,90240,90241],{},"Segmenting 'highway' termites before clustering",[589,90243,90244],{},"html pre.shiki code .sC2Qs, html code.shiki .sC2Qs{--shiki-default:#D73A49;--shiki-dark:#F97583;--shiki-sepia:#F92672}html pre.shiki code .s7F3e, html code.shiki .s7F3e{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#AE81FF}html pre.shiki code .sMOD_, html code.shiki .sMOD_{--shiki-default:#24292E;--shiki-dark:#E1E4E8;--shiki-sepia:#F8F8F2}html pre.shiki code .sXSQT, html code.shiki .sXSQT{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#F8F8F2}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html .sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html.sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html pre.shiki code .s8-w5, html code.shiki .s8-w5{--shiki-default:#6A737D;--shiki-dark:#6A737D;--shiki-sepia:#88846F}html pre.shiki code .s-m8C, html code.shiki .s-m8C{--shiki-default:#005CC5;--shiki-default-font-style:inherit;--shiki-dark:#79B8FF;--shiki-dark-font-style:inherit;--shiki-sepia:#66D9EF;--shiki-sepia-font-style:italic}html pre.shiki code .sstjo, html code.shiki .sstjo{--shiki-default:#032F62;--shiki-dark:#9ECBFF;--shiki-sepia:#E6DB74}html pre.shiki code .sq6CD, html code.shiki .sq6CD{--shiki-default:#D73A49;--shiki-default-font-style:inherit;--shiki-dark:#F97583;--shiki-dark-font-style:inherit;--shiki-sepia:#66D9EF;--shiki-sepia-font-style:italic}html pre.shiki code .srTi1, html code.shiki .srTi1{--shiki-default:#6F42C1;--shiki-dark:#B392F0;--shiki-sepia:#A6E22E}html pre.shiki code .so59x, html code.shiki .so59x{--shiki-default:#24292E;--shiki-default-font-style:inherit;--shiki-dark:#E1E4E8;--shiki-dark-font-style:inherit;--shiki-sepia:#FD971F;--shiki-sepia-font-style:italic}html pre.shiki code .sTrkL, html code.shiki .sTrkL{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#66D9EF}html pre.shiki code .sTHNf, html code.shiki .sTHNf{--shiki-default:#E36209;--shiki-default-font-style:inherit;--shiki-dark:#FFAB70;--shiki-dark-font-style:inherit;--shiki-sepia:#FD971F;--shiki-sepia-font-style:italic}",{"title":464,"searchDepth":488,"depth":488,"links":90246},[],"2017-04-02","Visualizing variations of the classic Langton's Ant cellular automata","/static/ant.png",{"layout":48045},"/2017/04/02/langton-ant-notebook",{"title":83660,"description":90248},"2017/04/02/langton-ant-notebook",[12886,90255],"cellular-automata","fpOQxn0ga1QYSE34ykzEj8f7CbcMw3v9S_hL7-SRQho",{"id":90258,"title":90259,"body":90260,"comments":609,"date":91988,"description":90264,"draft":602,"extension":605,"external":606,"image":91989,"meta":91990,"navigation":609,"path":91991,"seo":91992,"stem":91993,"tags":91994,"__hash__":91995},"blog/2017/03/04/settinging-up-django-with-heroku.md","Setting up a Django app on Heroku",{"type":8,"value":90261,"toc":91986},[90262,90265,90267,90270,90276,90279,90285,90291,90299,90357,90365,90369,90410,90435,90441,90444,90450,90453,90459,90464,90470,90473,90479,90484,90490,90496,90502,90509,90512,90518,90525,90529,90535,90550,90557,90560,90566,90576,90585,90591,90597,90600,90606,90609,90615,90618,90624,90631,90634,90639,90645,90648,90654,90657,90663,90671,90725,90728,90733,90736,90742,90750,90756,90759,90765,90771,90778,90783,90799,90805,90808,90812,90935,90938,90944,90949,91078,91085,91186,91192,91198,91205,91211,91219,91225,91228,91232,91235,91238,91244,91249,91256,91349,91355,91458,91471,91477,91483,91489,91498,91512,91550,91557,91563,91569,91967,91975,91981,91983],[11,90263,90264],{},"This is a simple guide to setting up a Django project on Heroku.",[11,90266,59441],{},[11,90268,90269],{},"The first step is to create a virutal environment in a new directory:",[459,90271,90274],{"className":90272,"code":90273,"language":997},[995],"$ mkdir proj && cd proj\n$ virtualenv -p python3 .\n$ source bin/activate\n(proj) $ mkdir src\n(proj) $ cd src\n(proj) $ pip install django==1.10.5\n(proj) $ django-admin.py startproject myproj .\n(proj) $ ls\nmyproj        manage.py\n",[30,90275,90273],{"__ignoreMap":464},[11,90277,90278],{},"This sets up a virtual environment and creates an empty Django project. The next step is to create a settings module.",[459,90280,90283],{"className":90281,"code":90282,"language":997},[995],"(proj) $ cd myproj\n(proj) $ mkdir settings && cd settings\n",[30,90284,90282],{"__ignoreMap":464},[11,90286,90287,90288,90290],{},"Next we want to add ",[30,90289,69455],{}," to settings to make it a python module.",[11,90292,90293],{},[51,90294,90295,90296,90298],{},"src/myproj/settings/",[15,90297,26029],{},".py",[459,90300,90301],{"className":13136,"code":69942,"language":12886,"meta":464,"style":464},[30,90302,90303,90313,90317,90327,90331,90337,90347,90353],{"__ignoreMap":464},[151,90304,90305,90307,90309,90311],{"class":469,"line":470},[151,90306,16853],{"class":1869},[151,90308,69951],{"class":503},[151,90310,16859],{"class":1869},[151,90312,69956],{"class":1869},[151,90314,90315],{"class":469,"line":488},[151,90316,1090],{"emptyLinePlaceholder":609},[151,90318,90319,90321,90323,90325],{"class":469,"line":500},[151,90320,16853],{"class":1869},[151,90322,69967],{"class":503},[151,90324,16859],{"class":1869},[151,90326,69956],{"class":1869},[151,90328,90329],{"class":469,"line":509},[151,90330,1090],{"emptyLinePlaceholder":609},[151,90332,90333,90335],{"class":469,"line":517},[151,90334,69980],{"class":1869},[151,90336,14372],{"class":503},[151,90338,90339,90341,90343,90345],{"class":469,"line":534},[151,90340,40344],{"class":1869},[151,90342,69989],{"class":503},[151,90344,16859],{"class":1869},[151,90346,69956],{"class":1869},[151,90348,90349,90351],{"class":469,"line":1413},[151,90350,69998],{"class":1869},[151,90352,14372],{"class":503},[151,90354,90355],{"class":469,"line":1418},[151,90356,70005],{"class":1869},[11,90358,90359,90360,90362,90363,208],{},"Next we want to change the ",[30,90361,50479],{}," (base directory) in ",[30,90364,50465],{},[11,90366,90367],{},[51,90368,50465],{},[459,90370,90372],{"className":13136,"code":90371,"language":12886,"meta":464,"style":464},"[...]\n# Build paths inside the project like this: os.path.join(BASE_DIR, ...)\nBASE_DIR = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))\n[...]\n",[30,90373,90374,90382,90387,90402],{"__ignoreMap":464},[151,90375,90376,90378,90380],{"class":469,"line":470},[151,90377,6698],{"class":503},[151,90379,27455],{"class":477},[151,90381,3691],{"class":503},[151,90383,90384],{"class":469,"line":488},[151,90385,90386],{"class":1527},"# Build paths inside the project like this: os.path.join(BASE_DIR, ...)\n",[151,90388,90389,90391,90393,90396,90399],{"class":469,"line":500},[151,90390,50479],{"class":477},[151,90392,19865],{"class":1869},[151,90394,90395],{"class":503}," os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(",[151,90397,90398],{"class":12360},"__file__",[151,90400,90401],{"class":503},"))))\n",[151,90403,90404,90406,90408],{"class":469,"line":509},[151,90405,6698],{"class":503},[151,90407,27455],{"class":477},[151,90409,3691],{"class":503},[11,90411,90412,90413,90415,90416,90418,90419,90422,90423,90425,90426,187,90429,90432,90433,643],{},"Next we need to move ",[30,90414,50465],{}," into the ",[30,90417,48746],{}," folder and rename it ",[30,90420,90421],{},"base.py"," and then copy ",[30,90424,90421],{}," twice as ",[30,90427,90428],{},"local.py",[30,90430,90431],{},"production.py"," these three files will live in ",[30,90434,48746],{},[459,90436,90439],{"className":90437,"code":90438,"language":997},[995],"(proj) $ mv settings.py settings\n(proj) $ cd settings\n(proj) $ mv settings.py base.py\n(proj) $ cp base.py local.py\n(proj) $ cp base.py production.py\n",[30,90440,90438],{"__ignoreMap":464},[11,90442,90443],{},"Next we need to install PostgreSQL:",[459,90445,90448],{"className":90446,"code":90447,"language":997},[995],"(proj) $ pip install psycopg2\n(proj) $ pip install gunicorn dj-database-url\n(proj) $ pip install django-crispy-forms\n(proj) $ pip install pillow\n",[30,90449,90447],{"__ignoreMap":464},[11,90451,90452],{},"At this point we can check to see if everything installed correctly:",[459,90454,90457],{"className":90455,"code":90456,"language":997},[995],"(proj) $ pip freeze\nappdirs==1.4.3\ndj-database-url==0.4.2\nDjango==1.10.5\ndjango-crispy-forms==1.6.1\ngunicorn==19.7.1\nolefile==0.44\npackaging==16.8\nPillow==4.0.0\npsycopg2==2.7.1\npyparsing==2.2.0\nsix==1.10.0\n",[30,90458,90456],{"__ignoreMap":464},[11,90460,90461,90462,208],{},"And then we can add these to a file in our base directory called ",[30,90463,38577],{},[459,90465,90468],{"className":90466,"code":90467,"language":997},[995],"(proj) $ pip freeze > requirements.txt\n",[30,90469,90467],{"__ignoreMap":464},[11,90471,90472],{},"Next we can run migrations and create a superuser:",[459,90474,90477],{"className":90475,"code":90476,"language":997},[995],"(proj) $ python manage.py migrate\n(proj) $ python manage.py createsuperuser\n",[30,90478,90476],{"__ignoreMap":464},[11,90480,90481,90482,208],{},"Next we need to initialize our git repository and create ",[30,90483,12546],{},[459,90485,90488],{"className":90486,"code":90487,"language":997},[995],"(proj) $ git init\n",[30,90489,90487],{"__ignoreMap":464},[11,90491,90492,90493,90495],{},"We can put ",[30,90494,12546],{}," in our base directory and add the following:",[459,90497,90500],{"className":90498,"code":90499,"language":997,"meta":464},[995],"myproj/settings/local.py\n",[30,90501,90499],{"__ignoreMap":464},[11,90503,90504,90505,643],{},"We also want to ignore several other python-related files in our directory. An easy way to do this is to add python gitignore. This can be found ",[20,90506,13074],{"href":90507,"rel":90508},"https://raw.githubusercontent.com/github/gitignore/master/Python.gitignore",[24],[11,90510,90511],{},"Next we can make our first commit:",[459,90513,90516],{"className":90514,"code":90515,"language":997},[995],"(proj) $ git add --all\n(proj) $ git commit -m \"initial commit\"\n",[30,90517,90515],{"__ignoreMap":464},[11,90519,90520,90521,90524],{},"The next step involves setting up Heroku. First we need to create a ",[30,90522,90523],{},"Procfile"," in our base directory:",[11,90526,90527],{},[51,90528,90523],{},[459,90530,90533],{"className":90531,"code":90532,"language":997,"meta":464},[995],"web: gunicorn myproj.wsgi --log-file -\n",[30,90534,90532],{"__ignoreMap":464},[11,90536,90537,90538,313,90541,10744,90544,106,90546,187,90548,643],{},"Next we can try to run Heroku locally, but first we need to add ",[30,90539,90540],{},"0.0.0.0",[30,90542,90543],{},"ALLOWED_HOSTS",[30,90545,90431],{},[30,90547,90421],{},[30,90549,90428],{},[11,90551,90552,90553,90556],{},"We should now see \"It worked!\" at ",[30,90554,90555],{},"0.0.0.0:5000",", which tells us that everything is working properly.",[11,90558,90559],{},"Next we need to create the project on Heroku:",[459,90561,90564],{"className":90562,"code":90563,"language":997},[995],"(proj) $ heroku login\n(proj) $ heroku create my-unique-project-name-123\n",[30,90565,90563],{"__ignoreMap":464},[11,90567,90568,90569,90572,90573],{},"Now we can see our project at ",[30,90570,90571],{},"my-unique-project-name-123.herokuapp.com",", and it should say ",[30,90574,90575],{},"Heroku | Welcome to your new app!",[11,90577,90578,90579,313,90581,10744,90583,643],{},"Next we will want to add ",[30,90580,90571],{},[30,90582,90543],{},[30,90584,50465],{},[11,90586,90587,90588,90524],{},"The very last step is to add the specific version of Python to a file called ",[30,90589,90590],{},"runtime.txt",[459,90592,90595],{"className":90593,"code":90594,"language":997},[995],"(proj) $ python -V\nPython 3.4.3\n(proj) $ echo \"python-3.4.3\" > runtime.txt\n",[30,90596,90594],{"__ignoreMap":464},[11,90598,90599],{},"Before we push to Heroku we need to change a setting on Heroku related to static files:",[459,90601,90604],{"className":90602,"code":90603,"language":997},[995],"(proj) $ heroku config:set DISABLE_COLLECTSTATIC=1\n",[30,90605,90603],{"__ignoreMap":464},[11,90607,90608],{},"Now we can finally push the git repository to Heroku:",[459,90610,90613],{"className":90611,"code":90612,"language":997},[995],"(proj) $ git push heroku master\n",[30,90614,90612],{"__ignoreMap":464},[11,90616,90617],{},"Now if we go to our site on heroku we should see:",[459,90619,90622],{"className":90620,"code":90621,"language":997},[995],"Not Found\n\nThe requested URL / was not found on this server.\n",[30,90623,90621],{"__ignoreMap":464},[11,90625,90626,90627,643],{},"There is a helpful guide on deploying Python and Django apps on Heroku's website ",[20,90628,13074],{"href":90629,"rel":90630},"https://devcenter.heroku.com/articles/deploying-python",[24],[11,90632,90633],{},"Here's an important excerpt regarding databases:",[210,90635,90636],{},[11,90637,90638],{},"For Django applications, a Heroku Postgres hobby-dev database is automatically provisioned. This populates the DATABASE_URL environment variable.\nNo add-ons are automatically provisioned if a pure Python application is detected. If you need a SQL database for your app, add one explicitly:",[459,90640,90643],{"className":90641,"code":90642,"language":997},[995],"$ heroku addons:create heroku-postgresql:hobby-dev\n",[30,90644,90642],{"__ignoreMap":464},[11,90646,90647],{},"So we need to run this command:",[459,90649,90652],{"className":90650,"code":90651,"language":997},[995],"\n(proj) $ heroku addons:create heroku-postgresql:hobby-dev\n\n",[30,90653,90651],{"__ignoreMap":464},[11,90655,90656],{},"Next we need to access the terminal on our Heroku server:",[459,90658,90661],{"className":90659,"code":90660,"language":997},[995],"(proj) $ heroku run bash\n",[30,90662,90660],{"__ignoreMap":464},[11,90664,90665,90666,37216,90668,90670],{},"Next we need to add database-related information to ",[30,90667,90431],{},[30,90669,50560],{}," section:",[459,90672,90674],{"className":13136,"code":90673,"language":12886,"meta":464,"style":464},"[...]\nimport dj_database_url\n\ndb_from_env = dj_database_url.config()\nDATABASES['default'].update(db_from_env)\n[...]\n",[30,90675,90676,90684,90691,90695,90705,90717],{"__ignoreMap":464},[151,90677,90678,90680,90682],{"class":469,"line":470},[151,90679,6698],{"class":503},[151,90681,27455],{"class":477},[151,90683,3691],{"class":503},[151,90685,90686,90688],{"class":469,"line":488},[151,90687,16859],{"class":1869},[151,90689,90690],{"class":503}," dj_database_url\n",[151,90692,90693],{"class":469,"line":500},[151,90694,1090],{"emptyLinePlaceholder":609},[151,90696,90697,90700,90702],{"class":469,"line":509},[151,90698,90699],{"class":503},"db_from_env ",[151,90701,1876],{"class":1869},[151,90703,90704],{"class":503}," dj_database_url.config()\n",[151,90706,90707,90709,90711,90714],{"class":469,"line":517},[151,90708,50560],{"class":477},[151,90710,6698],{"class":503},[151,90712,90713],{"class":481},"'default'",[151,90715,90716],{"class":503},"].update(db_from_env)\n",[151,90718,90719,90721,90723],{"class":469,"line":534},[151,90720,6698],{"class":503},[151,90722,27455],{"class":477},[151,90724,3691],{"class":503},[11,90726,90727],{},"Now we can push to Heroku:",[459,90729,90731],{"className":90730,"code":90612,"language":997},[995],[30,90732,90612],{"__ignoreMap":464},[11,90734,90735],{},"Next we can run our migrations:",[459,90737,90740],{"className":90738,"code":90739,"language":997},[995],"(proj) $ heroku run python manage.py makemigrations\n",[30,90741,90739],{"__ignoreMap":464},[11,90743,90744,90745,187,90747,90749],{},"Next we can run ",[30,90746,27589],{},[30,90748,51591],{}," on our Heroku server:",[459,90751,90754],{"className":90752,"code":90753,"language":997},[995],"(proj) $ heroku run python manage.py migrate && heroku run python manage.py createsuperuser\n",[30,90755,90753],{"__ignoreMap":464},[11,90757,90758],{},"Now we should be able to login to the admin page with the account we just created, but CSS is still not working at this point. Here's how we configure static files to get CSS to work:",[459,90760,90763],{"className":90761,"code":90762,"language":997},[995],"(proj) $ pip install whitenoise\n",[30,90764,90762],{"__ignoreMap":464},[11,90766,90767,90770],{},[30,90768,90769],{},"whitenoise"," is needed so Heroku can run our static files.",[11,90772,90773,90774,10744,90776,208],{},"Next we want to make sure to include ",[30,90775,90769],{},[30,90777,38577],{},[459,90779,90781],{"className":90780,"code":90467,"language":997},[995],[30,90782,90467],{"__ignoreMap":464},[11,90784,90785,90786,90789,90790,90792,90793,90795,90796,208],{},"Next we need to add the following item to the list of ",[30,90787,90788],{},"MIDDLEWARE"," components. This needs to be added to BOTH ",[30,90791,90431],{}," AND ",[30,90794,90421],{}," in order for the local files to show on both our heroku site and the locally served site with ",[30,90797,90798],{},"heroku local web",[459,90800,90803],{"className":90801,"code":90802,"language":997,"meta":464},[995],"MIDDLEWARE = [\n[...]\n'whitenoise.middleware.WhiteNoiseMiddleware',\n[...]\n]\n",[30,90804,90802],{"__ignoreMap":464},[11,90806,90807],{},"Next we have a few more items to add to our settings files:",[11,90809,90810],{},[51,90811,90431],{},[459,90813,90815],{"className":13136,"code":90814,"language":12886,"meta":464,"style":464},"\nSTATICFILES_DIRS = (\n    os.path.join(BASE_DIR, \"static\"),\n)\n\nSTATIC_ROOT = os.path.join(BASE_DIR, \"live-static\", \"static-root\")\n\nSTATICFILES_STORAGE = 'whitenoise.django.GzipManifestStaticFilesStorage'\n\n#STATIC_ROOT = \"/home/cfedeploy/webapps/cfehome_static_root/\"\n\nMEDIA_URL = \"/media/\"\n\nMEDIA_ROOT = os.path.join(BASE_DIR, \"live-static\", \"media-root\")\n\n",[30,90816,90817,90821,90830,90843,90847,90851,90873,90877,90886,90890,90895,90899,90909,90913],{"__ignoreMap":464},[151,90818,90819],{"class":469,"line":470},[151,90820,1090],{"emptyLinePlaceholder":609},[151,90822,90823,90826,90828],{"class":469,"line":488},[151,90824,90825],{"class":477},"STATICFILES_DIRS",[151,90827,19865],{"class":1869},[151,90829,37723],{"class":503},[151,90831,90832,90835,90837,90839,90841],{"class":469,"line":500},[151,90833,90834],{"class":503},"    os.path.join(",[151,90836,50479],{"class":477},[151,90838,106],{"class":503},[151,90840,50489],{"class":481},[151,90842,37985],{"class":503},[151,90844,90845],{"class":469,"line":509},[151,90846,3640],{"class":503},[151,90848,90849],{"class":469,"line":517},[151,90850,1090],{"emptyLinePlaceholder":609},[151,90852,90853,90855,90857,90859,90861,90863,90866,90868,90871],{"class":469,"line":534},[151,90854,50461],{"class":477},[151,90856,19865],{"class":1869},[151,90858,53082],{"class":503},[151,90860,50479],{"class":477},[151,90862,106],{"class":503},[151,90864,90865],{"class":481},"\"live-static\"",[151,90867,106],{"class":503},[151,90869,90870],{"class":481},"\"static-root\"",[151,90872,3640],{"class":503},[151,90874,90875],{"class":469,"line":1413},[151,90876,1090],{"emptyLinePlaceholder":609},[151,90878,90879,90881,90883],{"class":469,"line":1418},[151,90880,53096],{"class":477},[151,90882,19865],{"class":1869},[151,90884,90885],{"class":481}," 'whitenoise.django.GzipManifestStaticFilesStorage'\n",[151,90887,90888],{"class":469,"line":2462},[151,90889,1090],{"emptyLinePlaceholder":609},[151,90891,90892],{"class":469,"line":2471},[151,90893,90894],{"class":1527},"#STATIC_ROOT = \"/home/cfedeploy/webapps/cfehome_static_root/\"\n",[151,90896,90897],{"class":469,"line":2480},[151,90898,1090],{"emptyLinePlaceholder":609},[151,90900,90901,90904,90906],{"class":469,"line":2489},[151,90902,90903],{"class":477},"MEDIA_URL",[151,90905,19865],{"class":1869},[151,90907,90908],{"class":481}," \"/media/\"\n",[151,90910,90911],{"class":469,"line":2497},[151,90912,1090],{"emptyLinePlaceholder":609},[151,90914,90915,90918,90920,90922,90924,90926,90928,90930,90933],{"class":469,"line":3140},[151,90916,90917],{"class":477},"MEDIA_ROOT",[151,90919,19865],{"class":1869},[151,90921,53082],{"class":503},[151,90923,50479],{"class":477},[151,90925,106],{"class":503},[151,90927,90865],{"class":481},[151,90929,106],{"class":503},[151,90931,90932],{"class":481},"\"media-root\"",[151,90934,3640],{"class":503},[11,90936,90937],{},"And we also need to add some files to our base directory that will hold our static files:",[459,90939,90942],{"className":90940,"code":90941,"language":997},[995],"(proj) $ mkdir static\n(proj) $ echo \"body {color:#000;}\" > static/main.css\n(proj) $ mkdir live-static\n(proj) $ mkdir live-static/static-root\n(proj) $ mkdir live-static/media-root\n(proj) $\n(proj) $\n\n",[30,90943,90941],{"__ignoreMap":464},[11,90945,90946,90947,208],{},"And we also need to add the following to the end of ",[30,90948,90431],{},[459,90950,90952],{"className":13136,"code":90951,"language":12886,"meta":464,"style":464},"[...]\nSTATIC_URL = '/static/'\n\nSTATICFILES_DIRS = (\n    os.path.join(BASE_DIR, \"static\"),\n)\n\nSTATIC_ROOT = os.path.join(BASE_DIR, \"live-static\", \"static-root\")\n\nSTATICFILES_STORAGE = 'whitenoise.django.GzipManifestStaticFilesStorage'\n\n#STATIC_ROOT = \"/home/cfedeploy/webapps/cfehome_static_root/\"\n\nMEDIA_URL = \"/media/\"\n\nMEDIA_ROOT = os.path.join(BASE_DIR, \"live-static\", \"media-root\")\n",[30,90953,90954,90962,90970,90974,90982,90994,90998,91002,91022,91026,91034,91038,91042,91046,91054,91058],{"__ignoreMap":464},[151,90955,90956,90958,90960],{"class":469,"line":470},[151,90957,6698],{"class":503},[151,90959,27455],{"class":477},[151,90961,3691],{"class":503},[151,90963,90964,90966,90968],{"class":469,"line":488},[151,90965,53068],{"class":477},[151,90967,19865],{"class":1869},[151,90969,53073],{"class":481},[151,90971,90972],{"class":469,"line":500},[151,90973,1090],{"emptyLinePlaceholder":609},[151,90975,90976,90978,90980],{"class":469,"line":509},[151,90977,90825],{"class":477},[151,90979,19865],{"class":1869},[151,90981,37723],{"class":503},[151,90983,90984,90986,90988,90990,90992],{"class":469,"line":517},[151,90985,90834],{"class":503},[151,90987,50479],{"class":477},[151,90989,106],{"class":503},[151,90991,50489],{"class":481},[151,90993,37985],{"class":503},[151,90995,90996],{"class":469,"line":534},[151,90997,3640],{"class":503},[151,90999,91000],{"class":469,"line":1413},[151,91001,1090],{"emptyLinePlaceholder":609},[151,91003,91004,91006,91008,91010,91012,91014,91016,91018,91020],{"class":469,"line":1418},[151,91005,50461],{"class":477},[151,91007,19865],{"class":1869},[151,91009,53082],{"class":503},[151,91011,50479],{"class":477},[151,91013,106],{"class":503},[151,91015,90865],{"class":481},[151,91017,106],{"class":503},[151,91019,90870],{"class":481},[151,91021,3640],{"class":503},[151,91023,91024],{"class":469,"line":2462},[151,91025,1090],{"emptyLinePlaceholder":609},[151,91027,91028,91030,91032],{"class":469,"line":2471},[151,91029,53096],{"class":477},[151,91031,19865],{"class":1869},[151,91033,90885],{"class":481},[151,91035,91036],{"class":469,"line":2480},[151,91037,1090],{"emptyLinePlaceholder":609},[151,91039,91040],{"class":469,"line":2489},[151,91041,90894],{"class":1527},[151,91043,91044],{"class":469,"line":2497},[151,91045,1090],{"emptyLinePlaceholder":609},[151,91047,91048,91050,91052],{"class":469,"line":3140},[151,91049,90903],{"class":477},[151,91051,19865],{"class":1869},[151,91053,90908],{"class":481},[151,91055,91056],{"class":469,"line":3149},[151,91057,1090],{"emptyLinePlaceholder":609},[151,91059,91060,91062,91064,91066,91068,91070,91072,91074,91076],{"class":469,"line":3158},[151,91061,90917],{"class":477},[151,91063,19865],{"class":1869},[151,91065,53082],{"class":503},[151,91067,50479],{"class":477},[151,91069,106],{"class":503},[151,91071,90865],{"class":481},[151,91073,106],{"class":503},[151,91075,90932],{"class":481},[151,91077,3640],{"class":503},[11,91079,91080,187,91082,91084],{},[30,91081,90421],{},[30,91083,90428],{}," should have the following:",[459,91086,91088],{"className":13136,"code":91087,"language":12886,"meta":464,"style":464},"STATIC_URL = '/static/'\n\nSTATICFILES_DIRS = (\n    os.path.join(BASE_DIR, \"static\"),\n)\n\nSTATIC_ROOT = os.path.join(BASE_DIR, \"live-static\", \"static-root\")\n\nMEDIA_URL = \"/media/\"\n\nMEDIA_ROOT = os.path.join(BASE_DIR, \"live-static\", \"media-root\")\n",[30,91089,91090,91098,91102,91110,91122,91126,91130,91150,91154,91162,91166],{"__ignoreMap":464},[151,91091,91092,91094,91096],{"class":469,"line":470},[151,91093,53068],{"class":477},[151,91095,19865],{"class":1869},[151,91097,53073],{"class":481},[151,91099,91100],{"class":469,"line":488},[151,91101,1090],{"emptyLinePlaceholder":609},[151,91103,91104,91106,91108],{"class":469,"line":500},[151,91105,90825],{"class":477},[151,91107,19865],{"class":1869},[151,91109,37723],{"class":503},[151,91111,91112,91114,91116,91118,91120],{"class":469,"line":509},[151,91113,90834],{"class":503},[151,91115,50479],{"class":477},[151,91117,106],{"class":503},[151,91119,50489],{"class":481},[151,91121,37985],{"class":503},[151,91123,91124],{"class":469,"line":517},[151,91125,3640],{"class":503},[151,91127,91128],{"class":469,"line":534},[151,91129,1090],{"emptyLinePlaceholder":609},[151,91131,91132,91134,91136,91138,91140,91142,91144,91146,91148],{"class":469,"line":1413},[151,91133,50461],{"class":477},[151,91135,19865],{"class":1869},[151,91137,53082],{"class":503},[151,91139,50479],{"class":477},[151,91141,106],{"class":503},[151,91143,90865],{"class":481},[151,91145,106],{"class":503},[151,91147,90870],{"class":481},[151,91149,3640],{"class":503},[151,91151,91152],{"class":469,"line":1418},[151,91153,1090],{"emptyLinePlaceholder":609},[151,91155,91156,91158,91160],{"class":469,"line":2462},[151,91157,90903],{"class":477},[151,91159,19865],{"class":1869},[151,91161,90908],{"class":481},[151,91163,91164],{"class":469,"line":2471},[151,91165,1090],{"emptyLinePlaceholder":609},[151,91167,91168,91170,91172,91174,91176,91178,91180,91182,91184],{"class":469,"line":2480},[151,91169,90917],{"class":477},[151,91171,19865],{"class":1869},[151,91173,53082],{"class":503},[151,91175,50479],{"class":477},[151,91177,106],{"class":503},[151,91179,90865],{"class":481},[151,91181,106],{"class":503},[151,91183,90932],{"class":481},[151,91185,3640],{"class":503},[11,91187,91188,91189,91191],{},"Next we want to run ",[30,91190,27586],{}," locally:",[459,91193,91196],{"className":91194,"code":91195,"language":997},[995],"(proj) $ python manage.py collectstatic\n",[30,91197,91195],{"__ignoreMap":464},[11,91199,91200,91201,91204],{},"We also want to add a blank file to ",[30,91202,91203],{},"live-static/media-root"," so that it becomes tracked in git:",[459,91206,91209],{"className":91207,"code":91208,"language":997},[995],"(proj) $ echo \"some text\" > live-static/media-root/blank.txt\n",[30,91210,91208],{"__ignoreMap":464},[11,91212,91213,91214,313,91217,208],{},"Next we can commit these changes and push to Heroku, and check to see if the static files are working in the admin panel. We also want to set ",[30,91215,91216],{},"DISABLE_COLLECTSTATIC",[30,91218,9181],{},[459,91220,91223],{"className":91221,"code":91222,"language":997},[995],"(proj) $ git add --all\n(proj) $ git commit -m \"added static files\"\n(proj) $ git push heroku master\n(proj) $ heroku config:set DISABLE_COLLECTSTATIC=0\n",[30,91224,91222],{"__ignoreMap":464},[11,91226,91227],{},"Now we should be able to see the admin panel with working CSS on both the live and local heroku sites.",[14063,91229,91231],{"id":91230},"adding-an-app-and-configuring-bootstrap","Adding an app and configuring Bootstrap",[11,91233,91234],{},"Now that everything seems to be working we can start building our app.",[11,91236,91237],{},"Let's start by creating a new app:",[459,91239,91242],{"className":91240,"code":91241,"language":997},[995],"(proj) $ python manage.py startapp pages\n",[30,91243,91241],{"__ignoreMap":464},[11,91245,91246,91248],{},[30,91247,35341],{}," will be the name of an app that we create here.",[11,91250,91251,91252,91255],{},"In ",[30,91253,91254],{},"pages/views.py"," we can add a class-based view that will serve as the homepage:",[459,91257,91259],{"className":13136,"code":91258,"language":12886,"meta":464,"style":464},"from django.shortcuts import render\nfrom django.views.generic import View\n# Create your views here.\n\nclass HomeView(View):\n    def get(self, request, *args, **kwargs):\n        return render(request, 'pages/home.html', {})\n\n",[30,91260,91261,91273,91285,91290,91294,91308,91337],{"__ignoreMap":464},[151,91262,91263,91265,91268,91270],{"class":469,"line":470},[151,91264,16853],{"class":1869},[151,91266,91267],{"class":503}," django.shortcuts ",[151,91269,16859],{"class":1869},[151,91271,91272],{"class":503}," render\n",[151,91274,91275,91277,91280,91282],{"class":469,"line":488},[151,91276,16853],{"class":1869},[151,91278,91279],{"class":503}," django.views.generic ",[151,91281,16859],{"class":1869},[151,91283,91284],{"class":503}," View\n",[151,91286,91287],{"class":469,"line":500},[151,91288,91289],{"class":1527},"# Create your views here.\n",[151,91291,91292],{"class":469,"line":509},[151,91293,1090],{"emptyLinePlaceholder":609},[151,91295,91296,91298,91301,91303,91306],{"class":469,"line":517},[151,91297,16519],{"class":12347},[151,91299,91300],{"class":15254}," HomeView",[151,91302,12386],{"class":503},[151,91304,91305],{"class":15260},"View",[151,91307,15264],{"class":503},[151,91309,91310,91312,91315,91317,91319,91321,91323,91325,91327,91329,91331,91333,91335],{"class":469,"line":534},[151,91311,16566],{"class":12347},[151,91313,91314],{"class":473}," get",[151,91316,12386],{"class":503},[151,91318,15277],{"class":15232},[151,91320,106],{"class":503},[151,91322,59686],{"class":15232},[151,91324,106],{"class":503},[151,91326,23268],{"class":1869},[151,91328,40395],{"class":15232},[151,91330,106],{"class":503},[151,91332,24677],{"class":1869},[151,91334,37866],{"class":15232},[151,91336,15264],{"class":503},[151,91338,91339,91341,91343,91346],{"class":469,"line":1413},[151,91340,16833],{"class":1869},[151,91342,59902],{"class":503},[151,91344,91345],{"class":481},"'pages/home.html'",[151,91347,91348],{"class":503},", {})\n",[11,91350,91351,91352,91354],{},"We then update ",[30,91353,53781],{}," to include our new view:",[459,91356,91358],{"className":13136,"code":91357,"language":12886,"meta":464,"style":464},"from django.conf.urls import url\nfrom django.contrib import admin\nfrom services.views import HomeView\n\nurlpatterns = [\n    url(r'^admin/', admin.site.urls),\n    url(r'^$', HomeView.as_view(), name='home'),\n]\n\n",[30,91359,91360,91372,91384,91396,91400,91409,91429,91454],{"__ignoreMap":464},[151,91361,91362,91364,91367,91369],{"class":469,"line":470},[151,91363,16853],{"class":1869},[151,91365,91366],{"class":503}," django.conf.urls ",[151,91368,16859],{"class":1869},[151,91370,91371],{"class":503}," url\n",[151,91373,91374,91376,91379,91381],{"class":469,"line":488},[151,91375,16853],{"class":1869},[151,91377,91378],{"class":503}," django.contrib ",[151,91380,16859],{"class":1869},[151,91382,91383],{"class":503}," admin\n",[151,91385,91386,91388,91391,91393],{"class":469,"line":500},[151,91387,16853],{"class":1869},[151,91389,91390],{"class":503}," services.views ",[151,91392,16859],{"class":1869},[151,91394,91395],{"class":503}," HomeView\n",[151,91397,91398],{"class":469,"line":509},[151,91399,1090],{"emptyLinePlaceholder":609},[151,91401,91402,91405,91407],{"class":469,"line":517},[151,91403,91404],{"class":503},"urlpatterns ",[151,91406,1876],{"class":1869},[151,91408,13149],{"class":503},[151,91410,91411,91414,91416,91418,91420,91424,91426],{"class":469,"line":534},[151,91412,91413],{"class":503},"    url(",[151,91415,58741],{"class":12347},[151,91417,13223],{"class":481},[151,91419,57705],{"class":58746},[151,91421,91423],{"class":91422},"sFxd3","admin/",[151,91425,13223],{"class":481},[151,91427,91428],{"class":503},", admin.site.urls),\n",[151,91430,91431,91433,91435,91437,91440,91442,91445,91447,91449,91452],{"class":469,"line":1413},[151,91432,91413],{"class":503},[151,91434,58741],{"class":12347},[151,91436,13223],{"class":481},[151,91438,91439],{"class":58746},"^$",[151,91441,13223],{"class":481},[151,91443,91444],{"class":503},", HomeView.as_view(), ",[151,91446,20415],{"class":15210},[151,91448,1876],{"class":1869},[151,91450,91451],{"class":481},"'home'",[151,91453,37985],{"class":503},[151,91455,91456],{"class":469,"line":1418},[151,91457,3691],{"class":503},[11,91459,91460,91461,313,91463,91466,91467,187,91469,643],{},"Next we have to add ",[30,91462,35341],{},[30,91464,91465],{},"INSTALLED_APPS"," in both ",[30,91468,90431],{},[30,91470,90428],{},[11,91472,91473,91474,91476],{},"Next we need to make some folders within our new ",[30,91475,35341],{}," app:",[459,91478,91481],{"className":91479,"code":91480,"language":997},[995],"(proj) $ mkdir page/templates && cd pages/templates\n(proj) $ mkdir pages\n(proj) $ touch home.html\n",[30,91482,91480],{"__ignoreMap":464},[11,91484,91485,91486,208],{},"Then we need to add the following to ",[30,91487,91488],{},"home.html",[459,91490,91492],{"className":19811,"code":91491,"language":19813,"meta":464,"style":464},"{% extends \"base.html\" %} {% block content %} {% endblock content%}\n",[30,91493,91494],{"__ignoreMap":464},[151,91495,91496],{"class":469,"line":470},[151,91497,91491],{"class":503},[11,91499,91500,91501,10744,91503,10744,91506,106,91508,187,91510,208],{},"Next we need to update ",[30,91502,64221],{},[30,91504,91505],{},"TEMPLATES",[30,91507,90421],{},[30,91509,90428],{},[30,91511,90431],{},[459,91513,91515],{"className":13136,"code":91514,"language":12886,"meta":464,"style":464},"[...]\n'DIRS': [os.path.join(BASE_DIR, 'templates')],\n[...]\n",[30,91516,91517,91525,91542],{"__ignoreMap":464},[151,91518,91519,91521,91523],{"class":469,"line":470},[151,91520,6698],{"class":503},[151,91522,27455],{"class":477},[151,91524,3691],{"class":503},[151,91526,91527,91530,91533,91535,91537,91540],{"class":469,"line":488},[151,91528,91529],{"class":481},"'DIRS'",[151,91531,91532],{"class":503},": [os.path.join(",[151,91534,50479],{"class":477},[151,91536,106],{"class":503},[151,91538,91539],{"class":481},"'templates'",[151,91541,56006],{"class":503},[151,91543,91544,91546,91548],{"class":469,"line":500},[151,91545,6698],{"class":503},[151,91547,27455],{"class":477},[151,91549,3691],{"class":503},[11,91551,91552,91553,91556],{},"Next we need to add a ",[30,91554,91555],{},"templates"," folder to the root of our project:",[459,91558,91561],{"className":91559,"code":91560,"language":997},[995],"(proj) $ mkdir templates\n(proj) $ touch tempates/base.html\n",[30,91562,91560],{"__ignoreMap":464},[11,91564,91565,91566,208],{},"Next we can add a basic Bootstrap template to ",[30,91567,91568],{},"base.html",[459,91570,91572],{"className":19811,"code":91571,"language":19813,"meta":464,"style":464},"\u003C!DOCTYPE html>\n\u003Chtml lang=\"en\">\n  \u003Chead>\n    \u003Cmeta charset=\"utf-8\" />\n    \u003Cmeta http-equiv=\"X-UA-Compatible\" content=\"IE=edge\" />\n    \u003Cmeta name=\"viewport\" content=\"width=device-width, initial-scale=1\" />\n    \u003C!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags -->\n    \u003Ctitle>Bootstrap 101 Template\u003C/title>\n\n    \u003C!-- Latest compiled and minified CSS -->\n    \u003Clink\n      rel=\"stylesheet\"\n      href=\"https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css\"\n      integrity=\"sha384-BVYiiSIFeK1dGmJRAkycuHAHRg32OmUcww7on3RYdg4Va+PmSTsz/K68vbdEjh4u\"\n      crossorigin=\"anonymous\"\n    />\n\n    \u003C!-- Optional theme -->\n    \u003Clink\n      rel=\"stylesheet\"\n      href=\"https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap-theme.min.css\"\n      integrity=\"sha384-rHyoN1iRsVXV4nD0JutlnGaslCJuC7uwjduW9SVrLvRYooPp2bWYgmgJQIXwl/Sp\"\n      crossorigin=\"anonymous\"\n    />\n\n    \u003C!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->\n    \u003C!-- WARNING: Respond.js doesn't work if you view the page via file:// -->\n    \u003C!--[if lt IE 9]>\n      \u003Cscript src=\"https://oss.maxcdn.com/html5shiv/3.7.3/html5shiv.min.js\">\u003C/script>\n      \u003Cscript src=\"https://oss.maxcdn.com/respond/1.4.2/respond.min.js\">\u003C/script>\n    \u003C![endif]-->\n  \u003C/head>\n  \u003Cbody>\n    \u003Ch1>Hello, world!\u003C/h1>\n\n    \u003C!-- jQuery (necessary for Bootstrap's JavaScript plugins) -->\n    \u003Cscript src=\"https://ajax.googleapis.com/ajax/libs/jquery/1.12.4/jquery.min.js\">\u003C/script>\n    \u003C!-- Include all compiled plugins (below), or include individual files as needed -->\n    \u003C!-- Latest compiled and minified JavaScript -->\n    \u003Cscript\n      src=\"https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js\"\n      integrity=\"sha384-Tc5IQib027qvyjSMfHjOMaLkfuWVxZxUPnCJA7l2mCWNIpG9mGCD8wGNIcPD7Txa\"\n      crossorigin=\"anonymous\"\n    >\u003C/script>\n  \u003C/body>\n\u003C/html>\n",[30,91573,91574,91587,91603,91611,91628,91652,91674,91679,91692,91696,91701,91708,91718,91728,91738,91748,91753,91757,91762,91768,91776,91785,91794,91802,91806,91810,91815,91820,91825,91830,91835,91840,91848,91857,91870,91874,91879,91899,91904,91909,91916,91926,91935,91943,91951,91959],{"__ignoreMap":464},[151,91575,91576,91579,91582,91585],{"class":469,"line":470},[151,91577,91578],{"class":503},"\u003C!",[151,91580,91581],{"class":14368},"DOCTYPE",[151,91583,91584],{"class":473}," html",[151,91586,3742],{"class":503},[151,91588,91589,91591,91593,91596,91598,91601],{"class":469,"line":488},[151,91590,3613],{"class":503},[151,91592,19813],{"class":14368},[151,91594,91595],{"class":473}," lang",[151,91597,1876],{"class":503},[151,91599,91600],{"class":481},"\"en\"",[151,91602,3742],{"class":503},[151,91604,91605,91607,91609],{"class":469,"line":500},[151,91606,33991],{"class":503},[151,91608,20975],{"class":14368},[151,91610,3742],{"class":503},[151,91612,91613,91615,91618,91621,91623,91626],{"class":469,"line":509},[151,91614,34669],{"class":503},[151,91616,91617],{"class":14368},"meta",[151,91619,91620],{"class":473}," charset",[151,91622,1876],{"class":503},[151,91624,91625],{"class":481},"\"utf-8\"",[151,91627,34675],{"class":503},[151,91629,91630,91632,91634,91637,91639,91642,91645,91647,91650],{"class":469,"line":517},[151,91631,34669],{"class":503},[151,91633,91617],{"class":14368},[151,91635,91636],{"class":473}," http-equiv",[151,91638,1876],{"class":503},[151,91640,91641],{"class":481},"\"X-UA-Compatible\"",[151,91643,91644],{"class":473}," content",[151,91646,1876],{"class":503},[151,91648,91649],{"class":481},"\"IE=edge\"",[151,91651,34675],{"class":503},[151,91653,91654,91656,91658,91660,91662,91665,91667,91669,91672],{"class":469,"line":534},[151,91655,34669],{"class":503},[151,91657,91617],{"class":14368},[151,91659,61015],{"class":473},[151,91661,1876],{"class":503},[151,91663,91664],{"class":481},"\"viewport\"",[151,91666,91644],{"class":473},[151,91668,1876],{"class":503},[151,91670,91671],{"class":481},"\"width=device-width, initial-scale=1\"",[151,91673,34675],{"class":503},[151,91675,91676],{"class":469,"line":1413},[151,91677,91678],{"class":1527},"    \u003C!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags -->\n",[151,91680,91681,91683,91685,91688,91690],{"class":469,"line":1418},[151,91682,34669],{"class":503},[151,91684,19633],{"class":14368},[151,91686,91687],{"class":503},">Bootstrap 101 Template\u003C/",[151,91689,19633],{"class":14368},[151,91691,3742],{"class":503},[151,91693,91694],{"class":469,"line":2462},[151,91695,1090],{"emptyLinePlaceholder":609},[151,91697,91698],{"class":469,"line":2471},[151,91699,91700],{"class":1527},"    \u003C!-- Latest compiled and minified CSS -->\n",[151,91702,91703,91705],{"class":469,"line":2480},[151,91704,34669],{"class":503},[151,91706,91707],{"class":14368},"link\n",[151,91709,91710,91713,91715],{"class":469,"line":2489},[151,91711,91712],{"class":473},"      rel",[151,91714,1876],{"class":503},[151,91716,91717],{"class":481},"\"stylesheet\"\n",[151,91719,91720,91723,91725],{"class":469,"line":2497},[151,91721,91722],{"class":473},"      href",[151,91724,1876],{"class":503},[151,91726,91727],{"class":481},"\"https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css\"\n",[151,91729,91730,91733,91735],{"class":469,"line":3140},[151,91731,91732],{"class":473},"      integrity",[151,91734,1876],{"class":503},[151,91736,91737],{"class":481},"\"sha384-BVYiiSIFeK1dGmJRAkycuHAHRg32OmUcww7on3RYdg4Va+PmSTsz/K68vbdEjh4u\"\n",[151,91739,91740,91743,91745],{"class":469,"line":3149},[151,91741,91742],{"class":473},"      crossorigin",[151,91744,1876],{"class":503},[151,91746,91747],{"class":481},"\"anonymous\"\n",[151,91749,91750],{"class":469,"line":3158},[151,91751,91752],{"class":503},"    />\n",[151,91754,91755],{"class":469,"line":3167},[151,91756,1090],{"emptyLinePlaceholder":609},[151,91758,91759],{"class":469,"line":3175},[151,91760,91761],{"class":1527},"    \u003C!-- Optional theme -->\n",[151,91763,91764,91766],{"class":469,"line":3184},[151,91765,34669],{"class":503},[151,91767,91707],{"class":14368},[151,91769,91770,91772,91774],{"class":469,"line":3193},[151,91771,91712],{"class":473},[151,91773,1876],{"class":503},[151,91775,91717],{"class":481},[151,91777,91778,91780,91782],{"class":469,"line":3720},[151,91779,91722],{"class":473},[151,91781,1876],{"class":503},[151,91783,91784],{"class":481},"\"https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap-theme.min.css\"\n",[151,91786,91787,91789,91791],{"class":469,"line":3729},[151,91788,91732],{"class":473},[151,91790,1876],{"class":503},[151,91792,91793],{"class":481},"\"sha384-rHyoN1iRsVXV4nD0JutlnGaslCJuC7uwjduW9SVrLvRYooPp2bWYgmgJQIXwl/Sp\"\n",[151,91795,91796,91798,91800],{"class":469,"line":3735},[151,91797,91742],{"class":473},[151,91799,1876],{"class":503},[151,91801,91747],{"class":481},[151,91803,91804],{"class":469,"line":3745},[151,91805,91752],{"class":503},[151,91807,91808],{"class":469,"line":3754},[151,91809,1090],{"emptyLinePlaceholder":609},[151,91811,91812],{"class":469,"line":3760},[151,91813,91814],{"class":1527},"    \u003C!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->\n",[151,91816,91817],{"class":469,"line":3773},[151,91818,91819],{"class":1527},"    \u003C!-- WARNING: Respond.js doesn't work if you view the page via file:// -->\n",[151,91821,91822],{"class":469,"line":3782},[151,91823,91824],{"class":1527},"    \u003C!--[if lt IE 9]>\n",[151,91826,91827],{"class":469,"line":3791},[151,91828,91829],{"class":1527},"      \u003Cscript src=\"https://oss.maxcdn.com/html5shiv/3.7.3/html5shiv.min.js\">\u003C/script>\n",[151,91831,91832],{"class":469,"line":3803},[151,91833,91834],{"class":1527},"      \u003Cscript src=\"https://oss.maxcdn.com/respond/1.4.2/respond.min.js\">\u003C/script>\n",[151,91836,91837],{"class":469,"line":3811},[151,91838,91839],{"class":1527},"    \u003C![endif]-->\n",[151,91841,91842,91844,91846],{"class":469,"line":3820},[151,91843,34741],{"class":503},[151,91845,20975],{"class":14368},[151,91847,3742],{"class":503},[151,91849,91850,91852,91855],{"class":469,"line":7084},[151,91851,33991],{"class":503},[151,91853,91854],{"class":14368},"body",[151,91856,3742],{"class":503},[151,91858,91859,91861,91863,91866,91868],{"class":469,"line":7148},[151,91860,34669],{"class":503},[151,91862,14063],{"class":14368},[151,91864,91865],{"class":503},">Hello, world!\u003C/",[151,91867,14063],{"class":14368},[151,91869,3742],{"class":503},[151,91871,91872],{"class":469,"line":7211},[151,91873,1090],{"emptyLinePlaceholder":609},[151,91875,91876],{"class":469,"line":7273},[151,91877,91878],{"class":1527},"    \u003C!-- jQuery (necessary for Bootstrap's JavaScript plugins) -->\n",[151,91880,91881,91883,91885,91888,91890,91893,91895,91897],{"class":469,"line":7335},[151,91882,34669],{"class":503},[151,91884,19822],{"class":14368},[151,91886,91887],{"class":473}," src",[151,91889,1876],{"class":503},[151,91891,91892],{"class":481},"\"https://ajax.googleapis.com/ajax/libs/jquery/1.12.4/jquery.min.js\"",[151,91894,62997],{"class":503},[151,91896,19822],{"class":14368},[151,91898,3742],{"class":503},[151,91900,91901],{"class":469,"line":7398},[151,91902,91903],{"class":1527},"    \u003C!-- Include all compiled plugins (below), or include individual files as needed -->\n",[151,91905,91906],{"class":469,"line":7462},[151,91907,91908],{"class":1527},"    \u003C!-- Latest compiled and minified JavaScript -->\n",[151,91910,91911,91913],{"class":469,"line":7467},[151,91912,34669],{"class":503},[151,91914,91915],{"class":14368},"script\n",[151,91917,91918,91921,91923],{"class":469,"line":7532},[151,91919,91920],{"class":473},"      src",[151,91922,1876],{"class":503},[151,91924,91925],{"class":481},"\"https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js\"\n",[151,91927,91928,91930,91932],{"class":469,"line":7537},[151,91929,91732],{"class":473},[151,91931,1876],{"class":503},[151,91933,91934],{"class":481},"\"sha384-Tc5IQib027qvyjSMfHjOMaLkfuWVxZxUPnCJA7l2mCWNIpG9mGCD8wGNIcPD7Txa\"\n",[151,91936,91937,91939,91941],{"class":469,"line":7603},[151,91938,91742],{"class":473},[151,91940,1876],{"class":503},[151,91942,91747],{"class":481},[151,91944,91945,91947,91949],{"class":469,"line":7608},[151,91946,40958],{"class":503},[151,91948,19822],{"class":14368},[151,91950,3742],{"class":503},[151,91952,91953,91955,91957],{"class":469,"line":7673},[151,91954,34741],{"class":503},[151,91956,91854],{"class":14368},[151,91958,3742],{"class":503},[151,91960,91961,91963,91965],{"class":469,"line":7678},[151,91962,19966],{"class":503},[151,91964,19813],{"class":14368},[151,91966,3742],{"class":503},[11,91968,91969,91970,643],{},"This html was taken from ",[20,91971,91974],{"href":91972,"rel":91973},"http://getbootstrap.com/getting-started/",[24],"Bootstrap's 'Get Started' page",[11,91976,91977,91978,91980],{},"Now we can run ",[30,91979,90798],{}," and confirm that we see \"Hello, world!\" from the Bootstrap template.",[11,91982,59528],{},[589,91984,91985],{},"html pre.shiki code .sC2Qs, html code.shiki .sC2Qs{--shiki-default:#D73A49;--shiki-dark:#F97583;--shiki-sepia:#F92672}html pre.shiki code .sMOD_, html code.shiki .sMOD_{--shiki-default:#24292E;--shiki-dark:#E1E4E8;--shiki-sepia:#F8F8F2}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html .sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html.sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html pre.shiki code .s7F3e, html code.shiki .s7F3e{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#AE81FF}html pre.shiki code .s8-w5, html code.shiki .s8-w5{--shiki-default:#6A737D;--shiki-dark:#6A737D;--shiki-sepia:#88846F}html pre.shiki code .sXSQT, html code.shiki .sXSQT{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#F8F8F2}html pre.shiki code .sstjo, html code.shiki .sstjo{--shiki-default:#032F62;--shiki-dark:#9ECBFF;--shiki-sepia:#E6DB74}html pre.shiki code .sq6CD, html code.shiki .sq6CD{--shiki-default:#D73A49;--shiki-default-font-style:inherit;--shiki-dark:#F97583;--shiki-dark-font-style:inherit;--shiki-sepia:#66D9EF;--shiki-sepia-font-style:italic}html pre.shiki code .sz2Vg, html code.shiki .sz2Vg{--shiki-default:#6F42C1;--shiki-default-text-decoration:inherit;--shiki-dark:#B392F0;--shiki-dark-text-decoration:inherit;--shiki-sepia:#A6E22E;--shiki-sepia-text-decoration:underline}html pre.shiki code .s30JN, html code.shiki .s30JN{--shiki-default:#6F42C1;--shiki-default-font-style:inherit;--shiki-default-text-decoration:inherit;--shiki-dark:#B392F0;--shiki-dark-font-style:inherit;--shiki-dark-text-decoration:inherit;--shiki-sepia:#A6E22E;--shiki-sepia-font-style:italic;--shiki-sepia-text-decoration:underline}html pre.shiki code .srTi1, html code.shiki .srTi1{--shiki-default:#6F42C1;--shiki-dark:#B392F0;--shiki-sepia:#A6E22E}html pre.shiki code .so59x, html code.shiki .so59x{--shiki-default:#24292E;--shiki-default-font-style:inherit;--shiki-dark:#E1E4E8;--shiki-dark-font-style:inherit;--shiki-sepia:#FD971F;--shiki-sepia-font-style:italic}html pre.shiki code .sLkwE, html code.shiki .sLkwE{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#E6DB74}html pre.shiki code .sFxd3, html code.shiki .sFxd3{--shiki-default:#032F62;--shiki-dark:#DBEDFF;--shiki-sepia:#E6DB74}html pre.shiki code .sTHNf, html code.shiki .sTHNf{--shiki-default:#E36209;--shiki-default-font-style:inherit;--shiki-dark:#FFAB70;--shiki-dark-font-style:inherit;--shiki-sepia:#FD971F;--shiki-sepia-font-style:italic}html pre.shiki code .s5clZ, html code.shiki .s5clZ{--shiki-default:#22863A;--shiki-dark:#85E89D;--shiki-sepia:#F92672}",{"title":464,"searchDepth":488,"depth":488,"links":91987},[],"2017-03-04","/static/django-heroku.png",{"layout":48045},"/2017/03/04/settinging-up-django-with-heroku",{"title":90259,"description":90264},"2017/03/04/settinging-up-django-with-heroku",[30122,82868],"AbQH9G9LW_hxywC3kuULSAXk_ycU3fj4IPgnLXP3MGI",{"id":91997,"title":91998,"body":91999,"comments":609,"date":95204,"description":95205,"draft":602,"extension":605,"external":606,"image":95206,"meta":95207,"navigation":609,"path":95208,"seo":95209,"stem":95210,"tags":95211,"__hash__":95213},"blog/2017/03/03/graph_subreddit.md","Related subreddit graph exploration with NetworkX",{"type":8,"value":92000,"toc":95202},[92001,92005,92012,92015,92025,92076,92269,92275,92524,92527,92530,92538,92543,92549,92552,92558,92564,92801,92804,92818,92821,93569,93572,93605,93612,93615,93620,93623,93653,93656,93832,93837,93841,93849,93873,93894,93925,93947,93986,94012,94018,94021,94040,94046,94050,94053,94071,94077,94080,94220,94227,94236,94242,94250,94256,94264,94270,94278,94284,94292,94298,94306,94312,94320,94326,94334,94340,94348,94354,94362,94368,94376,94382,94390,94396,94404,94410,94418,94424,94439,94445,94456,94463,94596,94599,94604,94608,94615,94620,94627,94642,94645,94678,94684,94716,94721,94724,94782,94837,94843,94846,94924,94929,94933,94936,94977,94983,95007,95013,95057,95063,95066,95119,95125,95128,95157,95163,95172,95199],[14063,92002,92004],{"id":92003},"graphing-subreddits","Graphing Subreddits",[11,92006,92007,92008,92011],{},"This notebook explores some basic concepts of graph theory. A few weeks ago I set up a script to scrape data from ",[20,92009,92010],{"href":92010},"reddit.com"," with the goal of visualizing the network of related subreddits (forums on specific topics) and related data.",[11,92013,92014],{},"Reddit is home over 600,000 communities, known as subreddits, where people come to share information, opinions, links, etc. and discuss things in a open forum. Most subreddits display links to related subreddits. For example, /r/apple (the Apple subreddit) links to /r/iPhone, a subreddit all about the iPhone, and over a dozen other Apple-related subreddits.",[11,92016,92017,92018,92020,92021,92024],{},"If you visit reddit.com as a guest, you will see a list of popular subreddits. This list is located inside an ",[30,92019,19813],{}," tag called ",[30,92022,92023],{},"drop-choices",". Here it is:",[459,92026,92028],{"className":13136,"code":92027,"language":12886,"meta":464,"style":464},"from selenium import webdriver\nimport re\nimport time\nimport numpy as np\nfrom bs4 import BeautifulSoup\n",[30,92029,92030,92042,92048,92054,92064],{"__ignoreMap":464},[151,92031,92032,92034,92037,92039],{"class":469,"line":470},[151,92033,16853],{"class":1869},[151,92035,92036],{"class":503}," selenium ",[151,92038,16859],{"class":1869},[151,92040,92041],{"class":503}," webdriver\n",[151,92043,92044,92046],{"class":469,"line":488},[151,92045,16859],{"class":1869},[151,92047,58358],{"class":503},[151,92049,92050,92052],{"class":469,"line":500},[151,92051,16859],{"class":1869},[151,92053,24436],{"class":503},[151,92055,92056,92058,92060,92062],{"class":469,"line":509},[151,92057,16859],{"class":1869},[151,92059,24412],{"class":503},[151,92061,16998],{"class":1869},[151,92063,24417],{"class":503},[151,92065,92066,92068,92071,92073],{"class":469,"line":517},[151,92067,16853],{"class":1869},[151,92069,92070],{"class":503}," bs4 ",[151,92072,16859],{"class":1869},[151,92074,92075],{"class":503}," BeautifulSoup\n",[459,92077,92079],{"className":13136,"code":92078,"language":12886,"meta":464,"style":464},"driver = webdriver.PhantomJS()\ndriver.get('https://www.reddit.com/')\ntime.sleep(4 + np.random.random())\nhtml = driver.page_source.encode('utf-8')\n\ns = BeautifulSoup(html)\ndefaults = s.find('div', attrs={'class':'drop-choices'})\nsubs = re.compile(r\"\\/r\\/[\\w.]+\\/?\")\ndefault_subreddits = list(set(subs.findall(str(defaults))))\n\nfor x in default_subreddits: print '[' + x + '](https://reddit.com'+ x + '), ',\n",[30,92080,92081,92091,92101,92113,92126,92130,92139,92169,92208,92229,92233],{"__ignoreMap":464},[151,92082,92083,92086,92088],{"class":469,"line":470},[151,92084,92085],{"class":503},"driver ",[151,92087,1876],{"class":1869},[151,92089,92090],{"class":503}," webdriver.PhantomJS()\n",[151,92092,92093,92096,92099],{"class":469,"line":488},[151,92094,92095],{"class":503},"driver.get(",[151,92097,92098],{"class":481},"'https://www.reddit.com/'",[151,92100,3640],{"class":503},[151,92102,92103,92106,92108,92110],{"class":469,"line":500},[151,92104,92105],{"class":503},"time.sleep(",[151,92107,9187],{"class":477},[151,92109,23378],{"class":1869},[151,92111,92112],{"class":503}," np.random.random())\n",[151,92114,92115,92117,92119,92122,92124],{"class":469,"line":509},[151,92116,41053],{"class":503},[151,92118,1876],{"class":1869},[151,92120,92121],{"class":503}," driver.page_source.encode(",[151,92123,68697],{"class":481},[151,92125,3640],{"class":503},[151,92127,92128],{"class":469,"line":517},[151,92129,1090],{"emptyLinePlaceholder":609},[151,92131,92132,92134,92136],{"class":469,"line":534},[151,92133,74751],{"class":503},[151,92135,1876],{"class":1869},[151,92137,92138],{"class":503}," BeautifulSoup(html)\n",[151,92140,92141,92144,92146,92149,92152,92154,92156,92158,92160,92162,92164,92167],{"class":469,"line":1413},[151,92142,92143],{"class":503},"defaults ",[151,92145,1876],{"class":1869},[151,92147,92148],{"class":503}," s.find(",[151,92150,92151],{"class":481},"'div'",[151,92153,106],{"class":503},[151,92155,65333],{"class":15210},[151,92157,1876],{"class":1869},[151,92159,5729],{"class":503},[151,92161,71242],{"class":481},[151,92163,208],{"class":503},[151,92165,92166],{"class":481},"'drop-choices'",[151,92168,19610],{"class":503},[151,92170,92171,92174,92176,92179,92181,92183,92186,92188,92190,92192,92195,92198,92200,92202,92204,92206],{"class":469,"line":1418},[151,92172,92173],{"class":503},"subs ",[151,92175,1876],{"class":1869},[151,92177,92178],{"class":503}," re.compile(",[151,92180,58741],{"class":12347},[151,92182,8592],{"class":481},[151,92184,92185],{"class":58755},"\\/",[151,92187,58741],{"class":91422},[151,92189,92185],{"class":58755},[151,92191,6698],{"class":477},[151,92193,92194],{"class":58746},"\\w",[151,92196,92197],{"class":477},".]",[151,92199,22885],{"class":1869},[151,92201,92185],{"class":58755},[151,92203,10727],{"class":1869},[151,92205,8592],{"class":481},[151,92207,3640],{"class":503},[151,92209,92210,92213,92215,92217,92219,92221,92224,92226],{"class":469,"line":2462},[151,92211,92212],{"class":503},"default_subreddits ",[151,92214,1876],{"class":1869},[151,92216,59145],{"class":6205},[151,92218,12386],{"class":503},[151,92220,66796],{"class":6205},[151,92222,92223],{"class":503},"(subs.findall(",[151,92225,15343],{"class":6205},[151,92227,92228],{"class":503},"(defaults))))\n",[151,92230,92231],{"class":469,"line":2471},[151,92232,1090],{"emptyLinePlaceholder":609},[151,92234,92235,92237,92239,92241,92244,92246,92249,92251,92253,92255,92258,92260,92262,92264,92267],{"class":469,"line":2480},[151,92236,16732],{"class":1869},[151,92238,44552],{"class":503},[151,92240,16417],{"class":1869},[151,92242,92243],{"class":503}," default_subreddits: ",[151,92245,18513],{"class":2226},[151,92247,92248],{"class":481}," '['",[151,92250,23378],{"class":1869},[151,92252,44552],{"class":503},[151,92254,22885],{"class":1869},[151,92256,92257],{"class":481}," '](https://reddit.com'",[151,92259,22885],{"class":1869},[151,92261,44552],{"class":503},[151,92263,22885],{"class":1869},[151,92265,92266],{"class":481}," '), '",[151,92268,9417],{"class":503},[11,92270,92271,92272,208],{},"Here are the elements of ",[30,92273,92274],{},"default_subreddits",[210,92276,92277],{},[11,92278,92279,106,92284,106,92289,106,92294,106,92299,106,92304,106,92309,106,92314,106,92319,106,92324,106,92329,106,92334,106,92339,106,92344,106,92349,106,92354,106,92359,106,92364,106,92369,106,92374,106,92379,106,92384,106,92389,106,92394,106,92399,106,92404,106,92409,106,92414,106,92419,106,92424,106,92429,106,92434,106,92439,106,92444,106,92449,106,92454,106,92459,106,92464,106,92469,106,92474,106,92479,106,92484,106,92489,106,92494,106,92499,106,92504,106,92509,106,92514,106,92519,3634],{},[20,92280,92283],{"href":92281,"rel":92282},"https://reddit.com/r/LifeProTips/",[24],"/r/LifeProTips/",[20,92285,92288],{"href":92286,"rel":92287},"https://reddit.com/r/Futurology/",[24],"/r/Futurology/",[20,92290,92293],{"href":92291,"rel":92292},"https://reddit.com/r/OldSchoolCool/",[24],"/r/OldSchoolCool/",[20,92295,92298],{"href":92296,"rel":92297},"https://reddit.com/r/mildlyinteresting/",[24],"/r/mildlyinteresting/",[20,92300,92303],{"href":92301,"rel":92302},"https://reddit.com/r/askscience/",[24],"/r/askscience/",[20,92305,92308],{"href":92306,"rel":92307},"https://reddit.com/r/UpliftingNews/",[24],"/r/UpliftingNews/",[20,92310,92313],{"href":92311,"rel":92312},"https://reddit.com/r/aww/",[24],"/r/aww/",[20,92315,92318],{"href":92316,"rel":92317},"https://reddit.com/r/GetMotivated/",[24],"/r/GetMotivated/",[20,92320,92323],{"href":92321,"rel":92322},"https://reddit.com/r/personalfinance/",[24],"/r/personalfinance/",[20,92325,92328],{"href":92326,"rel":92327},"https://reddit.com/r/gadgets/",[24],"/r/gadgets/",[20,92330,92333],{"href":92331,"rel":92332},"https://reddit.com/r/science/",[24],"/r/science/",[20,92335,92338],{"href":92336,"rel":92337},"https://reddit.com/r/dataisbeautiful/",[24],"/r/dataisbeautiful/",[20,92340,92343],{"href":92341,"rel":92342},"https://reddit.com/r/DIY/",[24],"/r/DIY/",[20,92345,92348],{"href":92346,"rel":92347},"https://reddit.com/r/AskReddit/",[24],"/r/AskReddit/",[20,92350,92353],{"href":92351,"rel":92352},"https://reddit.com/r/space/",[24],"/r/space/",[20,92355,92358],{"href":92356,"rel":92357},"https://reddit.com/r/nosleep/",[24],"/r/nosleep/",[20,92360,92363],{"href":92361,"rel":92362},"https://reddit.com/r/Documentaries/",[24],"/r/Documentaries/",[20,92365,92368],{"href":92366,"rel":92367},"https://reddit.com/r/todayilearned/",[24],"/r/todayilearned/",[20,92370,92373],{"href":92371,"rel":92372},"https://reddit.com/r/television/",[24],"/r/television/",[20,92375,92378],{"href":92376,"rel":92377},"https://reddit.com/r/IAmA/",[24],"/r/IAmA/",[20,92380,92383],{"href":92381,"rel":92382},"https://reddit.com/r/Art/",[24],"/r/Art/",[20,92385,92388],{"href":92386,"rel":92387},"https://reddit.com/r/EarthPorn/",[24],"/r/EarthPorn/",[20,92390,92393],{"href":92391,"rel":92392},"https://reddit.com/r/books/",[24],"/r/books/",[20,92395,92398],{"href":92396,"rel":92397},"https://reddit.com/r/gifs/",[24],"/r/gifs/",[20,92400,92403],{"href":92401,"rel":92402},"https://reddit.com/r/Showerthoughts/",[24],"/r/Showerthoughts/",[20,92405,92408],{"href":92406,"rel":92407},"https://reddit.com/r/blog/",[24],"/r/blog/",[20,92410,92413],{"href":92411,"rel":92412},"https://reddit.com/r/news/",[24],"/r/news/",[20,92415,92418],{"href":92416,"rel":92417},"https://reddit.com/r/Jokes/",[24],"/r/Jokes/",[20,92420,92423],{"href":92421,"rel":92422},"https://reddit.com/r/TwoXChromosomes/",[24],"/r/TwoXChromosomes/",[20,92425,92428],{"href":92426,"rel":92427},"https://reddit.com/r/videos/",[24],"/r/videos/",[20,92430,92433],{"href":92431,"rel":92432},"https://reddit.com/r/philosophy/",[24],"/r/philosophy/",[20,92435,92438],{"href":92436,"rel":92437},"https://reddit.com/r/nottheonion/",[24],"/r/nottheonion/",[20,92440,92443],{"href":92441,"rel":92442},"https://reddit.com/r/explainlikeimfive/",[24],"/r/explainlikeimfive/",[20,92445,92448],{"href":92446,"rel":92447},"https://reddit.com/r/movies/",[24],"/r/movies/",[20,92450,92453],{"href":92451,"rel":92452},"https://reddit.com/r/Music/",[24],"/r/Music/",[20,92455,92458],{"href":92456,"rel":92457},"https://reddit.com/r/WritingPrompts/",[24],"/r/WritingPrompts/",[20,92460,92463],{"href":92461,"rel":92462},"https://reddit.com/r/worldnews/",[24],"/r/worldnews/",[20,92465,92468],{"href":92466,"rel":92467},"https://reddit.com/r/pics/",[24],"/r/pics/",[20,92470,92473],{"href":92471,"rel":92472},"https://reddit.com/r/history/",[24],"/r/history/",[20,92475,92478],{"href":92476,"rel":92477},"https://reddit.com/r/listentothis/",[24],"/r/listentothis/",[20,92480,92483],{"href":92481,"rel":92482},"https://reddit.com/r/sports/",[24],"/r/sports/",[20,92485,92488],{"href":92486,"rel":92487},"https://reddit.com/r/food/",[24],"/r/food/",[20,92490,92493],{"href":92491,"rel":92492},"https://reddit.com/r/creepy/",[24],"/r/creepy/",[20,92495,92498],{"href":92496,"rel":92497},"https://reddit.com/r/announcements/",[24],"/r/announcements/",[20,92500,92503],{"href":92501,"rel":92502},"https://reddit.com/r/gaming/",[24],"/r/gaming/",[20,92505,92508],{"href":92506,"rel":92507},"https://reddit.com/r/tifu/",[24],"/r/tifu/",[20,92510,92513],{"href":92511,"rel":92512},"https://reddit.com/r/funny/",[24],"/r/funny/",[20,92515,92518],{"href":92516,"rel":92517},"https://reddit.com/r/photoshopbattles/",[24],"/r/photoshopbattles/",[20,92520,92523],{"href":92521,"rel":92522},"https://reddit.com/r/InternetIsBeautiful/",[24],"/r/InternetIsBeautiful/",[11,92525,92526],{},"My goal here is to see how many subreddits we can reach as we branch off of these \"default\" subreddits into their related subreddits.",[11,92528,92529],{},"First, we need to set up data structures to hold data for subreddits and their related subreddits. And we need to define an algorithm for collecting data.",[11,92531,92532,92533,208],{},"Here's an intrdoduction to graphs from ",[20,92534,92537],{"href":92535,"rel":92536},"https://www.python.org/doc/essays/graphs/",[24],"python.org",[210,92539,92540],{},[11,92541,92542],{},"Few programming languages provide direct support for graphs as a data type, and Python is no exception. However, graphs are easily built out of lists and dictionaries. For instance, here's a simple graph (I can't use drawings in these columns, so I write down the graph's arcs):",[459,92544,92547],{"className":92545,"code":92546,"language":997,"meta":464},[995],"A -> B\nA -> C\nB -> C\nB -> D\nC -> D\nD -> C\nE -> F\nF -> C\n",[30,92548,92546],{"__ignoreMap":464},[11,92550,92551],{},"This graph has six nodes (A-F) and eight arcs. It can be represented by the following Python data structure:",[459,92553,92556],{"className":92554,"code":92555,"language":997,"meta":464},[995],"graph =     {'A': ['B', 'C'],\n             'B': ['C', 'D'],\n             'C': ['D'],\n             'D': ['C'],\n             'E': ['F'],\n             'F': ['C']}\n",[30,92557,92555],{"__ignoreMap":464},[11,92559,92560,92561,92563],{},"First let's define how we would go only one branch deep into this graph (i.e. find the related subreddits for ",[51,92562,54532],{}," the default subreddits). To collect the data, I first looped through the default subreddits and save the html of each subreddit to its own text file. Here's a script with comments:",[459,92565,92567],{"className":13136,"code":92566,"language":12886,"meta":464,"style":464},"#first we navigate to the correct folder where we will store the first level of related subreddits\nos.chdir(os.path.expanduser('~/Documents/Projects/Data/Subreddits/one/'))\n\n#next we instantiate the webdriver we will be using: PhantomJS\ndriver = webdriver.PhantomJS()\n\n#loop through the list of default subreddits\nfor num, subreddit in enumerate(default_subreddits):\n\n    #for each subreddit, we append the /r/subreddit path to the base URL (reddit.com)\n    driver.get('https://www.reddit.com'+subreddit)\n\n    #wait for two seconds\n    time.sleep(2 + np.random.random())\n\n    #save the html of the loaded page to a variable: html\n    html = driver.page_source.encode('utf-8')\n\n    #remove '/r/' from the subreddit name string\n    name = subreddit.split('/')[2]\n\n    #open a new file and give it the name of the subreddit we just scraped\n    subreddit_html_file = open(name+'.txt', 'w+')\n\n    #write the html contents to the file\n    subreddit_html_file.write(html)\n\n    #clost the file\n    subreddit_html_file.close()\n\n    #print out the number and name of the subreddit we just scrapped to make sure things are working\n    print str(num) + ' ' + subreddit,\n\n",[30,92568,92569,92574,92583,92587,92592,92600,92604,92609,92623,92627,92632,92645,92649,92654,92664,92668,92673,92685,92689,92694,92713,92717,92722,92746,92750,92755,92760,92764,92769,92774,92778,92783],{"__ignoreMap":464},[151,92570,92571],{"class":469,"line":470},[151,92572,92573],{"class":1527},"#first we navigate to the correct folder where we will store the first level of related subreddits\n",[151,92575,92576,92578,92581],{"class":469,"line":488},[151,92577,86611],{"class":503},[151,92579,92580],{"class":481},"'~/Documents/Projects/Data/Subreddits/one/'",[151,92582,12451],{"class":503},[151,92584,92585],{"class":469,"line":500},[151,92586,1090],{"emptyLinePlaceholder":609},[151,92588,92589],{"class":469,"line":509},[151,92590,92591],{"class":1527},"#next we instantiate the webdriver we will be using: PhantomJS\n",[151,92593,92594,92596,92598],{"class":469,"line":517},[151,92595,92085],{"class":503},[151,92597,1876],{"class":1869},[151,92599,92090],{"class":503},[151,92601,92602],{"class":469,"line":534},[151,92603,1090],{"emptyLinePlaceholder":609},[151,92605,92606],{"class":469,"line":1413},[151,92607,92608],{"class":1527},"#loop through the list of default subreddits\n",[151,92610,92611,92613,92616,92618,92620],{"class":469,"line":1418},[151,92612,16732],{"class":1869},[151,92614,92615],{"class":503}," num, subreddit ",[151,92617,16417],{"class":1869},[151,92619,17042],{"class":2226},[151,92621,92622],{"class":503},"(default_subreddits):\n",[151,92624,92625],{"class":469,"line":2462},[151,92626,1090],{"emptyLinePlaceholder":609},[151,92628,92629],{"class":469,"line":2471},[151,92630,92631],{"class":1527},"    #for each subreddit, we append the /r/subreddit path to the base URL (reddit.com)\n",[151,92633,92634,92637,92640,92642],{"class":469,"line":2480},[151,92635,92636],{"class":503},"    driver.get(",[151,92638,92639],{"class":481},"'https://www.reddit.com'",[151,92641,22885],{"class":1869},[151,92643,92644],{"class":503},"subreddit)\n",[151,92646,92647],{"class":469,"line":2489},[151,92648,1090],{"emptyLinePlaceholder":609},[151,92650,92651],{"class":469,"line":2497},[151,92652,92653],{"class":1527},"    #wait for two seconds\n",[151,92655,92656,92658,92660,92662],{"class":469,"line":3140},[151,92657,65408],{"class":503},[151,92659,6619],{"class":477},[151,92661,23378],{"class":1869},[151,92663,92112],{"class":503},[151,92665,92666],{"class":469,"line":3149},[151,92667,1090],{"emptyLinePlaceholder":609},[151,92669,92670],{"class":469,"line":3158},[151,92671,92672],{"class":1527},"    #save the html of the loaded page to a variable: html\n",[151,92674,92675,92677,92679,92681,92683],{"class":469,"line":3167},[151,92676,64776],{"class":503},[151,92678,1876],{"class":1869},[151,92680,92121],{"class":503},[151,92682,68697],{"class":481},[151,92684,3640],{"class":503},[151,92686,92687],{"class":469,"line":3175},[151,92688,1090],{"emptyLinePlaceholder":609},[151,92690,92691],{"class":469,"line":3184},[151,92692,92693],{"class":1527},"    #remove '/r/' from the subreddit name string\n",[151,92695,92696,92699,92701,92704,92707,92709,92711],{"class":469,"line":3193},[151,92697,92698],{"class":503},"    name ",[151,92700,1876],{"class":1869},[151,92702,92703],{"class":503}," subreddit.split(",[151,92705,92706],{"class":481},"'/'",[151,92708,40832],{"class":503},[151,92710,6619],{"class":477},[151,92712,3691],{"class":503},[151,92714,92715],{"class":469,"line":3720},[151,92716,1090],{"emptyLinePlaceholder":609},[151,92718,92719],{"class":469,"line":3729},[151,92720,92721],{"class":1527},"    #open a new file and give it the name of the subreddit we just scraped\n",[151,92723,92724,92727,92729,92731,92734,92736,92739,92741,92744],{"class":469,"line":3735},[151,92725,92726],{"class":503},"    subreddit_html_file ",[151,92728,1876],{"class":1869},[151,92730,16970],{"class":2226},[151,92732,92733],{"class":503},"(name",[151,92735,22885],{"class":1869},[151,92737,92738],{"class":481},"'.txt'",[151,92740,106],{"class":503},[151,92742,92743],{"class":481},"'w+'",[151,92745,3640],{"class":503},[151,92747,92748],{"class":469,"line":3745},[151,92749,1090],{"emptyLinePlaceholder":609},[151,92751,92752],{"class":469,"line":3754},[151,92753,92754],{"class":1527},"    #write the html contents to the file\n",[151,92756,92757],{"class":469,"line":3760},[151,92758,92759],{"class":503},"    subreddit_html_file.write(html)\n",[151,92761,92762],{"class":469,"line":3773},[151,92763,1090],{"emptyLinePlaceholder":609},[151,92765,92766],{"class":469,"line":3782},[151,92767,92768],{"class":1527},"    #clost the file\n",[151,92770,92771],{"class":469,"line":3791},[151,92772,92773],{"class":503},"    subreddit_html_file.close()\n",[151,92775,92776],{"class":469,"line":3803},[151,92777,1090],{"emptyLinePlaceholder":609},[151,92779,92780],{"class":469,"line":3811},[151,92781,92782],{"class":1527},"    #print out the number and name of the subreddit we just scrapped to make sure things are working\n",[151,92784,92785,92787,92789,92792,92794,92796,92798],{"class":469,"line":3820},[151,92786,24285],{"class":2226},[151,92788,84112],{"class":6205},[151,92790,92791],{"class":503},"(num) ",[151,92793,22885],{"class":1869},[151,92795,32143],{"class":481},[151,92797,23378],{"class":1869},[151,92799,92800],{"class":503}," subreddit,\n",[11,92802,92803],{},"Next, we want to go through each file and extract the information we want. Here's what we will be getting:",[76,92805,92806,92809,92812,92815],{},[79,92807,92808],{},"Number of subscribers",[79,92810,92811],{},"Subreddit description",[79,92813,92814],{},"Date created",[79,92816,92817],{},"Related subreddits",[11,92819,92820],{},"For this type of project, I prefer to loop through each page and creating several small dictionaries for each data point, then combine the small dictionaries into a large dictionary, and then append the dictionary to a list of dictionaries. Once I have looped through all of the pages, I can create a pandas DataFrame from the list of dictionaries. This allows me to easily manipulate the data. Here's the script that I used to do this:",[459,92822,92824],{"className":13136,"code":92823,"language":12886,"meta":464,"style":464},"#navigate to where the html files are stored (I moved them around a bit so it is not consistent with the script above)\nos.chdir('E://DATA/Subreddits/subreddits_html/')\n\n#generate a list of files that we will loop through\nfiles = os.listdir('E://DATA/Subreddits/subreddits_html/')\n\n#set up an empty list that we will append dictionaries to\ndict_list = []\n\n#loop through the files\nfor file_ in files:\n\n    #print out the name of the current file in the loop\n    print file_,\n\n    #open the file\n    f = open(file_, 'r')\n    #read the file contents to a local variable\n    html = f.read()\n    #create a BeautifulSoup object that we will use to parse the HTML\n    b = BeautifulSoup(html, 'lxml')\n\n    #get the subreddit name that we are working with (from the `file` variable)\n    subreddit_name = '/r/' + file_[:-4].lower()\n    #put the name into a dictionary\n    subreddit_name_dict = {'subreddit':subreddit_name}\n\n    #get number of subscribers\n    subs = b.find('span', attrs={'class':'subscribers'})\n    #if the number of subscribers is displayed on the page, then we find it and add it to a dictionary\n    if subs:\n        subs = b.find('span', attrs={'class':'subscribers'}).find('span', attrs={'class':'number'}).text.replace(',', '')\n        subs_dict = {'subscribers':int(subs)}\n    #if the number of subscribers is not displayed on the page, then we set the number of subscribers in the dictionary to None\n    else:\n        subs_dict = {'subscribers':None}\n\n    #similar process for the description: if the description is displayed, get it and save it to desc\n    #if it is not available, then desc will be set to `None`\n    desc = b.find('div', attrs={'class':'md'})\n    if desc:\n        desc = b.find('div', attrs={'class':'md'}).text\n        desc = desc.replace('\\n', ' ')\n    desc_dict = {'description':desc}\n\n    #here we use regular expressions to find links anywhere on the page that have the structure: \"/r/something/\"\n    rel_subr = re.compile(r\"\\/r\\/[\\w.]+\\/?\")\n    #make a list of these links based on the \"/r/something/\" pattern\n    related_subreddits = rel_subr.findall(html)\n\n    #save the list to a dictionary\n    subreddits_dict = {'related':related_subreddits}\n\n    #same processes for recording the date that the subreddit was created: get the date from an HTML element,\n    #then save it to a dictionary. There were two different formats available in the HTML so I grabbed both\n    age = b.find('span', attrs={'class':'age'})\n    if age:\n        time1 = age.find('time')['title']\n        time2 = age.find('time')['datetime']\n\n    #save the date to a dictionary\n    time_dict = {\"date1\":time1, \"date2\":time2}\n\n    #take all the dictionaries we just created and put them together into one big dictionary\n    dictionary = dict(subs_dict.items()+desc_dict.items()+subreddits_dict.items()+subreddit_name_dict.items()+time_dict.items())\n\n    #append the big dictionary to the list that we defined right before the beginning of the loop\n    dict_list.append(dictionary)\n\n    #deconstruct the Beautiful Soup object (this can eat up memory very quickly, so it is very important when processing lots of data)\n    b.decompose()\n\n    #clost the file\n    f.close()\n",[30,92825,92826,92831,92841,92845,92850,92862,92866,92871,92880,92884,92889,92900,92904,92909,92916,92920,92925,92941,92946,92954,92959,92972,92976,92981,93003,93008,93023,93027,93032,93062,93067,93074,93130,93149,93154,93160,93176,93180,93185,93190,93218,93225,93253,93274,93289,93293,93298,93333,93338,93348,93352,93357,93372,93376,93381,93386,93413,93420,93439,93457,93461,93466,93487,93491,93496,93528,93532,93537,93542,93546,93551,93556,93560,93564],{"__ignoreMap":464},[151,92827,92828],{"class":469,"line":470},[151,92829,92830],{"class":1527},"#navigate to where the html files are stored (I moved them around a bit so it is not consistent with the script above)\n",[151,92832,92833,92836,92839],{"class":469,"line":488},[151,92834,92835],{"class":503},"os.chdir(",[151,92837,92838],{"class":481},"'E://DATA/Subreddits/subreddits_html/'",[151,92840,3640],{"class":503},[151,92842,92843],{"class":469,"line":500},[151,92844,1090],{"emptyLinePlaceholder":609},[151,92846,92847],{"class":469,"line":509},[151,92848,92849],{"class":1527},"#generate a list of files that we will loop through\n",[151,92851,92852,92854,92856,92858,92860],{"class":469,"line":517},[151,92853,64699],{"class":503},[151,92855,1876],{"class":1869},[151,92857,44561],{"class":503},[151,92859,92838],{"class":481},[151,92861,3640],{"class":503},[151,92863,92864],{"class":469,"line":534},[151,92865,1090],{"emptyLinePlaceholder":609},[151,92867,92868],{"class":469,"line":1413},[151,92869,92870],{"class":1527},"#set up an empty list that we will append dictionaries to\n",[151,92872,92873,92876,92878],{"class":469,"line":1418},[151,92874,92875],{"class":503},"dict_list ",[151,92877,1876],{"class":1869},[151,92879,16606],{"class":503},[151,92881,92882],{"class":469,"line":2462},[151,92883,1090],{"emptyLinePlaceholder":609},[151,92885,92886],{"class":469,"line":2471},[151,92887,92888],{"class":1527},"#loop through the files\n",[151,92890,92891,92893,92896,92898],{"class":469,"line":2480},[151,92892,16732],{"class":1869},[151,92894,92895],{"class":503}," file_ ",[151,92897,16417],{"class":1869},[151,92899,65212],{"class":503},[151,92901,92902],{"class":469,"line":2489},[151,92903,1090],{"emptyLinePlaceholder":609},[151,92905,92906],{"class":469,"line":2497},[151,92907,92908],{"class":1527},"    #print out the name of the current file in the loop\n",[151,92910,92911,92913],{"class":469,"line":3140},[151,92912,24285],{"class":2226},[151,92914,92915],{"class":503}," file_,\n",[151,92917,92918],{"class":469,"line":3149},[151,92919,1090],{"emptyLinePlaceholder":609},[151,92921,92922],{"class":469,"line":3158},[151,92923,92924],{"class":1527},"    #open the file\n",[151,92926,92927,92930,92932,92934,92937,92939],{"class":469,"line":3167},[151,92928,92929],{"class":503},"    f ",[151,92931,1876],{"class":1869},[151,92933,16970],{"class":2226},[151,92935,92936],{"class":503},"(file_, ",[151,92938,44149],{"class":481},[151,92940,3640],{"class":503},[151,92942,92943],{"class":469,"line":3175},[151,92944,92945],{"class":1527},"    #read the file contents to a local variable\n",[151,92947,92948,92950,92952],{"class":469,"line":3184},[151,92949,64776],{"class":503},[151,92951,1876],{"class":1869},[151,92953,58645],{"class":503},[151,92955,92956],{"class":469,"line":3193},[151,92957,92958],{"class":1527},"    #create a BeautifulSoup object that we will use to parse the HTML\n",[151,92960,92961,92964,92966,92968,92970],{"class":469,"line":3720},[151,92962,92963],{"class":503},"    b ",[151,92965,1876],{"class":1869},[151,92967,64801],{"class":503},[151,92969,64804],{"class":481},[151,92971,3640],{"class":503},[151,92973,92974],{"class":469,"line":3729},[151,92975,1090],{"emptyLinePlaceholder":609},[151,92977,92978],{"class":469,"line":3735},[151,92979,92980],{"class":1527},"    #get the subreddit name that we are working with (from the `file` variable)\n",[151,92982,92983,92986,92988,92991,92993,92996,92998,93000],{"class":469,"line":3745},[151,92984,92985],{"class":503},"    subreddit_name ",[151,92987,1876],{"class":1869},[151,92989,92990],{"class":481}," '/r/'",[151,92992,23378],{"class":1869},[151,92994,92995],{"class":503}," file_[:",[151,92997,12445],{"class":1869},[151,92999,9187],{"class":477},[151,93001,93002],{"class":503},"].lower()\n",[151,93004,93005],{"class":469,"line":3754},[151,93006,93007],{"class":1527},"    #put the name into a dictionary\n",[151,93009,93010,93013,93015,93017,93020],{"class":469,"line":3760},[151,93011,93012],{"class":503},"    subreddit_name_dict ",[151,93014,1876],{"class":1869},[151,93016,52023],{"class":503},[151,93018,93019],{"class":481},"'subreddit'",[151,93021,93022],{"class":503},":subreddit_name}\n",[151,93024,93025],{"class":469,"line":3773},[151,93026,1090],{"emptyLinePlaceholder":609},[151,93028,93029],{"class":469,"line":3782},[151,93030,93031],{"class":1527},"    #get number of subscribers\n",[151,93033,93034,93037,93039,93042,93045,93047,93049,93051,93053,93055,93057,93060],{"class":469,"line":3791},[151,93035,93036],{"class":503},"    subs ",[151,93038,1876],{"class":1869},[151,93040,93041],{"class":503}," b.find(",[151,93043,93044],{"class":481},"'span'",[151,93046,106],{"class":503},[151,93048,65333],{"class":15210},[151,93050,1876],{"class":1869},[151,93052,5729],{"class":503},[151,93054,71242],{"class":481},[151,93056,208],{"class":503},[151,93058,93059],{"class":481},"'subscribers'",[151,93061,19610],{"class":503},[151,93063,93064],{"class":469,"line":3803},[151,93065,93066],{"class":1527},"    #if the number of subscribers is displayed on the page, then we find it and add it to a dictionary\n",[151,93068,93069,93071],{"class":469,"line":3811},[151,93070,23327],{"class":1869},[151,93072,93073],{"class":503}," subs:\n",[151,93075,93076,93079,93081,93083,93085,93087,93089,93091,93093,93095,93097,93099,93102,93104,93106,93108,93110,93112,93114,93116,93119,93122,93124,93126,93128],{"class":469,"line":3820},[151,93077,93078],{"class":503},"        subs ",[151,93080,1876],{"class":1869},[151,93082,93041],{"class":503},[151,93084,93044],{"class":481},[151,93086,106],{"class":503},[151,93088,65333],{"class":15210},[151,93090,1876],{"class":1869},[151,93092,5729],{"class":503},[151,93094,71242],{"class":481},[151,93096,208],{"class":503},[151,93098,93059],{"class":481},[151,93100,93101],{"class":503},"}).find(",[151,93103,93044],{"class":481},[151,93105,106],{"class":503},[151,93107,65333],{"class":15210},[151,93109,1876],{"class":1869},[151,93111,5729],{"class":503},[151,93113,71242],{"class":481},[151,93115,208],{"class":503},[151,93117,93118],{"class":481},"'number'",[151,93120,93121],{"class":503},"}).text.replace(",[151,93123,64948],{"class":481},[151,93125,106],{"class":503},[151,93127,2301],{"class":481},[151,93129,3640],{"class":503},[151,93131,93132,93135,93137,93139,93141,93143,93146],{"class":469,"line":7084},[151,93133,93134],{"class":503},"        subs_dict ",[151,93136,1876],{"class":1869},[151,93138,52023],{"class":503},[151,93140,93059],{"class":481},[151,93142,208],{"class":503},[151,93144,93145],{"class":6205},"int",[151,93147,93148],{"class":503},"(subs)}\n",[151,93150,93151],{"class":469,"line":7148},[151,93152,93153],{"class":1527},"    #if the number of subscribers is not displayed on the page, then we set the number of subscribers in the dictionary to None\n",[151,93155,93156,93158],{"class":469,"line":7211},[151,93157,38878],{"class":1869},[151,93159,14372],{"class":503},[151,93161,93162,93164,93166,93168,93170,93172,93174],{"class":469,"line":7273},[151,93163,93134],{"class":503},[151,93165,1876],{"class":1869},[151,93167,52023],{"class":503},[151,93169,93059],{"class":481},[151,93171,208],{"class":503},[151,93173,15437],{"class":477},[151,93175,6274],{"class":503},[151,93177,93178],{"class":469,"line":7335},[151,93179,1090],{"emptyLinePlaceholder":609},[151,93181,93182],{"class":469,"line":7398},[151,93183,93184],{"class":1527},"    #similar process for the description: if the description is displayed, get it and save it to desc\n",[151,93186,93187],{"class":469,"line":7462},[151,93188,93189],{"class":1527},"    #if it is not available, then desc will be set to `None`\n",[151,93191,93192,93195,93197,93199,93201,93203,93205,93207,93209,93211,93213,93216],{"class":469,"line":7467},[151,93193,93194],{"class":503},"    desc ",[151,93196,1876],{"class":1869},[151,93198,93041],{"class":503},[151,93200,92151],{"class":481},[151,93202,106],{"class":503},[151,93204,65333],{"class":15210},[151,93206,1876],{"class":1869},[151,93208,5729],{"class":503},[151,93210,71242],{"class":481},[151,93212,208],{"class":503},[151,93214,93215],{"class":481},"'md'",[151,93217,19610],{"class":503},[151,93219,93220,93222],{"class":469,"line":7532},[151,93221,23327],{"class":1869},[151,93223,93224],{"class":503}," desc:\n",[151,93226,93227,93230,93232,93234,93236,93238,93240,93242,93244,93246,93248,93250],{"class":469,"line":7537},[151,93228,93229],{"class":503},"        desc ",[151,93231,1876],{"class":1869},[151,93233,93041],{"class":503},[151,93235,92151],{"class":481},[151,93237,106],{"class":503},[151,93239,65333],{"class":15210},[151,93241,1876],{"class":1869},[151,93243,5729],{"class":503},[151,93245,71242],{"class":481},[151,93247,208],{"class":503},[151,93249,93215],{"class":481},[151,93251,93252],{"class":503},"}).text\n",[151,93254,93255,93257,93259,93262,93264,93266,93268,93270,93272],{"class":469,"line":7603},[151,93256,93229],{"class":503},[151,93258,1876],{"class":1869},[151,93260,93261],{"class":503}," desc.replace(",[151,93263,13223],{"class":481},[151,93265,8043],{"class":477},[151,93267,13223],{"class":481},[151,93269,106],{"class":503},[151,93271,86598],{"class":481},[151,93273,3640],{"class":503},[151,93275,93276,93279,93281,93283,93286],{"class":469,"line":7608},[151,93277,93278],{"class":503},"    desc_dict ",[151,93280,1876],{"class":1869},[151,93282,52023],{"class":503},[151,93284,93285],{"class":481},"'description'",[151,93287,93288],{"class":503},":desc}\n",[151,93290,93291],{"class":469,"line":7673},[151,93292,1090],{"emptyLinePlaceholder":609},[151,93294,93295],{"class":469,"line":7678},[151,93296,93297],{"class":1527},"    #here we use regular expressions to find links anywhere on the page that have the structure: \"/r/something/\"\n",[151,93299,93300,93303,93305,93307,93309,93311,93313,93315,93317,93319,93321,93323,93325,93327,93329,93331],{"class":469,"line":7708},[151,93301,93302],{"class":503},"    rel_subr ",[151,93304,1876],{"class":1869},[151,93306,92178],{"class":503},[151,93308,58741],{"class":12347},[151,93310,8592],{"class":481},[151,93312,92185],{"class":58755},[151,93314,58741],{"class":91422},[151,93316,92185],{"class":58755},[151,93318,6698],{"class":477},[151,93320,92194],{"class":58746},[151,93322,92197],{"class":477},[151,93324,22885],{"class":1869},[151,93326,92185],{"class":58755},[151,93328,10727],{"class":1869},[151,93330,8592],{"class":481},[151,93332,3640],{"class":503},[151,93334,93335],{"class":469,"line":7713},[151,93336,93337],{"class":1527},"    #make a list of these links based on the \"/r/something/\" pattern\n",[151,93339,93340,93343,93345],{"class":469,"line":7746},[151,93341,93342],{"class":503},"    related_subreddits ",[151,93344,1876],{"class":1869},[151,93346,93347],{"class":503}," rel_subr.findall(html)\n",[151,93349,93350],{"class":469,"line":7751},[151,93351,1090],{"emptyLinePlaceholder":609},[151,93353,93354],{"class":469,"line":7816},[151,93355,93356],{"class":1527},"    #save the list to a dictionary\n",[151,93358,93359,93362,93364,93366,93369],{"class":469,"line":7821},[151,93360,93361],{"class":503},"    subreddits_dict ",[151,93363,1876],{"class":1869},[151,93365,52023],{"class":503},[151,93367,93368],{"class":481},"'related'",[151,93370,93371],{"class":503},":related_subreddits}\n",[151,93373,93374],{"class":469,"line":7847},[151,93375,1090],{"emptyLinePlaceholder":609},[151,93377,93378],{"class":469,"line":7852},[151,93379,93380],{"class":1527},"    #same processes for recording the date that the subreddit was created: get the date from an HTML element,\n",[151,93382,93383],{"class":469,"line":7887},[151,93384,93385],{"class":1527},"    #then save it to a dictionary. There were two different formats available in the HTML so I grabbed both\n",[151,93387,93388,93391,93393,93395,93397,93399,93401,93403,93405,93407,93409,93411],{"class":469,"line":7892},[151,93389,93390],{"class":503},"    age ",[151,93392,1876],{"class":1869},[151,93394,93041],{"class":503},[151,93396,93044],{"class":481},[151,93398,106],{"class":503},[151,93400,65333],{"class":15210},[151,93402,1876],{"class":1869},[151,93404,5729],{"class":503},[151,93406,71242],{"class":481},[151,93408,208],{"class":503},[151,93410,45183],{"class":481},[151,93412,19610],{"class":503},[151,93414,93415,93417],{"class":469,"line":7924},[151,93416,23327],{"class":1869},[151,93418,93419],{"class":503}," age:\n",[151,93421,93422,93425,93427,93430,93433,93435,93437],{"class":469,"line":7929},[151,93423,93424],{"class":503},"        time1 ",[151,93426,1876],{"class":1869},[151,93428,93429],{"class":503}," age.find(",[151,93431,93432],{"class":481},"'time'",[151,93434,40832],{"class":503},[151,93436,61298],{"class":481},[151,93438,3691],{"class":503},[151,93440,93441,93444,93446,93448,93450,93452,93455],{"class":469,"line":7991},[151,93442,93443],{"class":503},"        time2 ",[151,93445,1876],{"class":1869},[151,93447,93429],{"class":503},[151,93449,93432],{"class":481},[151,93451,40832],{"class":503},[151,93453,93454],{"class":481},"'datetime'",[151,93456,3691],{"class":503},[151,93458,93459],{"class":469,"line":7996},[151,93460,1090],{"emptyLinePlaceholder":609},[151,93462,93463],{"class":469,"line":8078},[151,93464,93465],{"class":1527},"    #save the date to a dictionary\n",[151,93467,93468,93471,93473,93475,93478,93481,93484],{"class":469,"line":8140},[151,93469,93470],{"class":503},"    time_dict ",[151,93472,1876],{"class":1869},[151,93474,52023],{"class":503},[151,93476,93477],{"class":481},"\"date1\"",[151,93479,93480],{"class":503},":time1, ",[151,93482,93483],{"class":481},"\"date2\"",[151,93485,93486],{"class":503},":time2}\n",[151,93488,93489],{"class":469,"line":8145},[151,93490,1090],{"emptyLinePlaceholder":609},[151,93492,93493],{"class":469,"line":8259},[151,93494,93495],{"class":1527},"    #take all the dictionaries we just created and put them together into one big dictionary\n",[151,93497,93498,93501,93503,93505,93508,93510,93513,93515,93518,93520,93523,93525],{"class":469,"line":8264},[151,93499,93500],{"class":503},"    dictionary ",[151,93502,1876],{"class":1869},[151,93504,51817],{"class":6205},[151,93506,93507],{"class":503},"(subs_dict.items()",[151,93509,22885],{"class":1869},[151,93511,93512],{"class":503},"desc_dict.items()",[151,93514,22885],{"class":1869},[151,93516,93517],{"class":503},"subreddits_dict.items()",[151,93519,22885],{"class":1869},[151,93521,93522],{"class":503},"subreddit_name_dict.items()",[151,93524,22885],{"class":1869},[151,93526,93527],{"class":503},"time_dict.items())\n",[151,93529,93530],{"class":469,"line":8613},[151,93531,1090],{"emptyLinePlaceholder":609},[151,93533,93534],{"class":469,"line":8678},[151,93535,93536],{"class":1527},"    #append the big dictionary to the list that we defined right before the beginning of the loop\n",[151,93538,93539],{"class":469,"line":8742},[151,93540,93541],{"class":503},"    dict_list.append(dictionary)\n",[151,93543,93544],{"class":469,"line":8806},[151,93545,1090],{"emptyLinePlaceholder":609},[151,93547,93548],{"class":469,"line":8870},[151,93549,93550],{"class":1527},"    #deconstruct the Beautiful Soup object (this can eat up memory very quickly, so it is very important when processing lots of data)\n",[151,93552,93553],{"class":469,"line":8875},[151,93554,93555],{"class":503},"    b.decompose()\n",[151,93557,93558],{"class":469,"line":8881},[151,93559,1090],{"emptyLinePlaceholder":609},[151,93561,93562],{"class":469,"line":8886},[151,93563,92768],{"class":1527},[151,93565,93566],{"class":469,"line":8892},[151,93567,93568],{"class":503},"    f.close()\n",[11,93570,93571],{},"Next, let's save the results into a csv file. This let's us load the results quickly without having to scrape everyting again. To do this we can use the pandas library.",[459,93573,93575],{"className":13136,"code":93574,"language":12886,"meta":464,"style":464},"import pandas as pd\ndf0 = pd.DataFrame(dict_list, index=None)\n",[30,93576,93577,93587],{"__ignoreMap":464},[151,93578,93579,93581,93583,93585],{"class":469,"line":470},[151,93580,16859],{"class":1869},[151,93582,83790],{"class":503},[151,93584,16998],{"class":1869},[151,93586,83795],{"class":503},[151,93588,93589,93592,93594,93597,93599,93601,93603],{"class":469,"line":488},[151,93590,93591],{"class":503},"df0 ",[151,93593,1876],{"class":1869},[151,93595,93596],{"class":503}," pd.DataFrame(dict_list, ",[151,93598,86494],{"class":15210},[151,93600,1876],{"class":1869},[151,93602,15437],{"class":477},[151,93604,3640],{"class":503},[11,93606,93607,93608,93611],{},"At this point, we can go through the ",[30,93609,93610],{},"related"," column in the DataFrame and put together a list of all the related subreddits. With this list, we can simply repeat the process over and over again. However, each time we start with a new list of subreddits, we want to make sure that they have not already been collected.",[11,93613,93614],{},"Next I will read in one DataFrame that represents related subreddits \"three levels deep\" relative to the default subreddits.",[11,93616,93617],{},[15,93618,93619],{},"Default --> Related --> Related --> Related",[11,93621,93622],{},"This DataFrame represents the collection of subreddits from all of these \"layers\" of the graph.",[459,93624,93626],{"className":13136,"code":93625,"language":12886,"meta":464,"style":464},"import pandas as pd\nmaster_df = pd.read_pickle('pickle/master_df.p')\n",[30,93627,93628,93638],{"__ignoreMap":464},[151,93629,93630,93632,93634,93636],{"class":469,"line":470},[151,93631,16859],{"class":1869},[151,93633,83790],{"class":503},[151,93635,16998],{"class":1869},[151,93637,83795],{"class":503},[151,93639,93640,93643,93645,93648,93651],{"class":469,"line":488},[151,93641,93642],{"class":503},"master_df ",[151,93644,1876],{"class":1869},[151,93646,93647],{"class":503}," pd.read_pickle(",[151,93649,93650],{"class":481},"'pickle/master_df.p'",[151,93652,3640],{"class":503},[11,93654,93655],{},"Now we can do a quick visualization of the growth in number of subreddits since the website's start in 2005.",[459,93657,93659],{"className":13136,"code":93658,"language":12886,"meta":464,"style":464},"import warnings\nwarnings.filterwarnings('ignore')\n%matplotlib inline\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport numpy as np\n\nmaster_df_ = master_df[master_df.notnull()]\nmaster_df_.date1 = pd.to_datetime(master_df_['date1'])\n\nlist_of_dates = master_df_.date1.sort_values()\n\ncounts = np.arange(0, len(list_of_dates))\n_ = plt.plot(list_of_dates, counts)\n_ = plt.title('Number of subreddits over time')\n_ = plt.xlabel('Date')\n_ = plt.ylabel('Cummulative Count')\nplt.savefig(os.path.expanduser('~/Documents/GitHub/briancaffey.github.io/static/subreddit_graph/subreddits_count.png'))\n",[30,93660,93661,93668,93678,93684,93694,93704,93714,93718,93728,93743,93747,93756,93760,93776,93784,93797,93809,93822],{"__ignoreMap":464},[151,93662,93663,93665],{"class":469,"line":470},[151,93664,16859],{"class":1869},[151,93666,93667],{"class":503}," warnings\n",[151,93669,93670,93673,93676],{"class":469,"line":488},[151,93671,93672],{"class":503},"warnings.filterwarnings(",[151,93674,93675],{"class":481},"'ignore'",[151,93677,3640],{"class":503},[151,93679,93680,93682],{"class":469,"line":500},[151,93681,44519],{"class":1869},[151,93683,44522],{"class":503},[151,93685,93686,93688,93690,93692],{"class":469,"line":509},[151,93687,16859],{"class":1869},[151,93689,44073],{"class":503},[151,93691,16998],{"class":1869},[151,93693,44078],{"class":503},[151,93695,93696,93698,93700,93702],{"class":469,"line":517},[151,93697,16859],{"class":1869},[151,93699,83853],{"class":503},[151,93701,16998],{"class":1869},[151,93703,83858],{"class":503},[151,93705,93706,93708,93710,93712],{"class":469,"line":534},[151,93707,16859],{"class":1869},[151,93709,24412],{"class":503},[151,93711,16998],{"class":1869},[151,93713,24417],{"class":503},[151,93715,93716],{"class":469,"line":1413},[151,93717,1090],{"emptyLinePlaceholder":609},[151,93719,93720,93723,93725],{"class":469,"line":1418},[151,93721,93722],{"class":503},"master_df_ ",[151,93724,1876],{"class":1869},[151,93726,93727],{"class":503}," master_df[master_df.notnull()]\n",[151,93729,93730,93733,93735,93738,93741],{"class":469,"line":2462},[151,93731,93732],{"class":503},"master_df_.date1 ",[151,93734,1876],{"class":1869},[151,93736,93737],{"class":503}," pd.to_datetime(master_df_[",[151,93739,93740],{"class":481},"'date1'",[151,93742,38820],{"class":503},[151,93744,93745],{"class":469,"line":2471},[151,93746,1090],{"emptyLinePlaceholder":609},[151,93748,93749,93751,93753],{"class":469,"line":2480},[151,93750,70755],{"class":503},[151,93752,1876],{"class":1869},[151,93754,93755],{"class":503}," master_df_.date1.sort_values()\n",[151,93757,93758],{"class":469,"line":2489},[151,93759,1090],{"emptyLinePlaceholder":609},[151,93761,93762,93764,93766,93768,93770,93772,93774],{"class":469,"line":2497},[151,93763,70770],{"class":503},[151,93765,1876],{"class":1869},[151,93767,70775],{"class":503},[151,93769,9181],{"class":477},[151,93771,106],{"class":503},[151,93773,65875],{"class":2226},[151,93775,70784],{"class":503},[151,93777,93778,93780,93782],{"class":469,"line":3140},[151,93779,70807],{"class":503},[151,93781,1876],{"class":1869},[151,93783,70812],{"class":503},[151,93785,93786,93788,93790,93792,93795],{"class":469,"line":3149},[151,93787,70807],{"class":503},[151,93789,1876],{"class":1869},[151,93791,70821],{"class":503},[151,93793,93794],{"class":481},"'Number of subreddits over time'",[151,93796,3640],{"class":503},[151,93798,93799,93801,93803,93805,93807],{"class":469,"line":3158},[151,93800,70807],{"class":503},[151,93802,1876],{"class":1869},[151,93804,70835],{"class":503},[151,93806,70838],{"class":481},[151,93808,3640],{"class":503},[151,93810,93811,93813,93815,93817,93820],{"class":469,"line":3167},[151,93812,70807],{"class":503},[151,93814,1876],{"class":1869},[151,93816,70849],{"class":503},[151,93818,93819],{"class":481},"'Cummulative Count'",[151,93821,3640],{"class":503},[151,93823,93824,93827,93830],{"class":469,"line":3175},[151,93825,93826],{"class":503},"plt.savefig(os.path.expanduser(",[151,93828,93829],{"class":481},"'~/Documents/GitHub/briancaffey.github.io/static/subreddit_graph/subreddits_count.png'",[151,93831,12451],{"class":503},[11,93833,93834],{},[2718,93835],{"alt":20386,"src":93836},"/static/subreddit_graph/subreddits_count.png",[14063,93838,93840],{"id":93839},"setting-up-a-graph-with-networkx","Setting up a graph with NetworkX",[11,93842,93843,93844,643],{},"Next we can start to look at the collection of reddits and related subreddits as a graph. I will be using a Python package for network and graph analysis called ",[20,93845,93848],{"href":93846,"rel":93847},"https://networkx.github.io",[24],"NetworkX",[459,93850,93852],{"className":13136,"code":93851,"language":12886,"meta":464,"style":464},"#Let's make sure that we have only unique entries in the dataframe.\nmaster_df_u = master_df_.drop_duplicates('subreddit')\n",[30,93853,93854,93859],{"__ignoreMap":464},[151,93855,93856],{"class":469,"line":470},[151,93857,93858],{"class":1527},"#Let's make sure that we have only unique entries in the dataframe.\n",[151,93860,93861,93864,93866,93869,93871],{"class":469,"line":488},[151,93862,93863],{"class":503},"master_df_u ",[151,93865,1876],{"class":1869},[151,93867,93868],{"class":503}," master_df_.drop_duplicates(",[151,93870,93019],{"class":481},[151,93872,3640],{"class":503},[459,93874,93876],{"className":13136,"code":93875,"language":12886,"meta":464,"style":464},"master_df_u = master_df_u.drop(master_df_u.index[master_df_u.subreddit=='/r/track__subreddits_'])\n",[30,93877,93878],{"__ignoreMap":464},[151,93879,93880,93882,93884,93887,93889,93892],{"class":469,"line":470},[151,93881,93863],{"class":503},[151,93883,1876],{"class":1869},[151,93885,93886],{"class":503}," master_df_u.drop(master_df_u.index[master_df_u.subreddit",[151,93888,17223],{"class":1869},[151,93890,93891],{"class":481},"'/r/track__subreddits_'",[151,93893,38820],{"class":503},[459,93895,93897],{"className":13136,"code":93896,"language":12886,"meta":464,"style":464},"#here we define a dictionary where the keys are subreddits and the values are lists of related subreddits\ngraph = {x:y for x, y in zip(master_df_u.subreddit, master_df_u.related)}\n",[30,93898,93899,93904],{"__ignoreMap":464},[151,93900,93901],{"class":469,"line":470},[151,93902,93903],{"class":1527},"#here we define a dictionary where the keys are subreddits and the values are lists of related subreddits\n",[151,93905,93906,93909,93911,93914,93916,93918,93920,93922],{"class":469,"line":488},[151,93907,93908],{"class":503},"graph ",[151,93910,1876],{"class":1869},[151,93912,93913],{"class":503}," {x:y ",[151,93915,16732],{"class":1869},[151,93917,88158],{"class":503},[151,93919,16417],{"class":1869},[151,93921,44908],{"class":2226},[151,93923,93924],{"class":503},"(master_df_u.subreddit, master_df_u.related)}\n",[459,93926,93928],{"className":13136,"code":93927,"language":12886,"meta":464,"style":464},"#NetworkX comes with the python Anaconda distribution\nimport networkx as nx\n",[30,93929,93930,93935],{"__ignoreMap":464},[151,93931,93932],{"class":469,"line":470},[151,93933,93934],{"class":1527},"#NetworkX comes with the python Anaconda distribution\n",[151,93936,93937,93939,93942,93944],{"class":469,"line":488},[151,93938,16859],{"class":1869},[151,93940,93941],{"class":503}," networkx ",[151,93943,16998],{"class":1869},[151,93945,93946],{"class":503}," nx\n",[459,93948,93950],{"className":13136,"code":93949,"language":12886,"meta":464,"style":464},"G=nx.Graph()\nG=nx.from_dict_of_lists(graph)\n#making the graph undirected takes all of the vertices between nodes and makes them bi-directional\nG1 = G.to_undirected()\n",[30,93951,93952,93962,93971,93976],{"__ignoreMap":464},[151,93953,93954,93957,93959],{"class":469,"line":470},[151,93955,93956],{"class":503},"G",[151,93958,1876],{"class":1869},[151,93960,93961],{"class":503},"nx.Graph()\n",[151,93963,93964,93966,93968],{"class":469,"line":488},[151,93965,93956],{"class":503},[151,93967,1876],{"class":1869},[151,93969,93970],{"class":503},"nx.from_dict_of_lists(graph)\n",[151,93972,93973],{"class":469,"line":500},[151,93974,93975],{"class":1527},"#making the graph undirected takes all of the vertices between nodes and makes them bi-directional\n",[151,93977,93978,93981,93983],{"class":469,"line":509},[151,93979,93980],{"class":503},"G1 ",[151,93982,1876],{"class":1869},[151,93984,93985],{"class":503}," G.to_undirected()\n",[459,93987,93989],{"className":13136,"code":93988,"language":12886,"meta":464,"style":464},"choice = np.random.choice(master_df_u.subreddit, 2)\nprint choice\n",[30,93990,93991,94005],{"__ignoreMap":464},[151,93992,93993,93996,93998,94001,94003],{"class":469,"line":470},[151,93994,93995],{"class":503},"choice ",[151,93997,1876],{"class":1869},[151,93999,94000],{"class":503}," np.random.choice(master_df_u.subreddit, ",[151,94002,6619],{"class":477},[151,94004,3640],{"class":503},[151,94006,94007,94009],{"class":469,"line":488},[151,94008,18513],{"class":2226},[151,94010,94011],{"class":503}," choice\n",[459,94013,94016],{"className":94014,"code":94015,"language":997},[995],"['/r/streetboarding' '/r/stephenking']\n",[30,94017,94015],{"__ignoreMap":464},[11,94019,94020],{},"Let's test out some of the functions from NetworkX for graph analysis. First, let's take the two randomly selected nodes defined above and test to see if there exists a path between them:",[459,94022,94024],{"className":13136,"code":94023,"language":12886,"meta":464,"style":464},"nx.has_path(G1, choice[0], choice[1])\n",[30,94025,94026],{"__ignoreMap":464},[151,94027,94028,94031,94033,94036,94038],{"class":469,"line":470},[151,94029,94030],{"class":503},"nx.has_path(G1, choice[",[151,94032,9181],{"class":477},[151,94034,94035],{"class":503},"], choice[",[151,94037,6760],{"class":477},[151,94039,38820],{"class":503},[459,94041,94044],{"className":94042,"code":94043,"language":997},[995],"True\n",[30,94045,94043],{"__ignoreMap":464},[14063,94047,94049],{"id":94048},"shortest-path","Shortest path",[11,94051,94052],{},"Now let's see (at least one of) the shortest path that exists between these nodes:",[459,94054,94056],{"className":13136,"code":94055,"language":12886,"meta":464,"style":464},"nx.shortest_path(G1, choice[0], choice[1])\n",[30,94057,94058],{"__ignoreMap":464},[151,94059,94060,94063,94065,94067,94069],{"class":469,"line":470},[151,94061,94062],{"class":503},"nx.shortest_path(G1, choice[",[151,94064,9181],{"class":477},[151,94066,94035],{"class":503},[151,94068,6760],{"class":477},[151,94070,38820],{"class":503},[459,94072,94075],{"className":94073,"code":94074,"language":997},[995],"['/r/streetboarding',\n '/r/freebord',\n '/r/adrenaline',\n '/r/imaginaryadrenaline',\n '/r/imaginarystephenking',\n '/r/stephenking']\n",[30,94076,94074],{"__ignoreMap":464},[11,94078,94079],{},"Let's write a function that selects two random subreddits and then prints a shortest path if it exists:",[459,94081,94083],{"className":13136,"code":94082,"language":12886,"meta":464,"style":464},"def short_path():\n    choices = np.random.choice(master_df_u.subreddit, 2)\n    if nx.has_path(G1, choices[0], choices[1]) == True:\n        path = nx.shortest_path(G1, choices[0], choices[1])\n        print choices[0] + ' and ' + choices[1] + ' are joined by: \\n' + str(path)\n    else:\n        print \"No path exists between \" + choices[0] + ' and ' + choices[1]\n",[30,94084,94085,94094,94107,94129,94147,94187,94193],{"__ignoreMap":464},[151,94086,94087,94089,94092],{"class":469,"line":470},[151,94088,16925],{"class":12347},[151,94090,94091],{"class":473}," short_path",[151,94093,16931],{"class":503},[151,94095,94096,94099,94101,94103,94105],{"class":469,"line":488},[151,94097,94098],{"class":503},"    choices ",[151,94100,1876],{"class":1869},[151,94102,94000],{"class":503},[151,94104,6619],{"class":477},[151,94106,3640],{"class":503},[151,94108,94109,94111,94114,94116,94119,94121,94123,94125,94127],{"class":469,"line":500},[151,94110,23327],{"class":1869},[151,94112,94113],{"class":503}," nx.has_path(G1, choices[",[151,94115,9181],{"class":477},[151,94117,94118],{"class":503},"], choices[",[151,94120,6760],{"class":477},[151,94122,24247],{"class":503},[151,94124,17223],{"class":1869},[151,94126,68564],{"class":477},[151,94128,14372],{"class":503},[151,94130,94131,94134,94136,94139,94141,94143,94145],{"class":469,"line":509},[151,94132,94133],{"class":503},"        path ",[151,94135,1876],{"class":1869},[151,94137,94138],{"class":503}," nx.shortest_path(G1, choices[",[151,94140,9181],{"class":477},[151,94142,94118],{"class":503},[151,94144,6760],{"class":477},[151,94146,38820],{"class":503},[151,94148,94149,94151,94154,94156,94158,94160,94163,94165,94167,94169,94171,94173,94176,94178,94180,94182,94184],{"class":469,"line":517},[151,94150,18355],{"class":2226},[151,94152,94153],{"class":503}," choices[",[151,94155,9181],{"class":477},[151,94157,16654],{"class":503},[151,94159,22885],{"class":1869},[151,94161,94162],{"class":481}," ' and '",[151,94164,23378],{"class":1869},[151,94166,94153],{"class":503},[151,94168,6760],{"class":477},[151,94170,16654],{"class":503},[151,94172,22885],{"class":1869},[151,94174,94175],{"class":481}," ' are joined by: ",[151,94177,8043],{"class":477},[151,94179,13223],{"class":481},[151,94181,23378],{"class":1869},[151,94183,84112],{"class":6205},[151,94185,94186],{"class":503},"(path)\n",[151,94188,94189,94191],{"class":469,"line":534},[151,94190,38878],{"class":1869},[151,94192,14372],{"class":503},[151,94194,94195,94197,94200,94202,94204,94206,94208,94210,94212,94214,94216,94218],{"class":469,"line":1413},[151,94196,18355],{"class":2226},[151,94198,94199],{"class":481}," \"No path exists between \"",[151,94201,23378],{"class":1869},[151,94203,94153],{"class":503},[151,94205,9181],{"class":477},[151,94207,16654],{"class":503},[151,94209,22885],{"class":1869},[151,94211,94162],{"class":481},[151,94213,23378],{"class":1869},[151,94215,94153],{"class":503},[151,94217,6760],{"class":477},[151,94219,3691],{"class":503},[11,94221,94222,94223,94226],{},"Here's a collection of results from the ",[30,94224,94225],{},"short_path"," function defined above that start to paint a picuture of the broad set of topics covered by reddit.com:",[459,94228,94230],{"className":13136,"code":94229,"language":12886,"meta":464,"style":464},"short_path()\n",[30,94231,94232],{"__ignoreMap":464},[151,94233,94234],{"class":469,"line":470},[151,94235,94229],{"class":503},[459,94237,94240],{"className":94238,"code":94239,"language":997},[995],"/r/personalizationadvice and /r/beautifulfemales are joined by:\n['/r/personalizationadvice', '/r/coloranalysis', '/r/fashion', '/r/redcarpet', '/r/gentlemanboners', '/r/beautifulfemales']\n",[30,94241,94239],{"__ignoreMap":464},[459,94243,94244],{"className":13136,"code":94229,"language":12886,"meta":464,"style":464},[30,94245,94246],{"__ignoreMap":464},[151,94247,94248],{"class":469,"line":470},[151,94249,94229],{"class":503},[459,94251,94254],{"className":94252,"code":94253,"language":997},[995],"/r/caffeine and /r/shittyramen are joined by:\n['/r/caffeine', '/r/toast', '/r/cooking', '/r/ramen', '/r/shittyramen']\n",[30,94255,94253],{"__ignoreMap":464},[459,94257,94258],{"className":13136,"code":94229,"language":12886,"meta":464,"style":464},[30,94259,94260],{"__ignoreMap":464},[151,94261,94262],{"class":469,"line":470},[151,94263,94229],{"class":503},[459,94265,94268],{"className":94266,"code":94267,"language":997},[995],"/r/watchingcongress and /r/iwantthatonashirt are joined by:\n['/r/watchingcongress', '/r/stand', '/r/snowden', '/r/undelete', '/r/trees', '/r/iwantthatonashirt']\n",[30,94269,94267],{"__ignoreMap":464},[459,94271,94272],{"className":13136,"code":94229,"language":12886,"meta":464,"style":464},[30,94273,94274],{"__ignoreMap":464},[151,94275,94276],{"class":469,"line":470},[151,94277,94229],{"class":503},[459,94279,94282],{"className":94280,"code":94281,"language":997},[995],"/r/asksciencediscussion and /r/dogsonhardwoodfloors are joined by:\n['/r/asksciencediscussion', '/r/badscience', '/r/badlinguistics', '/r/animalsbeingjerks', '/r/startledcats', '/r/dogsonhardwoodfloors']\n",[30,94283,94281],{"__ignoreMap":464},[459,94285,94286],{"className":13136,"code":94229,"language":12886,"meta":464,"style":464},[30,94287,94288],{"__ignoreMap":464},[151,94289,94290],{"class":469,"line":470},[151,94291,94229],{"class":503},[459,94293,94296],{"className":94294,"code":94295,"language":997},[995],"/r/randommail and /r/mini are joined by:\n['/r/randommail', '/r/spiceexchange', '/r/cameraswapping', '/r/itookapicture', '/r/carporn', '/r/mini']\n",[30,94297,94295],{"__ignoreMap":464},[459,94299,94300],{"className":13136,"code":94229,"language":12886,"meta":464,"style":464},[30,94301,94302],{"__ignoreMap":464},[151,94303,94304],{"class":469,"line":470},[151,94305,94229],{"class":503},[459,94307,94310],{"className":94308,"code":94309,"language":997},[995],"/r/catsinsinks and /r/nzmovies are joined by:\n['/r/catsinsinks', '/r/wetcats', '/r/tinysubredditoftheday', '/r/sheep', '/r/nzmetahub', '/r/nzmovies']\n",[30,94311,94309],{"__ignoreMap":464},[459,94313,94314],{"className":13136,"code":94229,"language":12886,"meta":464,"style":464},[30,94315,94316],{"__ignoreMap":464},[151,94317,94318],{"class":469,"line":470},[151,94319,94229],{"class":503},[459,94321,94324],{"className":94322,"code":94323,"language":997},[995],"/r/thoriumreactor and /r/sailing are joined by:\n['/r/thoriumreactor', '/r/energy', '/r/spev', '/r/sailing']\n",[30,94325,94323],{"__ignoreMap":464},[459,94327,94328],{"className":13136,"code":94229,"language":12886,"meta":464,"style":464},[30,94329,94330],{"__ignoreMap":464},[151,94331,94332],{"class":469,"line":470},[151,94333,94229],{"class":503},[459,94335,94338],{"className":94336,"code":94337,"language":997},[995],"/r/deathnote and /r/vegetarianism are joined by:\n['/r/deathnote', '/r/television', '/r/netflixbestof', '/r/naturefilms', '/r/environment', '/r/vegetarianism']\n",[30,94339,94337],{"__ignoreMap":464},[459,94341,94342],{"className":13136,"code":94229,"language":12886,"meta":464,"style":464},[30,94343,94344],{"__ignoreMap":464},[151,94345,94346],{"class":469,"line":470},[151,94347,94229],{"class":503},[459,94349,94352],{"className":94350,"code":94351,"language":997},[995],"/r/mississippir4r and /r/mathematics are joined by:\n['/r/mississippir4r', '/r/mississippi', '/r/prisonreform', '/r/socialscience', '/r/alltech', '/r/mathematics']\n",[30,94353,94351],{"__ignoreMap":464},[459,94355,94356],{"className":13136,"code":94229,"language":12886,"meta":464,"style":464},[30,94357,94358],{"__ignoreMap":464},[151,94359,94360],{"class":469,"line":470},[151,94361,94229],{"class":503},[459,94363,94366],{"className":94364,"code":94365,"language":997},[995],"/r/britainsgottalent and /r/irelandbaldwin are joined by:\n['/r/britainsgottalent', '/r/britishtv', '/r/that70sshow', '/r/mila_kunis', '/r/christinaricci', '/r/irelandbaldwin']\n",[30,94367,94365],{"__ignoreMap":464},[459,94369,94370],{"className":13136,"code":94229,"language":12886,"meta":464,"style":464},[30,94371,94372],{"__ignoreMap":464},[151,94373,94374],{"class":469,"line":470},[151,94375,94229],{"class":503},[459,94377,94380],{"className":94378,"code":94379,"language":997},[995],"/r/the_donald and /r/ladybusiness are joined by:\n['/r/the_donald', '/r/shitliberalssay', '/r/trollxchromosomes', '/r/ladybusiness']\n",[30,94381,94379],{"__ignoreMap":464},[459,94383,94384],{"className":13136,"code":94229,"language":12886,"meta":464,"style":464},[30,94385,94386],{"__ignoreMap":464},[151,94387,94388],{"class":469,"line":470},[151,94389,94229],{"class":503},[459,94391,94394],{"className":94392,"code":94393,"language":997},[995],"/r/selfharm and /r/medlabprofessionals are joined by:\n['/r/selfharm', '/r/adhd', '/r/neuroimaging', '/r/pharmacy', '/r/medlabprofessionals']\n",[30,94395,94393],{"__ignoreMap":464},[459,94397,94398],{"className":13136,"code":94229,"language":12886,"meta":464,"style":464},[30,94399,94400],{"__ignoreMap":464},[151,94401,94402],{"class":469,"line":470},[151,94403,94229],{"class":503},[459,94405,94408],{"className":94406,"code":94407,"language":997},[995],"/r/coverart and /r/phillycraftbeer are joined by:\n['/r/coverart', '/r/nostalgia', '/r/upvotedbecausegirl', '/r/wtf', '/r/remindsmeofdf', '/r/beer', '/r/phillycraftbeer']\n",[30,94409,94407],{"__ignoreMap":464},[459,94411,94412],{"className":13136,"code":94229,"language":12886,"meta":464,"style":464},[30,94413,94414],{"__ignoreMap":464},[151,94415,94416],{"class":469,"line":470},[151,94417,94229],{"class":503},[459,94419,94422],{"className":94420,"code":94421,"language":997},[995],"/r/hotguyswithlonghair and /r/castles are joined by:\n['/r/hotguyswithlonghair', '/r/majesticmanes', '/r/ladyboners', '/r/imaginaryladyboners', '/r/imaginarycastles', '/r/castles']\n",[30,94423,94421],{"__ignoreMap":464},[11,94425,94426,94427,94432,94433,94438],{},"Taking a look ",[20,94428,94431],{"href":94429,"rel":94430},"http://networkx.readthedocs.io/en/networkx-1.11/_modules/networkx/algorithms/shortest_paths/unweighted.html?highlight=bidirectional_shortest_path",[24],"under the hood"," of NetworkX and examining the algorith that finds the ",[20,94434,94437],{"href":94435,"rel":94436},"http://networkx.readthedocs.io/en/networkx-1.11/_modules/networkx/algorithms/shortest_paths/generic.html#shortest_path",[24],"shortest path"," between any two nodes in a graph, we find that it simply boils down to:",[459,94440,94443],{"className":94441,"code":94442,"language":997},[995],"def shortest_path(G, source=None, target=None, weight=None):\n    paths=nx.bidirectional_shortest_path(G,source,target)\n    return paths\n",[30,94444,94442],{"__ignoreMap":464},[11,94446,94447,94448,94451,94452,94455],{},"You can read more about the ",[30,94449,94450],{},"bidirectional_shortest_path"," function ",[20,94453,13074],{"href":94429,"rel":94454},[24]," in the NetworkX documentation.",[11,94457,94458,94459,94462],{},"When I was first experimenting with graph algorithms, I had an interesting result using an algorithm intruduced ",[20,94460,13074],{"href":92535,"rel":94461},[24]," in the Python documentation. Here's the algorithm:",[459,94464,94466],{"className":13136,"code":94465,"language":12886,"meta":464,"style":464},"def find_path(graph, start, end, path=[]):\n    path = path + [start]\n    if start == end:\n        return path\n    if not graph.has_key(start):\n        return None\n    for node in graph[start]:\n        if node not in path:\n            newpath = find_path(graph, node, end, path)\n            if newpath: return newpath\n    return None\n",[30,94467,94468,94497,94511,94523,94529,94538,94544,94555,94568,94578,94590],{"__ignoreMap":464},[151,94469,94470,94472,94475,94477,94480,94482,94484,94486,94488,94490,94492,94494],{"class":469,"line":470},[151,94471,16925],{"class":12347},[151,94473,94474],{"class":473}," find_path",[151,94476,12386],{"class":503},[151,94478,94479],{"class":15232},"graph",[151,94481,106],{"class":503},[151,94483,36608],{"class":15232},[151,94485,106],{"class":503},[151,94487,24306],{"class":15232},[151,94489,106],{"class":503},[151,94491,14462],{"class":15232},[151,94493,1876],{"class":1869},[151,94495,94496],{"class":503},"[]):\n",[151,94498,94499,94501,94503,94506,94508],{"class":469,"line":488},[151,94500,27993],{"class":503},[151,94502,1876],{"class":1869},[151,94504,94505],{"class":503}," path ",[151,94507,22885],{"class":1869},[151,94509,94510],{"class":503}," [start]\n",[151,94512,94513,94515,94518,94520],{"class":469,"line":500},[151,94514,23327],{"class":1869},[151,94516,94517],{"class":503}," start ",[151,94519,17223],{"class":1869},[151,94521,94522],{"class":503}," end:\n",[151,94524,94525,94527],{"class":469,"line":509},[151,94526,16833],{"class":1869},[151,94528,44062],{"class":503},[151,94530,94531,94533,94535],{"class":469,"line":517},[151,94532,23327],{"class":1869},[151,94534,4191],{"class":1869},[151,94536,94537],{"class":503}," graph.has_key(start):\n",[151,94539,94540,94542],{"class":469,"line":534},[151,94541,16833],{"class":1869},[151,94543,53115],{"class":477},[151,94545,94546,94548,94550,94552],{"class":469,"line":1413},[151,94547,16411],{"class":1869},[151,94549,16619],{"class":503},[151,94551,16417],{"class":1869},[151,94553,94554],{"class":503}," graph[start]:\n",[151,94556,94557,94559,94561,94563,94565],{"class":469,"line":1418},[151,94558,23357],{"class":1869},[151,94560,16619],{"class":503},[151,94562,241],{"class":1869},[151,94564,2820],{"class":1869},[151,94566,94567],{"class":503}," path:\n",[151,94569,94570,94573,94575],{"class":469,"line":2462},[151,94571,94572],{"class":503},"            newpath ",[151,94574,1876],{"class":1869},[151,94576,94577],{"class":503}," find_path(graph, node, end, path)\n",[151,94579,94580,94582,94585,94587],{"class":469,"line":2471},[151,94581,40442],{"class":1869},[151,94583,94584],{"class":503}," newpath: ",[151,94586,63121],{"class":1869},[151,94588,94589],{"class":503}," newpath\n",[151,94591,94592,94594],{"class":469,"line":2480},[151,94593,17496],{"class":1869},[151,94595,53115],{"class":477},[11,94597,94598],{},"The above algorthim uses a process called backtracking to exaustively try all possibilities until it returns a solution. It creates an interesting \"random walk\" through groups of related subreddits. Here's the result of calling the above function on our graph (only 2 layers deep) with two random nodes: /r/persianrap and /r/nosleep:",[210,94600,94601],{},[11,94602,94603],{},"/r/persianrap /r/middleeasternmusic /r/arabic /r/arabs /r/libyancrisis /r/syriancivilwar /r/yemenicrisis /r/sinaiinsurgency /r/jihadinfocus /r/credibledefense /r/geopolitics /r/forgottennews /r/libyanconflict /r/menaconflicts /r/iran /r/iranianlgbt /r/zoroastrianism /r/kurdistan /r/rojava /r/anarchism /r/imaginarypolitics /r/imaginaryimmortals /r/imaginaryclerics /r/imaginarylakes /r/imaginaryaliens /r/imaginarygnomes /r/imaginaryladyboners /r/imaginaryturtleworlds /r/imaginarysunnydale /r/imaginarydwarves /r/imaginarywizards /r/imaginaryvikings /r/imaginarycolorscapes /r/imaginarysteampunk /r/imaginarytemples /r/imaginaryblueprints /r/comicbookart /r/imaginarytechnology /r/mtgporn /r/imaginaryoldkingdom /r/imaginaryfactories /r/imaginaryfederation /r/imaginarylovers /r/imaginarynarnia /r/imaginarydwellings /r/imaginaryscience /r/imaginarytaverns /r/imaginarybattlefields /r/cityporn /r/japanpics /r/nationalphotosubs /r/austriapics /r/southkoreapics /r/taiwanpics /r/ghanapics /r/kenyapics /r/norwaypics /r/vzlapics /r/perupics /r/antarcticapics /r/greatlakespics /r/lakeporn /r/pornoverlords /r/thingscutinhalfporn /r/manufacturing /r/cnc /r/askengineers /r/sciencesubreddits /r/math /r/simulate /r/cosmology /r/reddittothefuture /r/scifi /r/lost /r/the100books /r/the100 /r/theblacklist /r/nbc /r/dundermifflin /r/sonsofanarchy /r/twentyfour /r/banshee /r/hbo /r/siliconvalleyhbo /r/siliconvalley /r/california /r/tahoe /r/skiing /r/snowshoeing /r/xcountryskiing /r/wintergear /r/skijumping /r/winter /r/bigmountain /r/mountaineering /r/campingandhiking /r/earthporn /r/nature /r/birding /r/invasivespecies /r/zoology /r/entomology /r/rainforest /r/botany /r/wildlife /r/allscience /r/earthscience /r/energy /r/biomass /r/renewablenews /r/syngas /r/climatenews /r/composting /r/vermiculture /r/organicfarming /r/livestock /r/animalwelfare /r/randomactsofpetfood /r/animalreddits /r/cockatiel /r/catpics /r/tortoises /r/whales /r/cetacea /r/lifeaquatic /r/hrw /r/green_peace /r/environmental_policy /r/conservation /r/depthhub /r/indepthsports /r/deeperhubbeta /r/lectures /r/spacepolicy /r/skylon /r/ula /r/isro /r/engineteststands /r/jupiters /r/imaginarystarscapes /r/spacequestions /r/spaceflight /r/moon /r/dione /r/europa /r/oortcloud /r/dwarfplanetceres /r/saturn /r/asteroidbelt /r/mars /r/rhea /r/venus /r/astrophys /r/spacevideos /r/transhuman /r/timereddits /r/virtualreality /r/vive /r/oculus /r/learnvrdev /r/unity3d /r/gamedev /r/crowdfunding /r/crowdsourcing /r/mturk /r/swagbucks /r/beermoney /r/flipping /r/shoplifting /r/thriftstorehauls /r/dvdcollection /r/televisionposterporn /r/concertposterporn /r/movieposterporn /r/lv426 /r/predator /r/arnoldschwarzenegger /r/alanpartridge /r/americandad /r/timanderic /r/homemovies /r/gravityfalls /r/homestarrunner /r/telltale /r/thewalkingdeadgame /r/thewalkingdeadgifs /r/twdnomansland /r/heycarl /r/twdroadtosurvival /r/thewalkingdead /r/zombies /r/guns /r/swissguns /r/opencarry /r/libertarian /r/geolibertarianism /r/basicincome /r/basicincomeactivism /r/mhoc /r/modelaustralia /r/rmtk /r/thenetherlands /r/tokkiefeesboek /r/nujijinactie /r/ik_ihe /r/youirl /r/fite_me_irl /r/2meirl4meirl /r/depression /r/randomactsofcards /r/philately /r/coins /r/coins4sale /r/ancientcoins /r/ancientrome /r/flatblue /r/bestofwritingprompts /r/writingprompts /r/promptoftheday /r/flashfiction /r/keepwriting /r/getmotivated /r/mentors /r/favors /r/recordthis /r/videography /r/animation /r/3dsmax /r/computergraphics /r/cinema4d /r/design /r/ui_design /r/designjobs /r/heavymind /r/wtfart /r/alternativeart /r/imaginaryninjas /r/imaginaryruins /r/isometric /r/imaginaryislands /r/imaginaryverse /r/icandrawthat /r/caricatures /r/imaginaryneweden /r/imaginaryequestria /r/imaginaryaww /r/imaginarycyberpunk /r/chinafuturism /r/scifirealism /r/inegentlemanboners /r/imaginarywtf /r/imaginaryelementals /r/imaginarydinosaurs /r/dinosaurs /r/speculativeevolution /r/hybridanimals /r/photoshopbattles /r/cutouts /r/battleshops /r/graphic_design /r/visualization /r/statistics /r/oncourtanalytics /r/nbaanalytics /r/nba /r/pacers /r/atlantahawks /r/basketball /r/mavericks /r/fcdallas /r/theticket /r/dallasstars /r/bostonbruins /r/patriots /r/tennesseetitans /r/nashvillesounds /r/predators /r/flyers /r/hockeyfandom /r/caps /r/nhl /r/detroitredwings /r/sabres /r/floridapanthers /r/habs /r/montrealimpact /r/alouettes /r/cfl /r/stadiumporn /r/nfl /r/madden /r/eurobowl /r/fantasyfb /r/fantasyfootball /r/49ers /r/footballgamefilm /r/footballstrategy /r/cfb /r/collegebaseball /r/mlbdraft /r/baseball /r/cubs /r/cardinals /r/saintlouisfc /r/stlouisblues /r/stlouis /r/stlouisbiking /r/mobicycling /r/bicycling /r/vintage_bicycles /r/miamibiking /r/fatbike /r/cycling /r/strava /r/phillycycling /r/wheelbuild /r/bikewrench /r/velo /r/bikepolo /r/bicycletouring /r/bicyclingcirclejerk /r/bikecommuting /r/ukbike /r/leedscycling /r/londoncycling /r/fixedgearbicycle /r/cyclingfashion /r/peloton /r/mtb /r/climbingporn /r/adrenaline /r/motocross /r/bmxracing /r/wake /r/snowboardingnoobs /r/freebord /r/snowboarding /r/sledding /r/outdoors /r/soposts /r/cordcutters /r/netflixviavpn /r/hulu /r/firetv /r/netflixbestof /r/raisinghope /r/madmen /r/earthsgottalent /r/bobsburgers /r/fringe /r/louie /r/theoriginals /r/iansomerhalder /r/kat_graham /r/indianaevans /r/janelevy /r/gagegolightly /r/sarahhyland /r/starlets /r/ninadobrev /r/kathrynnewton /r/arielwinter /r/ashleygreene /r/gentlemanboners /r/bandporn /r/musicpics /r/listentomusic /r/listentonew /r/subraddits /r/dtipics /r/damnthatsinteresting /r/interestingasfuck /r/unexpected /r/wtf /r/weird /r/animalsbeingderps /r/animalsbeingconfused /r/humansbeingbros /r/hulpdiensten /r/askle /r/protectandserve /r/good_cop_free_donut /r/bad_cop_follow_up /r/amifreetogo /r/copwatch /r/puppycide /r/underreportednews /r/mediaquotes /r/savedyouaclick /r/news /r/neutralnews /r/ask_politics /r/politicalopinions /r/gunsarecool /r/renewableenergy /r/web_design /r/somebodymakethis /r/somethingimade /r/crafts /r/kidscrafts /r/daddit /r/formulafeeders /r/boobsandbottles /r/csectioncentral /r/predaddit /r/dadbloggers /r/mombloggers /r/cutekids /r/bigfeats /r/scienceparents /r/lv9hrvv /r/sahp /r/tryingforababy /r/waiting_to_try /r/pcos /r/infertility /r/birthparents /r/tfabchartstalkers /r/firsttimettc /r/cautiousbtb /r/ttchealthy /r/xxketo /r/ketoscience /r/ketogains /r/leangains /r/gettingshredded /r/bulkorcut /r/gainit /r/decidingtobebetter /r/zen /r/buddhism /r/astralprojection /r/spirituality /r/hinduism /r/yoga /r/veganfitness /r/posture /r/health /r/ukhealthcare /r/pharmacy /r/nursing /r/doctorswithoutborders /r/humanitarian /r/assistance /r/paranormalhelp /r/paranormal /r/333 /r/askparanormal /r/intelligence /r/blackhat /r/netsec /r/technology /r/newyorkfuturistparty /r/rad_decentralization /r/massachusettsfp /r/opensource /r/alabamafp /r/darknetplan /r/torrents /r/i2p /r/privacy /r/badgovnofreedom /r/censorship /r/governmentoppression /r/descentintotyranny /r/wikileaks /r/dncleaks /r/hillaryforprison /r/the_donald /r/shitredditsays /r/srsmythos /r/srstrees /r/entwives /r/lesbients /r/actuallesbians /r/lesbianromance /r/lesbianerotica /r/l4l /r/dyke /r/ladyladyboners /r/bisexual /r/bisexy /r/biwomen /r/pansexual /r/genderqueer /r/transspace /r/lgbtlibrary /r/lgbtnews /r/dixiequeer /r/lgbt /r/sex /r/helpmecope /r/bpd /r/rapecounseling /r/trueoffmychest /r/suicidewatch /r/bipolarsos /r/bipolar /r/mentalpod /r/adhd /r/hoarding /r/declutter /r/thrifty /r/tinyhouses /r/leanfire /r/lowcar /r/zerowaste /r/simpleliving /r/livingofftheland /r/hunting /r/animaltracking /r/survival /r/vedc /r/4x4 /r/classiccars /r/automotivetraining /r/autodetailing /r/cartalk /r/mercedes_benz /r/motorsports /r/rallycross /r/worldrallycross /r/blancpain /r/nascarhometracks /r/arcaracing /r/stadiumsupertrucks /r/hydroplanes /r/sailing /r/boatbuilding /r/woodworking /r/cottage_industry /r/farriers /r/blacksmith /r/bladesmith /r/knives /r/swissarmyknives /r/switzerland /r/bern /r/sanktgallen /r/liechtenstein /r/erasmus /r/de /r/germanpuns /r/schland /r/rvacka /r/sloensko /r/slovakia /r/belarus /r/andorra /r/europe /r/hungary /r/francophonie /r/thailand /r/vietnam /r/vietnampics /r/travel /r/geography /r/climate /r/drought /r/waterutilities /r/drylands /r/irrigation /r/water /r/onthewaterfront /r/wetlands /r/marinelife /r/ocean /r/seasteading /r/frontier_colonization /r/arcology /r/retrofuturism /r/goldenpath /r/politics /r/moderationtheory /r/wdp /r/outoftheloop /r/wherearetheynow /r/entertainment /r/portlandia /r/themichaeljfoxshow /r/backtothefuture /r/bladerunner /r/filmnoir /r/vintageladyboners /r/classicfilms /r/foreignmovies /r/britishfilms /r/canadianfilm /r/newjerseyfilm /r/newzealandfilm /r/newzealand /r/wellington /r/nzmetahub /r/newzealandhistory /r/scottishhistory /r/scots /r/scottishproblems /r/britishproblems /r/swedishproblems /r/pinsamt /r/sweden /r/svenskpolitik /r/arbetarrorelsen /r/socialism /r/shittydebatecommunism /r/shittysocialscience /r/shittyideasforadmins /r/shittytheoryofreddit /r/shittybuildingporn /r/shittylifeprotips /r/shittyshitredditsays /r/shittyquotesporn /r/shittyama /r/askashittyparent /r/shittyprogramming /r/shittyaskalawyer /r/badlegaladvice /r/badscience /r/badeconomics /r/badhistory /r/historicalrage /r/metarage /r/ragenovels /r/fffffffuuuuuuuuuuuu /r/gaaaaaaayyyyyyyyyyyy /r/lgbteens /r/needafriend /r/rant /r/showerthoughts /r/markmywords /r/calledit /r/futurewhatif /r/sportswhatif /r/alternatehistory /r/maps /r/xkcd /r/kerbalspaceprogram /r/spacesimgames /r/eve /r/scifigaming /r/masseffect /r/imaginarymasseffect /r/imaginaryvampires /r/imaginarytowers /r/imaginarybestof /r/pics /r/spaceporn /r/auroraporn /r/weatherporn /r/sfwpornnetwork /r/fwepp /r/shittyearthporn /r/shittyaskreddit /r/askashittyphilosopher /r/shittyaskhistory /r/shittysuboftheweek /r/shittyaskcooking /r/shittyhub /r/coolguides /r/trendingsubreddits /r/monkslookingatbeer /r/beerporn /r/beerwithaview /r/shittybeerwithaview /r/shittyfoodporn /r/enttreats /r/trees /r/eldertrees /r/vaporents /r/crainn /r/eirhub /r/fairepublicofireland /r/gaeltacht /r/westmeath /r/tipperary /r/limerick /r/kilkenny /r/ireland /r/irejobs /r/resumes /r/careerguidance /r/flatone /r/centralillinois /r/chicubs /r/whitesox /r/minnesotatwins /r/minnesotavikings /r/greenbaypackers /r/jaguars /r/miamidolphins /r/nflroundtable /r/detroitlions /r/forhonor /r/vikingstv /r/hannibaltv /r/thepathhulu /r/batesmotel /r/hannibal /r/hitchcock /r/silentmoviegifs /r/moviestunts /r/bollywoodrealism /r/indiamain /r/indianews /r/asia /r/oldindia /r/explorepakistan /r/churchporn /r/medievalporn /r/castles /r/historyporn /r/thewaywewere /r/1970s /r/classicmovietrailers /r/warmovies /r/moviecritic /r/trailers /r/liveaction /r/animedeals /r/dbz /r/toonami /r/regularshow /r/thelifeandtimesoftim /r/aquajail /r/modern_family /r/supernatural /r/mishacollins /r/jaredpadalecki /r/fandomnatural /r/fangirls /r/trollxgirlgamers /r/trollmedia /r/trollgaming /r/trollmua /r/justtrollxthings /r/trollxmoms /r/trollmeta /r/trollychromosome /r/oney /r/askwomen /r/okcupid /r/relationship_advice /r/help /r/bugs /r/redditdev /r/enhancement /r/yoursub /r/horrorreviewed /r/truecreepy /r/metatruereddit /r/truepolitics /r/truehub /r/truegaming /r/askgames /r/freegamesonandroid /r/androidapps /r/apphookup /r/browsemyreddit /r/findareddit /r/trap /r/naut /r/militaryfinance /r/army /r/militarystories /r/nationalguard /r/uscg /r/usa /r/murica /r/lonestar /r/whataburger /r/fastfood /r/cocacola /r/kelloggs /r/kellawwggs /r/awwducational /r/marinebiologygifs /r/biologygifs /r/chemicalreactiongifs /r/homechemistry /r/holdmybeaker /r/holdmybeer /r/movieoftheday /r/sharknado /r/syfy /r/killjoys /r/theexpanse /r/truedetective /r/boardwalkempire /r/mobcast /r/1920s /r/1960s /r/beatles /r/minimaluminiumalism /r/ghostsrights /r/botsrights /r/totallynotrobots /r/robotics /r/manna /r/singularity /r/futureporn /r/singularitarianism /r/automate /r/darkfuturology /r/controlproblem /r/aiethics /r/ainothuman /r/neuraljokes /r/3amjokes /r/mommajokes /r/antijokes /r/absolutelynotme_irl /r/toomeirlformeirl /r/meirl /r/tree_irl /r/fishpost /r/mod_irl /r/pics_irl /r/teleshits /r/bitstrips /r/stopbullyingcomics /r/animalsbeingjerks /r/surfinganimals /r/unorthocat /r/catsubs /r/stuffoncats /r/catsinbusinessattire /r/catsinsinks /r/catsonkeyboards /r/mechanicalkeyboards /r/hackedgadgets /r/techsupportmacgyver /r/techsupport /r/programming /r/algorithms /r/datamining /r/datasets /r/wordcloud /r/datavizrequests /r/funnycharts /r/mapporn /r/mapmaking /r/worldbuilding /r/scificoncepts /r/apocalypseporn /r/imaginaryjerk /r/braveryjerk /r/circlejerk /r/politicaldiscussion /r/politicalfactchecking /r/moderatepolitics /r/truereddit /r/malelifestyle /r/fitness /r/swimming /r/freediving /r/bikeshop /r/climbing /r/climbharder /r/bouldering /r/climbergirls /r/womenshredders /r/skatergirls /r/girlsurfers /r/kiteboarding /r/longboarding /r/streetboarding /r/letsgosnowboarding /r/spliddit /r/backcountry /r/wjdbbl2 /r/caving /r/nationalparks /r/parkrangers /r/thesca /r/searchandrescue /r/wildernessbackpacking /r/campinggear /r/flashlight /r/camping /r/yellowstone /r/wmnf /r/pacificcresttrail /r/cdt /r/ultralight /r/backpacking /r/travelpartners /r/adventures /r/libraryofshadows /r/shortscarystories /r/shortscarystoriesooc /r/nosleepooc /r/nosleep",[14063,94605,94607],{"id":94606},"centrality","Centrality",[11,94609,94610,94611,208],{},"Centrality is anohter important topic in graph theory. Here's a brief introduction to centrality from ",[20,94612,64669],{"href":94613,"rel":94614},"https://en.wikipedia.org/wiki/Centrality",[24],[210,94616,94617],{},[11,94618,94619],{},"In graph theory and network analysis, indicators of centrality identify the most important vertices within a graph. Applications include identifying the most influential person(s) in a social network, key infrastructure nodes in the Internet or urban networks, and super-spreaders of disease.",[11,94621,94622,94623,94626],{},"There are several different methods of measuring centrality in a graph. Here I use ",[30,94624,94625],{},"eigenvector_centrality_numpy",", a function included in NetworkX. It takes in a graph and returns a dictionary with graph nodes as keys and node centrality as values.",[459,94628,94630],{"className":13136,"code":94629,"language":12886,"meta":464,"style":464},"centrality = nx.eigenvector_centrality_numpy(G1)\n",[30,94631,94632],{"__ignoreMap":464},[151,94633,94634,94637,94639],{"class":469,"line":470},[151,94635,94636],{"class":503},"centrality ",[151,94638,1876],{"class":1869},[151,94640,94641],{"class":503}," nx.eigenvector_centrality_numpy(G1)\n",[11,94643,94644],{},"Let's see which subreddit has the highest centrality:",[459,94646,94648],{"className":13136,"code":94647,"language":12886,"meta":464,"style":464},"print max(centrality, key=centrality.get), centrality[max(centrality, key=centrality.get)]\n",[30,94649,94650],{"__ignoreMap":464},[151,94651,94652,94654,94656,94659,94661,94663,94666,94669,94671,94673,94675],{"class":469,"line":470},[151,94653,18513],{"class":2226},[151,94655,67577],{"class":2226},[151,94657,94658],{"class":503},"(centrality, ",[151,94660,18175],{"class":15210},[151,94662,1876],{"class":1869},[151,94664,94665],{"class":503},"centrality.get), centrality[",[151,94667,94668],{"class":2226},"max",[151,94670,94658],{"class":503},[151,94672,18175],{"class":15210},[151,94674,1876],{"class":1869},[151,94676,94677],{"class":503},"centrality.get)]\n",[459,94679,94682],{"className":94680,"code":94681,"language":997},[995],"/r/imaginarybattlefields 0.0721530261127\n",[30,94683,94681],{"__ignoreMap":464},[459,94685,94687],{"className":13136,"code":94686,"language":12886,"meta":464,"style":464},"len(centrality) == len(sorted(centrality.values(), reverse=True))\n",[30,94688,94689],{"__ignoreMap":464},[151,94690,94691,94693,94696,94698,94700,94702,94704,94707,94710,94712,94714],{"class":469,"line":470},[151,94692,65875],{"class":2226},[151,94694,94695],{"class":503},"(centrality) ",[151,94697,17223],{"class":1869},[151,94699,45035],{"class":2226},[151,94701,12386],{"class":503},[151,94703,43956],{"class":2226},[151,94705,94706],{"class":503},"(centrality.values(), ",[151,94708,94709],{"class":15210},"reverse",[151,94711,1876],{"class":1869},[151,94713,36962],{"class":477},[151,94715,12451],{"class":503},[459,94717,94719],{"className":94718,"code":94043,"language":997},[995],[30,94720,94043],{"__ignoreMap":464},[11,94722,94723],{},"Since all of the centrality values are unique, we can look up nodes by their centrality values.",[459,94725,94727],{"className":13136,"code":94726,"language":12886,"meta":464,"style":464},"subr_list = []\nfor node in centrality:\n    subr_list.append((node, centrality[node]))\n\nsorted_subr_list = subr_list.sort(key=lambda x: x[1])\n",[30,94728,94729,94738,94749,94754,94758],{"__ignoreMap":464},[151,94730,94731,94734,94736],{"class":469,"line":470},[151,94732,94733],{"class":503},"subr_list ",[151,94735,1876],{"class":1869},[151,94737,16606],{"class":503},[151,94739,94740,94742,94744,94746],{"class":469,"line":488},[151,94741,16732],{"class":1869},[151,94743,16619],{"class":503},[151,94745,16417],{"class":1869},[151,94747,94748],{"class":503}," centrality:\n",[151,94750,94751],{"class":469,"line":500},[151,94752,94753],{"class":503},"    subr_list.append((node, centrality[node]))\n",[151,94755,94756],{"class":469,"line":509},[151,94757,1090],{"emptyLinePlaceholder":609},[151,94759,94760,94763,94765,94768,94770,94772,94774,94776,94778,94780],{"class":469,"line":517},[151,94761,94762],{"class":503},"sorted_subr_list ",[151,94764,1876],{"class":1869},[151,94766,94767],{"class":503}," subr_list.sort(",[151,94769,18175],{"class":15210},[151,94771,1876],{"class":1869},[151,94773,43773],{"class":12347},[151,94775,27729],{"class":15232},[151,94777,44811],{"class":503},[151,94779,6760],{"class":477},[151,94781,38820],{"class":503},[459,94783,94785],{"className":13136,"code":94784,"language":12886,"meta":464,"style":464},"for x in sorted(subr_list, key=lambda x: x[1], reverse=True)[:200]: print x[0],\n",[30,94786,94787],{"__ignoreMap":464},[151,94788,94789,94791,94793,94795,94797,94800,94802,94804,94806,94808,94810,94812,94814,94816,94818,94820,94823,94825,94828,94830,94833,94835],{"class":469,"line":470},[151,94790,16732],{"class":1869},[151,94792,44552],{"class":503},[151,94794,16417],{"class":1869},[151,94796,44767],{"class":2226},[151,94798,94799],{"class":503},"(subr_list, ",[151,94801,18175],{"class":15210},[151,94803,1876],{"class":1869},[151,94805,43773],{"class":12347},[151,94807,27729],{"class":15232},[151,94809,44811],{"class":503},[151,94811,6760],{"class":477},[151,94813,60308],{"class":503},[151,94815,94709],{"class":15210},[151,94817,1876],{"class":1869},[151,94819,36962],{"class":477},[151,94821,94822],{"class":503},")[:",[151,94824,41624],{"class":477},[151,94826,94827],{"class":503},"]: ",[151,94829,18513],{"class":2226},[151,94831,94832],{"class":503}," x[",[151,94834,9181],{"class":477},[151,94836,18746],{"class":503},[459,94838,94841],{"className":94839,"code":94840,"language":997},[995],"/r/imaginarybattlefields /r/imaginarycityscapes /r/imaginarywastelands /r/imaginarywildlands /r/imaginaryleviathans /r/imaginarydragons /r/imaginarystarscapes /r/imaginarywesteros /r/imaginaryartifacts /r/imaginaryangels /r/imaginarymaps /r/imaginarybehemoths /r/imaginarydemons /r/imaginaryelves /r/imaginarycentaurs /r/imaginaryfuturewar /r/imaginarysoldiers /r/imaginaryhistory /r/imaginaryarmor /r/imaginarystarships /r/imaginarynetwork /r/imaginaryjedi /r/imaginarydinosaurs /r/imaginarysteampunk /r/imaginarycyberpunk /r/imaginaryarchers /r/imaginaryvehicles /r/imaginaryanime /r/imaginaryfallout /r/imaginaryastronauts /r/imaginarymusic /r/imaginaryfactories /r/imaginaryequestria /r/imaginarywarships /r/imaginaryazeroth /r/imaginaryarrakis /r/imaginarydisney /r/imaginarypolitics /r/imaginaryhorrors /r/imaginarywinterscapes /r/imaginaryseascapes /r/imaginarypirates /r/imaginarywarriors /r/imaginarymiddleearth /r/imaginarygallifrey /r/imaginarymechs /r/imaginarypropaganda /r/imaginarymerfolk /r/imaginaryvikings /r/imaginaryundead /r/imaginarybeasts /r/imaginarymutants /r/imaginaryruins /r/imaginarytamriel /r/imaginaryforests /r/imaginaryelementals /r/imaginaryskyscapes /r/imaginarymonuments /r/imaginarywaterfalls /r/imaginaryworlds /r/imaginarywizards /r/imaginaryinteriors /r/imaginaryhogwarts /r/imaginarytowers /r/imaginaryarchitecture /r/imaginaryweaponry /r/imaginarygaming /r/imaginarycastles /r/imaginaryrobotics /r/imaginarybooks /r/imaginarygnomes /r/imaginaryvillages /r/imaginarydeserts /r/imaginarywerewolves /r/imaginarydieselpunk /r/imaginaryvampires /r/imaginaryadrenaline /r/imaginarykanto /r/imaginarynatives /r/imaginaryrivers /r/imaginarytemples /r/imaginaryassassins /r/imaginaryvolcanoes /r/imaginaryclerics /r/imaginaryprisons /r/imaginarygiants /r/imaginarycowboys /r/imaginaryhumans /r/imaginarydwarves /r/imaginarycaves /r/imaginarytrolls /r/imaginarywalls /r/imaginarylakes /r/imaginarywitches /r/imaginaryorcs /r/imaginarycanyons /r/imaginaryasylums /r/imaginaryimmortals /r/imaginaryaliens /r/imaginarynobles /r/imaginaryspirits /r/imaginaryaetherpunk /r/imaginarytrees /r/imaginaryislands /r/imaginaryninjas /r/imaginaryscience /r/imaginarymountains /r/imaginaryknights /r/imaginarygoblins /r/imaginaryfaeries /r/imaginarygotham /r/imaginarycybernetics /r/imaginaryooo /r/imaginaryderelicts /r/imaginaryfood /r/imaginaryworldeaters /r/imaginarymindscapes /r/imaginaryaww /r/imaginarymarvel /r/imaginaryweather /r/imaginarynewnewyork /r/imaginaryspidey /r/imaginaryautumnscapes /r/imaginarywarhammer /r/imaginaryfeels /r/imaginarywitcher /r/imaginaryvessels /r/imaginarytaverns /r/imaginarybestof /r/imaginaryairships /r/imaginaryportals /r/imaginaryfashion /r/imaginarylovers /r/imaginarydc /r/imaginaryanimals /r/imaginaryhellscapes /r/imaginarycolorscapes /r/imaginarymonstergirls /r/imaginaryswamps /r/imaginarymythology /r/imaginaryscholars /r/imaginaryladyboners /r/imaginaryfuturism /r/imaginaryaviation /r/imaginarypathways /r/imaginarygatherings /r/imaginarybodyscapes /r/imaginaryoverwatch /r/imaginarydwellings /r/imaginarystephenking /r/specart /r/inegentlemanboners /r/comicbookart /r/imaginarymasseffect /r/imaginaryhalo /r/imaginaryjerk /r/backgroundart /r/futureporn /r/imaginarywallpapers /r/imaginaryfamilies /r/imaginarylibraries /r/imaginaryturtleworlds /r/imaginarydesigns /r/wallpapers /r/apocalypseporn /r/comicbookporn /r/isometric /r/imaginarybakerst /r/imaginaryverse /r/imaginarysunnydale /r/imaginaryfederation /r/imaginarysanctuary /r/starshipporn /r/imaginarystarcraft /r/imaginaryoldkingdom /r/imaginarynarnia /r/imaginarycybertron /r/gameworlds /r/imaginarycarnage /r/imaginaryboners /r/icandrawthat /r/imaginarycosmere /r/imaginaryaperture /r/armoredwomen /r/imaginarywtf /r/unusualart /r/imaginaryblueprints /r/alternativeart /r/sympatheticmonsters /r/adorabledragons /r/imaginarysummerscapes /r/imaginarygayboners /r/imaginarystash /r/artistoftheday /r/imaginaryglaciers /r/imaginaryhybrids /r/imaginaryadventurers /r/imaginarymetropolis /r/craftsoficeandfire /r/popartnouveau\n",[30,94842,94840],{"__ignoreMap":464},[11,94844,94845],{},"There seems to be a network of \"imaginary\" subreddits that have the highest centrality. The members of this network probably all link to themselves as well as many other subreddits as the \"imaginary\" topics span a wide range content. This network may be drowning out other nodes that would otherwise have a high centrality relative to the rest of the subreddits. It might be interesting to eliminate these nodes from the graph and recalculate centrality. Let's look at the distribution of centrality values:",[459,94847,94849],{"className":13136,"code":94848,"language":12886,"meta":464,"style":464},"_ = plt.plot(sorted(centrality.values(), reverse=True)[:1000])\n_ = plt.title('Subreddit Centrality (top 1000)')\n_ = plt.xlabel('Rank')\n_ = plt.ylabel('Centrality')\nplt.savefig(os.path.expanduser('~/Documents/GitHub/briancaffey.github.io/img/subreddit_graph/centrality.png'))\n",[30,94850,94851,94876,94889,94902,94915],{"__ignoreMap":464},[151,94852,94853,94855,94857,94860,94862,94864,94866,94868,94870,94872,94874],{"class":469,"line":470},[151,94854,70807],{"class":503},[151,94856,1876],{"class":1869},[151,94858,94859],{"class":503}," plt.plot(",[151,94861,43956],{"class":2226},[151,94863,94706],{"class":503},[151,94865,94709],{"class":15210},[151,94867,1876],{"class":1869},[151,94869,36962],{"class":477},[151,94871,94822],{"class":503},[151,94873,45779],{"class":477},[151,94875,38820],{"class":503},[151,94877,94878,94880,94882,94884,94887],{"class":469,"line":488},[151,94879,70807],{"class":503},[151,94881,1876],{"class":1869},[151,94883,70821],{"class":503},[151,94885,94886],{"class":481},"'Subreddit Centrality (top 1000)'",[151,94888,3640],{"class":503},[151,94890,94891,94893,94895,94897,94900],{"class":469,"line":500},[151,94892,70807],{"class":503},[151,94894,1876],{"class":1869},[151,94896,70835],{"class":503},[151,94898,94899],{"class":481},"'Rank'",[151,94901,3640],{"class":503},[151,94903,94904,94906,94908,94910,94913],{"class":469,"line":509},[151,94905,70807],{"class":503},[151,94907,1876],{"class":1869},[151,94909,70849],{"class":503},[151,94911,94912],{"class":481},"'Centrality'",[151,94914,3640],{"class":503},[151,94916,94917,94919,94922],{"class":469,"line":517},[151,94918,93826],{"class":503},[151,94920,94921],{"class":481},"'~/Documents/GitHub/briancaffey.github.io/img/subreddit_graph/centrality.png'",[151,94923,12451],{"class":503},[11,94925,94926],{},[2718,94927],{"alt":20386,"src":94928},"/static/subreddit_graph/centrality.png",[14063,94930,94932],{"id":94931},"connectedness","Connectedness",[11,94934,94935],{},"Let's take a look at the graph as a whole. One thing I'm not sure of is whether or not the entire graph is connected. This means that any node can be reached from any other node. Since we constructed the graph from 49 unrelated nodes, it is possible that the graph is unconnected. This would mean that one or more of the default subreddits and its subreddits is not connected with the rest of the graph. In searching for the shortest path I did not come across any pairs of nodes that did not have a path between themselves. I wouldn't be surprised if there are a handful of nodes that stand on their own.",[459,94937,94939],{"className":13136,"code":94938,"language":12886,"meta":464,"style":464},"#size of graph: nodes and edges (or, subreddits and connecting links)\nprint \"Our graph has \" + str(nx.number_of_nodes(G1)) + ' nodes and ' + str(nx.number_of_edges(G1)) + ' edges.'\n",[30,94940,94941,94946],{"__ignoreMap":464},[151,94942,94943],{"class":469,"line":470},[151,94944,94945],{"class":1527},"#size of graph: nodes and edges (or, subreddits and connecting links)\n",[151,94947,94948,94950,94953,94955,94957,94960,94962,94965,94967,94969,94972,94974],{"class":469,"line":488},[151,94949,18513],{"class":2226},[151,94951,94952],{"class":481}," \"Our graph has \"",[151,94954,23378],{"class":1869},[151,94956,84112],{"class":6205},[151,94958,94959],{"class":503},"(nx.number_of_nodes(G1)) ",[151,94961,22885],{"class":1869},[151,94963,94964],{"class":481}," ' nodes and '",[151,94966,23378],{"class":1869},[151,94968,84112],{"class":6205},[151,94970,94971],{"class":503},"(nx.number_of_edges(G1)) ",[151,94973,22885],{"class":1869},[151,94975,94976],{"class":481}," ' edges.'\n",[459,94978,94981],{"className":94979,"code":94980,"language":997},[995],"Our graph has 29854 nodes and 149491 edges.\n",[30,94982,94980],{"__ignoreMap":464},[459,94984,94986],{"className":13136,"code":94985,"language":12886,"meta":464,"style":464},"print \"True of False: our graph is connected... \" + str(nx.is_connected(G1)) + '!'\n",[30,94987,94988],{"__ignoreMap":464},[151,94989,94990,94992,94995,94997,94999,95002,95004],{"class":469,"line":470},[151,94991,18513],{"class":2226},[151,94993,94994],{"class":481}," \"True of False: our graph is connected... \"",[151,94996,23378],{"class":1869},[151,94998,84112],{"class":6205},[151,95000,95001],{"class":503},"(nx.is_connected(G1)) ",[151,95003,22885],{"class":1869},[151,95005,95006],{"class":481}," '!'\n",[459,95008,95011],{"className":95009,"code":95010,"language":997},[995],"True of False: our graph is connected... False!\n",[30,95012,95010],{"__ignoreMap":464},[459,95014,95016],{"className":13136,"code":95015,"language":12886,"meta":464,"style":464},"Gc = max(nx.connected_component_subgraphs(G1), key=len)\nprint \"The largest connected component subgraph has \" + str(nx.number_of_nodes(Gc)) + \" nodes. \"\n",[30,95017,95018,95038],{"__ignoreMap":464},[151,95019,95020,95023,95025,95027,95030,95032,95034,95036],{"class":469,"line":470},[151,95021,95022],{"class":503},"Gc ",[151,95024,1876],{"class":1869},[151,95026,67577],{"class":2226},[151,95028,95029],{"class":503},"(nx.connected_component_subgraphs(G1), ",[151,95031,18175],{"class":15210},[151,95033,1876],{"class":1869},[151,95035,65875],{"class":2226},[151,95037,3640],{"class":503},[151,95039,95040,95042,95045,95047,95049,95052,95054],{"class":469,"line":488},[151,95041,18513],{"class":2226},[151,95043,95044],{"class":481}," \"The largest connected component subgraph has \"",[151,95046,23378],{"class":1869},[151,95048,84112],{"class":6205},[151,95050,95051],{"class":503},"(nx.number_of_nodes(Gc)) ",[151,95053,22885],{"class":1869},[151,95055,95056],{"class":481}," \" nodes. \"\n",[459,95058,95061],{"className":95059,"code":95060,"language":997},[995],"The largest connected component subgraph has 29840 nodes.\n",[30,95062,95060],{"__ignoreMap":464},[11,95064,95065],{},"There are 14 nodes that are not connected to the main connected component. Let's list them.",[459,95067,95069],{"className":13136,"code":95068,"language":12886,"meta":464,"style":464},"for x in list(set(nx.to_dict_of_lists(G1, nodelist=None).keys()) - set(nx.to_dict_of_lists(Gc, nodelist=None).keys())): print x,\n",[30,95070,95071],{"__ignoreMap":464},[151,95072,95073,95075,95077,95079,95081,95083,95085,95088,95091,95093,95095,95098,95100,95102,95105,95107,95109,95111,95114,95116],{"class":469,"line":470},[151,95074,16732],{"class":1869},[151,95076,44552],{"class":503},[151,95078,16417],{"class":1869},[151,95080,59145],{"class":6205},[151,95082,12386],{"class":503},[151,95084,66796],{"class":6205},[151,95086,95087],{"class":503},"(nx.to_dict_of_lists(G1, ",[151,95089,95090],{"class":15210},"nodelist",[151,95092,1876],{"class":1869},[151,95094,15437],{"class":477},[151,95096,95097],{"class":503},").keys()) ",[151,95099,12445],{"class":1869},[151,95101,2309],{"class":6205},[151,95103,95104],{"class":503},"(nx.to_dict_of_lists(Gc, ",[151,95106,95090],{"class":15210},[151,95108,1876],{"class":1869},[151,95110,15437],{"class":477},[151,95112,95113],{"class":503},").keys())): ",[151,95115,18513],{"class":2226},[151,95117,95118],{"class":503}," x,\n",[459,95120,95123],{"className":95121,"code":95122,"language":997},[995],"/r/spacediscussions /r/wtfit.gif /r/space. /r/subreddit_graph /r/vidalia /r/listentothis. /r/history. /r/all. /r/ghostdriver /r/personalfinance. /r/toombscounty /r/gaming /r/science /r/books.\n",[30,95124,95122],{"__ignoreMap":464},[11,95126,95127],{},"Some of the large communities on reddit include /r/books, /r/gaming and /r/science. These subreddits list related subreddits on separate wiki pages since there are many related subreddits for each one. They were most likely all captured in the subsequent levels of the graph, but they also did not link back to /r/science. Here's an example:",[459,95129,95131],{"className":13136,"code":95130,"language":12886,"meta":464,"style":464},"for x in master_df_u.loc[master_df_u.subreddit=='/r/physics'].related: print x\n",[30,95132,95133],{"__ignoreMap":464},[151,95134,95135,95137,95139,95141,95144,95146,95149,95152,95154],{"class":469,"line":470},[151,95136,16732],{"class":1869},[151,95138,44552],{"class":503},[151,95140,16417],{"class":1869},[151,95142,95143],{"class":503}," master_df_u.loc[master_df_u.subreddit",[151,95145,17223],{"class":1869},[151,95147,95148],{"class":481},"'/r/physics'",[151,95150,95151],{"class":503},"].related: ",[151,95153,18513],{"class":2226},[151,95155,95156],{"class":503}," x\n",[459,95158,95161],{"className":95159,"code":95160,"language":997},[995],"['/r/physicsjokes', '/r/gradadmissions', '/r/homeworkhelp', '/r/scienceimages', '/r/askacademia', '/r/physicsgifs', '/r/physicsstudents', '/r/gradschool', '/r/askphysics', '/r/physics']\n",[30,95162,95160],{"__ignoreMap":464},[11,95164,95165,95166,95171],{},"I've got some additional ideas to explore in another post on this topic, such as finding cliques and maximual cliques, and doing graph visualizations with D3.js. If you are interested in playing with the data, you can clone ",[20,95167,95170],{"href":95168,"rel":95169},"https://github.com/briancaffey/reddit-graph-analysis",[24],"my GitHub repo"," and load the pickled DataFrames like this:",[459,95173,95175],{"className":13136,"code":95174,"language":12886,"meta":464,"style":464},"import pandas as pd\ndf = pd.read_pickle('pickle/master_df.p')\n",[30,95176,95177,95187],{"__ignoreMap":464},[151,95178,95179,95181,95183,95185],{"class":469,"line":470},[151,95180,16859],{"class":1869},[151,95182,83790],{"class":503},[151,95184,16998],{"class":1869},[151,95186,83795],{"class":503},[151,95188,95189,95191,95193,95195,95197],{"class":469,"line":488},[151,95190,70720],{"class":503},[151,95192,1876],{"class":1869},[151,95194,93647],{"class":503},[151,95196,93650],{"class":481},[151,95198,3640],{"class":503},[589,95200,95201],{},"html pre.shiki code .sC2Qs, html code.shiki .sC2Qs{--shiki-default:#D73A49;--shiki-dark:#F97583;--shiki-sepia:#F92672}html pre.shiki code .sMOD_, html code.shiki .sMOD_{--shiki-default:#24292E;--shiki-dark:#E1E4E8;--shiki-sepia:#F8F8F2}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html .sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html.sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html pre.shiki code .sstjo, html code.shiki .sstjo{--shiki-default:#032F62;--shiki-dark:#9ECBFF;--shiki-sepia:#E6DB74}html pre.shiki code .s7F3e, html code.shiki .s7F3e{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#AE81FF}html pre.shiki code .sTHNf, html code.shiki .sTHNf{--shiki-default:#E36209;--shiki-default-font-style:inherit;--shiki-dark:#FFAB70;--shiki-dark-font-style:inherit;--shiki-sepia:#FD971F;--shiki-sepia-font-style:italic}html pre.shiki code .sq6CD, html code.shiki .sq6CD{--shiki-default:#D73A49;--shiki-default-font-style:inherit;--shiki-dark:#F97583;--shiki-dark-font-style:inherit;--shiki-sepia:#66D9EF;--shiki-sepia-font-style:italic}html pre.shiki code .sHuvb, html code.shiki .sHuvb{--shiki-default:#22863A;--shiki-default-font-weight:bold;--shiki-dark:#85E89D;--shiki-dark-font-weight:bold;--shiki-sepia:#AE81FF;--shiki-sepia-font-weight:inherit}html pre.shiki code .sFxd3, html code.shiki .sFxd3{--shiki-default:#032F62;--shiki-dark:#DBEDFF;--shiki-sepia:#E6DB74}html pre.shiki code .sLkwE, html code.shiki .sLkwE{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#E6DB74}html pre.shiki code .s-m8C, html code.shiki .s-m8C{--shiki-default:#005CC5;--shiki-default-font-style:inherit;--shiki-dark:#79B8FF;--shiki-dark-font-style:inherit;--shiki-sepia:#66D9EF;--shiki-sepia-font-style:italic}html pre.shiki code .sTrkL, html code.shiki .sTrkL{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#66D9EF}html pre.shiki code .s8-w5, html code.shiki .s8-w5{--shiki-default:#6A737D;--shiki-dark:#6A737D;--shiki-sepia:#88846F}html pre.shiki code .srTi1, html code.shiki .srTi1{--shiki-default:#6F42C1;--shiki-dark:#B392F0;--shiki-sepia:#A6E22E}html pre.shiki code .so59x, html code.shiki .so59x{--shiki-default:#24292E;--shiki-default-font-style:inherit;--shiki-dark:#E1E4E8;--shiki-dark-font-style:inherit;--shiki-sepia:#FD971F;--shiki-sepia-font-style:italic}",{"title":464,"searchDepth":488,"depth":488,"links":95203},[],"2017-03-03","This notebook explores some basic concepts of graph theory. A few weeks ago I set up a script to scrape data from reddit.com with the goal of visualizing the network of related subreddits (forums on specific topics) and related data.","/static/subreddits.png",{"layout":48045},"/2017/03/03/graph_subreddit",{"title":91998,"description":95205},"2017/03/03/graph_subreddit",[23769,12886,46089,12355,95212],"graphs","pD0No-htWYTWM38CUMsXhZHHedMdq9iyf-1Y_vEF19Y",{"id":95215,"title":95216,"body":95217,"comments":609,"date":115029,"description":115030,"draft":602,"extension":605,"external":606,"image":97370,"meta":115031,"navigation":609,"path":115032,"seo":115033,"stem":115034,"tags":115035,"__hash__":115037},"blog/2017/01/01/pc-data.md","PCPartPicker data",{"type":8,"value":95218,"toc":115012},[95219,95227,95247,95251,95254,95477,95486,95489,95579,95586,95589,95823,95837,95843,95846,96016,96022,96030,97312,97315,97366,97371,97382,97385,97390,97393,97418,97421,97425,97428,97540,97543,97633,97643,97651,98795,98799,98802,99006,99009,99520,99525,99528,99531,99663,99673,99804,99812,99817,99921,99927,99981,99990,99999,100192,100197,100217,100223,100248,100253,100262,100265,100268,100321,100324,100365,100370,100373,100389,100403,100406,100510,100515,100518,100521,100654,100659,100662,100665,100817,100822,100825,100827,100830,100841,100951,100956,100959,101093,101098,101107,101112,101118,101123,101273,101278,101281,101413,101418,101421,101621,101626,101629,101632,102047,102050,102163,102169,102272,102340,102346,102599,102604,102607,102632,102637,102640,102643,102659,102664,102672,102675,102697,102702,102715,102720,102733,102736,102739,102742,102746,102752,102963,102968,102971,102975,102978,102981,103131,103136,103139,103142,103287,103292,103295,103441,103446,103450,103458,103463,103466,103646,103651,103654,103842,103847,103850,104029,104034,104037,104287,104292,104296,104299,104302,104524,104529,104532,104538,104543,104546,104682,104687,104690,104884,104889,104891,104894,104897,105193,105198,105201,105204,105758,105763,105771,105774,106112,106117,106120,106126,106129,106135,106138,106145,106151,106159,106166,106169,106382,106387,106390,106395,106398,106596,106601,106604,107003,107008,107011,107014,107174,107179,107183,107186,107189,107361,107366,107369,107495,107500,107503,107703,107708,107711,107914,107919,107922,108119,108124,108127,108277,108282,108285,108439,108444,108447,108648,108653,108656,109334,109339,109342,110116,110121,110124,110629,110634,110640,110644,110653,110658,110667,110946,110951,110954,110957,111131,111136,111139,111344,111349,111352,111572,111577,111580,111584,111587,111786,111791,111794,111797,112143,112148,112153,112156,112491,112496,112501,112504,112507,112511,112514,112681,112686,112703,112711,112716,112719,112722,112726,112729,112732,112735,113190,113195,113198,113201,113403,113408,113412,113415,113538,113543,113546,113550,113555,113559,113564,113568,113573,113577,113582,113586,113591,113594,113597,113600,113604,113613,113776,113782,113788,113795,113918,113921,114041,114047,114058,114061,114084,114087,114092,114097,114102,114105,114110,114115,114120,114123,114132,114135,114139,114147,114152,114155,114404,114440,114445,114448,114452,114455,114458,114470,114720,114724,114734,115009],[11,95220,95221,95222,95226],{},"In the summer of 2016 I built two high-end computers, something I haven't done since 2011. I used ",[20,95223,95225],{"href":95224},"pcpartpicker.com","PCPartPicker"," to research the components and read about PC builds similar to the ones I had in mind. It's a relatively new site that has a strong community of builders, helpful tools to help with part compatibility as well as extensive user reviews on PC components.",[11,95228,95229,95230,95233,95234,95239,95240,95246],{},"Pouring over pages and pages of computer builds, CPUs, video cards and motherboards got me very interested in visualizing the data on both individual components and builds, and the relationship between component specifications and prices. This post will detail the process by which I collected, cleaned, visualized and analyzed all of the data on ",[20,95231,95232],{"href":95224},"PCPartPicker.com",". I've also done some work with natural language process (NLP) to do sentiment analysis on the collection of written user reviews for individual PC parts. If you read to the end you will get to see the two computers I built last summer: ",[20,95235,95236],{"href":23702},[51,95237,95238],{},"Ascension I"," (my personal machine) and ",[20,95241,95242,95245],{"href":23702},[51,95243,95244],{},"Beast Mode II"," (BM2)"," (for my cousin).",[56,95248,95250],{"id":95249},"data-collection-cleaning","Data Collection & Cleaning",[11,95252,95253],{},"For the most part, the data I wanted to collect was well organized and conveniently structured. Since the content on almost all of the pages on PCPartPicker is loaded dynamically, I decided to use a JavaScript engine called PhantomJS to retrieve the HTML after the page loaded. Scraping data would otherwise return only a skeleton of the DOM with no data. Here is the JavaScript file that I used with PhantomJS to scrape data:",[459,95255,95257],{"className":19459,"code":95256,"language":19461,"meta":464,"style":464},"'use strict'\nvar system = require('system')\nvar args = system.args\nvar page = require('webpage').create()\n\npage.onConsoleMessage = function (msg) {\n  console.log(msg)\n}\nvar url = args[1]\npage.open(url, function (status) {\n  if (status === 'success') {\n    page.evaluate(function () {\n      console.log(document.body.innerHTML)\n    })\n  } else {\n    console.log('not success')\n  }\n  phantom.exit(1)\n})\n",[30,95258,95259,95264,95283,95295,95318,95322,95342,95352,95356,95371,95389,95404,95418,95428,95433,95442,95456,95460,95473],{"__ignoreMap":464},[151,95260,95261],{"class":469,"line":470},[151,95262,95263],{"class":481},"'use strict'\n",[151,95265,95266,95268,95271,95273,95276,95278,95281],{"class":469,"line":488},[151,95267,29289],{"class":12347},[151,95269,95270],{"class":503}," system ",[151,95272,1876],{"class":1869},[151,95274,95275],{"class":473}," require",[151,95277,12386],{"class":503},[151,95279,95280],{"class":481},"'system'",[151,95282,3640],{"class":503},[151,95284,95285,95287,95290,95292],{"class":469,"line":500},[151,95286,29289],{"class":12347},[151,95288,95289],{"class":503}," args ",[151,95291,1876],{"class":1869},[151,95293,95294],{"class":503}," system.args\n",[151,95296,95297,95299,95302,95304,95306,95308,95311,95313,95316],{"class":469,"line":509},[151,95298,29289],{"class":12347},[151,95300,95301],{"class":503}," page ",[151,95303,1876],{"class":1869},[151,95305,95275],{"class":473},[151,95307,12386],{"class":503},[151,95309,95310],{"class":481},"'webpage'",[151,95312,13576],{"class":503},[151,95314,95315],{"class":473},"create",[151,95317,12461],{"class":503},[151,95319,95320],{"class":469,"line":517},[151,95321,1090],{"emptyLinePlaceholder":609},[151,95323,95324,95327,95330,95332,95335,95337,95340],{"class":469,"line":534},[151,95325,95326],{"class":503},"page.",[151,95328,95329],{"class":473},"onConsoleMessage",[151,95331,19865],{"class":1869},[151,95333,95334],{"class":12347}," function",[151,95336,129],{"class":503},[151,95338,95339],{"class":15210},"msg",[151,95341,23288],{"class":503},[151,95343,95344,95347,95349],{"class":469,"line":1413},[151,95345,95346],{"class":503},"  console.",[151,95348,70339],{"class":473},[151,95350,95351],{"class":503},"(msg)\n",[151,95353,95354],{"class":469,"line":1418},[151,95355,6274],{"class":503},[151,95357,95358,95360,95362,95364,95367,95369],{"class":469,"line":2462},[151,95359,29289],{"class":12347},[151,95361,60071],{"class":503},[151,95363,1876],{"class":1869},[151,95365,95366],{"class":503}," args[",[151,95368,6760],{"class":477},[151,95370,3691],{"class":503},[151,95372,95373,95375,95378,95381,95383,95385,95387],{"class":469,"line":2471},[151,95374,95326],{"class":503},[151,95376,95377],{"class":473},"open",[151,95379,95380],{"class":503},"(url, ",[151,95382,59958],{"class":12347},[151,95384,129],{"class":503},[151,95386,36594],{"class":15210},[151,95388,23288],{"class":503},[151,95390,95391,95394,95397,95399,95402],{"class":469,"line":2480},[151,95392,95393],{"class":1869},"  if",[151,95395,95396],{"class":503}," (status ",[151,95398,34077],{"class":1869},[151,95400,95401],{"class":481}," 'success'",[151,95403,23288],{"class":503},[151,95405,95406,95409,95412,95414,95416],{"class":469,"line":2489},[151,95407,95408],{"class":503},"    page.",[151,95410,95411],{"class":473},"evaluate",[151,95413,12386],{"class":503},[151,95415,59958],{"class":12347},[151,95417,34033],{"class":503},[151,95419,95420,95423,95425],{"class":469,"line":2497},[151,95421,95422],{"class":503},"      console.",[151,95424,70339],{"class":473},[151,95426,95427],{"class":503},"(document.body.innerHTML)\n",[151,95429,95430],{"class":469,"line":3140},[151,95431,95432],{"class":503},"    })\n",[151,95434,95435,95438,95440],{"class":469,"line":3149},[151,95436,95437],{"class":503},"  } ",[151,95439,77868],{"class":1869},[151,95441,19833],{"class":503},[151,95443,95444,95447,95449,95451,95454],{"class":469,"line":3158},[151,95445,95446],{"class":503},"    console.",[151,95448,70339],{"class":473},[151,95450,12386],{"class":503},[151,95452,95453],{"class":481},"'not success'",[151,95455,3640],{"class":503},[151,95457,95458],{"class":469,"line":3167},[151,95459,19957],{"class":503},[151,95461,95462,95465,95467,95469,95471],{"class":469,"line":3175},[151,95463,95464],{"class":503},"  phantom.",[151,95466,83143],{"class":473},[151,95468,12386],{"class":503},[151,95470,6760],{"class":477},[151,95472,3640],{"class":503},[151,95474,95475],{"class":469,"line":3184},[151,95476,19610],{"class":503},[11,95478,95479,95480,95485],{},"PhantomJS was pretty painful to use, and I've heard of better options for web browser automation and web scrapping like ",[20,95481,95484],{"href":95482,"rel":95483},"http://www.seleniumhq.org/",[24],"Selenium",", so I would probably use that for future projects. PhantomJS works fairly well, and this script, combined with the bash scripts below should be able to serve as a decent template for any similar type of web scraping project.",[11,95487,95488],{},"The first step is to loop through pages containing links of pages for both completed computer builds and computer parts using bash. Here's the bash script I used to scrape links to computer builds:",[459,95490,95492],{"className":461,"code":95491,"language":463,"meta":464,"style":464},"#!/use/bin/env bash\nfor i in `seq 1 1125`;\ndo\n    phantomjs production.js \"https://pcpartpicker.com/builds/#page=\"$i > pages/pages_$i.txt\n    result=$(python -c \"import random;print(random.uniform(2.1, 3.4))\")\n    sleep $result\ndone\n",[30,95493,95494,95499,95520,95524,95549,95567,95575],{"__ignoreMap":464},[151,95495,95496],{"class":469,"line":470},[151,95497,95498],{"class":1527},"#!/use/bin/env bash\n",[151,95500,95501,95503,95505,95507,95509,95511,95513,95516,95518],{"class":469,"line":488},[151,95502,16732],{"class":1869},[151,95504,67225],{"class":503},[151,95506,16417],{"class":1869},[151,95508,2218],{"class":481},[151,95510,75220],{"class":473},[151,95512,12448],{"class":477},[151,95514,95515],{"class":477}," 1125",[151,95517,2798],{"class":481},[151,95519,20086],{"class":503},[151,95521,95522],{"class":469,"line":500},[151,95523,31389],{"class":1869},[151,95525,95526,95529,95532,95535,95538,95540,95543,95546],{"class":469,"line":509},[151,95527,95528],{"class":473},"    phantomjs",[151,95530,95531],{"class":481}," production.js",[151,95533,95534],{"class":481}," \"https://pcpartpicker.com/builds/#page=\"",[151,95536,95537],{"class":503},"$i ",[151,95539,3663],{"class":1869},[151,95541,95542],{"class":481}," pages/pages_",[151,95544,95545],{"class":503},"$i",[151,95547,95548],{"class":481},".txt\n",[151,95550,95551,95554,95556,95558,95560,95562,95565],{"class":469,"line":517},[151,95552,95553],{"class":503},"    result",[151,95555,1876],{"class":1869},[151,95557,31456],{"class":503},[151,95559,12886],{"class":473},[151,95561,75072],{"class":477},[151,95563,95564],{"class":481}," \"import random;print(random.uniform(2.1, 3.4))\"",[151,95566,3640],{"class":503},[151,95568,95569,95572],{"class":469,"line":534},[151,95570,95571],{"class":473},"    sleep",[151,95573,95574],{"class":503}," $result\n",[151,95576,95577],{"class":469,"line":1413},[151,95578,31933],{"class":1869},[11,95580,95581,95582,95585],{},"The result of ",[30,95583,95584],{},"phantomjs"," saves HTML for a given page in the loop to a text file, and then waits for a few seconds before scraping the next page.",[11,95587,95588],{},"I then looped through these pages and pulled out all of the target links for PC builds and PC parts into separate text files using Beautiful Soup. Here's the script for that in python:",[459,95590,95592],{"className":13136,"code":95591,"language":12886,"meta":464,"style":464},"import os\nimport pandas as pd\nfrom bs4 import BeautifulSoup\n\nos.chdir('pages')\nall_links = []\nfor i in os.listdir(os.getcwd()):\n    text = open(i, \"r\")\n    html = text.read()\n    b = BeautifulSoup(html)\n    links = b.findAll('a', attrs={\"class\":\"build-link\"})\n    for a in links:\n        path = a['href']\n        all_links.append(path)\n    b.decompose()\n    text.close()\n\nos.chdir('../')\nurl_file = open('links.txt', 'a')\n\nfor x in all_links:\n    url_file.write('%s\\n' % x)\nurl_file.close()\n",[30,95593,95594,95600,95610,95620,95624,95632,95641,95652,95667,95676,95684,95714,95726,95740,95745,95749,95754,95758,95766,95786,95790,95801,95818],{"__ignoreMap":464},[151,95595,95596,95598],{"class":469,"line":470},[151,95597,16859],{"class":1869},[151,95599,24070],{"class":503},[151,95601,95602,95604,95606,95608],{"class":469,"line":488},[151,95603,16859],{"class":1869},[151,95605,83790],{"class":503},[151,95607,16998],{"class":1869},[151,95609,83795],{"class":503},[151,95611,95612,95614,95616,95618],{"class":469,"line":500},[151,95613,16853],{"class":1869},[151,95615,92070],{"class":503},[151,95617,16859],{"class":1869},[151,95619,92075],{"class":503},[151,95621,95622],{"class":469,"line":509},[151,95623,1090],{"emptyLinePlaceholder":609},[151,95625,95626,95628,95630],{"class":469,"line":517},[151,95627,92835],{"class":503},[151,95629,59770],{"class":481},[151,95631,3640],{"class":503},[151,95633,95634,95637,95639],{"class":469,"line":534},[151,95635,95636],{"class":503},"all_links ",[151,95638,1876],{"class":1869},[151,95640,16606],{"class":503},[151,95642,95643,95645,95647,95649],{"class":469,"line":1413},[151,95644,16732],{"class":1869},[151,95646,67225],{"class":503},[151,95648,16417],{"class":1869},[151,95650,95651],{"class":503}," os.listdir(os.getcwd()):\n",[151,95653,95654,95656,95658,95660,95663,95665],{"class":469,"line":1418},[151,95655,24275],{"class":503},[151,95657,1876],{"class":1869},[151,95659,16970],{"class":2226},[151,95661,95662],{"class":503},"(i, ",[151,95664,16992],{"class":481},[151,95666,3640],{"class":503},[151,95668,95669,95671,95673],{"class":469,"line":2462},[151,95670,64776],{"class":503},[151,95672,1876],{"class":1869},[151,95674,95675],{"class":503}," text.read()\n",[151,95677,95678,95680,95682],{"class":469,"line":2471},[151,95679,92963],{"class":503},[151,95681,1876],{"class":1869},[151,95683,92138],{"class":503},[151,95685,95686,95689,95691,95694,95696,95698,95700,95702,95704,95707,95709,95712],{"class":469,"line":2480},[151,95687,95688],{"class":503},"    links ",[151,95690,1876],{"class":1869},[151,95692,95693],{"class":503}," b.findAll(",[151,95695,65375],{"class":481},[151,95697,106],{"class":503},[151,95699,65333],{"class":15210},[151,95701,1876],{"class":1869},[151,95703,5729],{"class":503},[151,95705,95706],{"class":481},"\"class\"",[151,95708,208],{"class":503},[151,95710,95711],{"class":481},"\"build-link\"",[151,95713,19610],{"class":503},[151,95715,95716,95718,95721,95723],{"class":469,"line":2489},[151,95717,16411],{"class":1869},[151,95719,95720],{"class":503}," a ",[151,95722,16417],{"class":1869},[151,95724,95725],{"class":503}," links:\n",[151,95727,95728,95730,95732,95735,95738],{"class":469,"line":2497},[151,95729,94133],{"class":503},[151,95731,1876],{"class":1869},[151,95733,95734],{"class":503}," a[",[151,95736,95737],{"class":481},"'href'",[151,95739,3691],{"class":503},[151,95741,95742],{"class":469,"line":3140},[151,95743,95744],{"class":503},"        all_links.append(path)\n",[151,95746,95747],{"class":469,"line":3149},[151,95748,93555],{"class":503},[151,95750,95751],{"class":469,"line":3158},[151,95752,95753],{"class":503},"    text.close()\n",[151,95755,95756],{"class":469,"line":3167},[151,95757,1090],{"emptyLinePlaceholder":609},[151,95759,95760,95762,95764],{"class":469,"line":3175},[151,95761,92835],{"class":503},[151,95763,86566],{"class":481},[151,95765,3640],{"class":503},[151,95767,95768,95771,95773,95775,95777,95780,95782,95784],{"class":469,"line":3184},[151,95769,95770],{"class":503},"url_file ",[151,95772,1876],{"class":1869},[151,95774,16970],{"class":2226},[151,95776,12386],{"class":503},[151,95778,95779],{"class":481},"'links.txt'",[151,95781,106],{"class":503},[151,95783,65375],{"class":481},[151,95785,3640],{"class":503},[151,95787,95788],{"class":469,"line":3193},[151,95789,1090],{"emptyLinePlaceholder":609},[151,95791,95792,95794,95796,95798],{"class":469,"line":3720},[151,95793,16732],{"class":1869},[151,95795,44552],{"class":503},[151,95797,16417],{"class":1869},[151,95799,95800],{"class":503}," all_links:\n",[151,95802,95803,95806,95808,95811,95813,95815],{"class":469,"line":3729},[151,95804,95805],{"class":503},"    url_file.write(",[151,95807,13223],{"class":481},[151,95809,95810],{"class":477},"%s\\n",[151,95812,13223],{"class":481},[151,95814,71811],{"class":1869},[151,95816,95817],{"class":503}," x)\n",[151,95819,95820],{"class":469,"line":3735},[151,95821,95822],{"class":503},"url_file.close()\n",[11,95824,95825,95826,95829,95830,95833,95834,208],{},"The first loop goes through all of the pages scraped with PhantomJS and puts each ",[30,95827,95828],{},"build-link"," in a list called ",[30,95831,95832],{},"all_links",". The second loop writes each link to a new line in ",[30,95835,95836],{},"links.txt",[459,95838,95841],{"className":95839,"code":95840,"language":997,"meta":464},[995],"/b/4x4D4D\n/b/ZhVnTW\n/b/8fNG3C\n/b/7KWZxr\n/b/Qm6hP6\n/b/X4rH99\n...\n",[30,95842,95840],{"__ignoreMap":464},[11,95844,95845],{},"The next step is one more bash script that scrapes HTML from individual builds and saves it to individual text files.",[459,95847,95849],{"className":461,"code":95848,"language":463,"meta":464,"style":464},"#!/use/bin/env bash\n\ncounter=1\nwhile read NAME; do\n    echo \"[STATUS] Scraping build number $counter: $NAME...\"\n    file=\"new_build-\"${NAME//\\/b\\//$counter-}\n    phantomjs scrape_build.js \"https://pcpartpicker.com$NAME\" > builds/$file.txt\n    result=$(python -c \"import random;print(random.uniform(2.5, 4.4))\")\n    echo \"[STATUS] Completed scraping build number $counter ($NAME). Sleeping for $result seconds...\"\n    sleep $result\n    ((counter += 1))\ndone \u003C links.txt\n",[30,95850,95851,95855,95859,95868,95882,95899,95926,95950,95967,95989,95995,96006],{"__ignoreMap":464},[151,95852,95853],{"class":469,"line":470},[151,95854,95498],{"class":1527},[151,95856,95857],{"class":469,"line":488},[151,95858,1090],{"emptyLinePlaceholder":609},[151,95860,95861,95864,95866],{"class":469,"line":500},[151,95862,95863],{"class":503},"counter",[151,95865,1876],{"class":1869},[151,95867,1963],{"class":481},[151,95869,95870,95872,95875,95877,95880],{"class":469,"line":509},[151,95871,68561],{"class":1869},[151,95873,95874],{"class":2226}," read",[151,95876,3950],{"class":481},[151,95878,95879],{"class":503},"; ",[151,95881,31389],{"class":1869},[151,95883,95884,95887,95890,95893,95895,95897],{"class":469,"line":517},[151,95885,95886],{"class":2226},"    echo",[151,95888,95889],{"class":481}," \"[STATUS] Scraping build number ",[151,95891,95892],{"class":503},"$counter",[151,95894,6208],{"class":481},[151,95896,64366],{"class":503},[151,95898,31637],{"class":481},[151,95900,95901,95904,95906,95909,95912,95915,95917,95919,95921,95923],{"class":469,"line":534},[151,95902,95903],{"class":503},"    file",[151,95905,1876],{"class":1869},[151,95907,95908],{"class":481},"\"new_build-\"",[151,95910,95911],{"class":503},"${NAME",[151,95913,95914],{"class":1869},"//",[151,95916,92185],{"class":477},[151,95918,78488],{"class":503},[151,95920,92185],{"class":477},[151,95922,19883],{"class":1869},[151,95924,95925],{"class":503},"$counter-}\n",[151,95927,95928,95930,95933,95936,95938,95940,95942,95945,95948],{"class":469,"line":1413},[151,95929,95528],{"class":473},[151,95931,95932],{"class":481}," scrape_build.js",[151,95934,95935],{"class":481}," \"https://pcpartpicker.com",[151,95937,64366],{"class":503},[151,95939,8592],{"class":481},[151,95941,4832],{"class":1869},[151,95943,95944],{"class":481}," builds/",[151,95946,95947],{"class":503},"$file",[151,95949,95548],{"class":481},[151,95951,95952,95954,95956,95958,95960,95962,95965],{"class":469,"line":1418},[151,95953,95553],{"class":503},[151,95955,1876],{"class":1869},[151,95957,31456],{"class":503},[151,95959,12886],{"class":473},[151,95961,75072],{"class":477},[151,95963,95964],{"class":481}," \"import random;print(random.uniform(2.5, 4.4))\"",[151,95966,3640],{"class":503},[151,95968,95969,95971,95974,95976,95978,95980,95983,95986],{"class":469,"line":2462},[151,95970,95886],{"class":2226},[151,95972,95973],{"class":481}," \"[STATUS] Completed scraping build number ",[151,95975,95892],{"class":503},[151,95977,129],{"class":481},[151,95979,64366],{"class":503},[151,95981,95982],{"class":481},"). Sleeping for ",[151,95984,95985],{"class":503},"$result",[151,95987,95988],{"class":481}," seconds...\"\n",[151,95990,95991,95993],{"class":469,"line":2471},[151,95992,95571],{"class":473},[151,95994,95574],{"class":503},[151,95996,95997,96000,96002,96004],{"class":469,"line":2480},[151,95998,95999],{"class":503},"    ((counter ",[151,96001,24780],{"class":1869},[151,96003,12448],{"class":477},[151,96005,12451],{"class":503},[151,96007,96008,96011,96013],{"class":469,"line":2489},[151,96009,96010],{"class":1869},"done",[151,96012,75367],{"class":1869},[151,96014,96015],{"class":503}," links.txt\n",[11,96017,96018,96019,96021],{},"The script takes lines from ",[30,96020,95836],{}," and appends them to the base URL, scrapes the HTML from the resulting link and saves it to a text file.",[11,96023,96024,96025,96029],{},"The next step is to go through the text files for each PC build and create a pandas DataFrame containing data for each PC build (~25,000 builds total). Here's a sample PC build link: ",[20,96026,96027],{"href":96027,"rel":96028},"https://pcpartpicker.com/b/VD3bt6",[24],". This heavily commented script shows step-by-step how to scrape all of the relevant data from the HTML for PC builds:",[459,96031,96033],{"className":13136,"code":96032,"language":12886,"meta":464,"style":464},"import os\nimport pandas as pd\nfrom bs4 import BeautifulSoup\n\nos.chdir('builds/builds_html')\nbuild_dict_list = []\n\nfiles = os.listdir(os.getcwd())\n\nfor i in files:\n\n    try:\n        a = open(i, \"r\")\n        b = BeautifulSoup(a)\n\n        #Labels: Each build list has labels, for example: Video Card: GeForce GTX 1080\n        labels = b.find('ul', attrs={\"class\":\"parts\"}).findAll('p', attrs={\"label\"})\n\n        #The number of labels varies between builds\n        #Some builds have multiple parts of the same type\n        cols = [label.contents[0] for label in labels]\n\n        #This will keep track of how many of each part there are\n        count = [0 for x in range(len(cols))]\n\n        #This handles multiple parts of the same type (e.g. 2 or more graphics cards)\n        for x in range(len(cols)):\n            count[x] = 1 + cols[0:x].count(cols[x])\n\n        #Rename the labels with the corresponding count appended to the end\n        new_cols = [str(x)+\"_\"+str(y) for (x,y) in zip(cols, count)]\n\n        #Values (Names of parts)\n        values = b.find('ul', attrs={\"class\":\"parts\"}).findAll('a', attrs={\"name\"})\n        vals = [value.contents[0] for value in values]\n\n        #Links\n        link_list = b.find('ul', attrs={\"class\":\"parts\"}).findAll('a', attrs={\"name\"})\n        link_list = [link['href'] for link in link_list]\n\n        #Prices: there is a an individual price for each part and a total price for the build\n        prices = b.find('ul', attrs={\"class\":\"parts\"}).findAll('div', attrs={\"price\"})\n        price_list = [price.contents[0].strip() for price in prices]\n\n        #This creates a dictionary for each component of the build\n        #Contains the component name, link and price (the price reported by the user)\n        build_part_dict = {part: {\"Link\":link,\"Price\":price,\"Name\":name} for part, link, price, name in zip(new_cols, link_list, price_list,vals)}\n\n        #Total price\n        total_price = float(b.find(\"li\",attrs={\"class\":\"partlist-total\"}).find('span', attrs={\"class\":\"price\"}).contents[0].strip(\"$\"))\n        total_price_dict = {\"total\":total_price}\n\n        #Description\n        description_paragraphs = b.find('div', attrs={'class':'description block'}).find_all('p')\n        description_clean = ''\n        for paragraph in description_paragraphs:\n            add = paragraph.text\n            description_clean += add + ' '\n        description_dict = {'Description':description_clean}\n\n        #Builder\n        build_name = b.find('h1', attrs={\"class\":\"name\"}).contents[0]\n        builder_link = b.find('p', attrs={\"class\":\"owner\"}).find('a')['href']\n        builder_dict = {\"Builder\":build_name}\n        owner_dict = {\"Owner\":builder_link}\n\n        #Details: Details include clock speed, temperatures, date built, etc.\n        #The process is similar to sraping the components, but there is no need to worry about duplicate parts\n        detail_vals = [x.strip(u\" \").strip(u'\\n').strip() for x in b.find('div', attrs={\"class\":\"part-details\"}) if \"\\n \" in x]\n        details = [x.contents[0] for x in b.find('div', attrs={\"class\":\"part-details\"}).find_all('h4')]\n        detail_dict = {detail:detail_val for detail_val, detail in zip(detail_vals, details)}\n\n        #Permalink\n        perma_link = b.find('input', attrs={\"type\":\"text\"})['value'].split(\"/b/\")[1]\n        build_dict = {\"Permalink\": perma_link, \"Builder\":builder_link}\n\n        #Join all of the above dictionaries into a master build dictionary\n        master_dict = dict(build_dict.items()+builder_dict.items()+build_part_dict.items()+detail_dict.items()+total_price_dict.items()+owner_dict.items() + description_dict.items())\n\n        #Add the dictionary to build_dict_list defined above\n        build_dict_list.append(master_dict)\n\n        #This is important for preventing memory leaks in bs4\n        b.decompose()\n        a.close()\n\n    #Some builds have been deleted, even though the links are still listed, so we pass the build if there are any issues scraping data from it\n    except:\n        pass\n\n#Create a pandas DataFrame from build_dict_list\ndf = pd.DataFrame(build_dict_list)\nos.chdir('/Users/andrewcaffey/Documents/Projects/Data/PCPP/builds/')\n#Save the DataFrame to a csv file\ndf.to_csv('builds.csv', encoding='utf-8')\n",[30,96034,96035,96041,96051,96061,96065,96074,96083,96087,96096,96100,96110,96114,96120,96135,96145,96149,96154,96200,96204,96209,96214,96238,96242,96247,96273,96277,96282,96299,96318,96322,96327,96364,96368,96373,96415,96438,96442,96447,96488,96511,96515,96520,96562,96587,96591,96596,96601,96640,96644,96649,96711,96726,96730,96735,96768,96778,96790,96800,96815,96830,96834,96839,96871,96907,96922,96937,96941,96946,96951,97020,97065,97087,97091,97096,97140,97159,97163,97168,97210,97214,97219,97224,97228,97233,97238,97243,97247,97252,97258,97262,97266,97271,97280,97289,97294],{"__ignoreMap":464},[151,96036,96037,96039],{"class":469,"line":470},[151,96038,16859],{"class":1869},[151,96040,24070],{"class":503},[151,96042,96043,96045,96047,96049],{"class":469,"line":488},[151,96044,16859],{"class":1869},[151,96046,83790],{"class":503},[151,96048,16998],{"class":1869},[151,96050,83795],{"class":503},[151,96052,96053,96055,96057,96059],{"class":469,"line":500},[151,96054,16853],{"class":1869},[151,96056,92070],{"class":503},[151,96058,16859],{"class":1869},[151,96060,92075],{"class":503},[151,96062,96063],{"class":469,"line":509},[151,96064,1090],{"emptyLinePlaceholder":609},[151,96066,96067,96069,96072],{"class":469,"line":517},[151,96068,92835],{"class":503},[151,96070,96071],{"class":481},"'builds/builds_html'",[151,96073,3640],{"class":503},[151,96075,96076,96079,96081],{"class":469,"line":534},[151,96077,96078],{"class":503},"build_dict_list ",[151,96080,1876],{"class":1869},[151,96082,16606],{"class":503},[151,96084,96085],{"class":469,"line":1413},[151,96086,1090],{"emptyLinePlaceholder":609},[151,96088,96089,96091,96093],{"class":469,"line":1418},[151,96090,64699],{"class":503},[151,96092,1876],{"class":1869},[151,96094,96095],{"class":503}," os.listdir(os.getcwd())\n",[151,96097,96098],{"class":469,"line":2462},[151,96099,1090],{"emptyLinePlaceholder":609},[151,96101,96102,96104,96106,96108],{"class":469,"line":2471},[151,96103,16732],{"class":1869},[151,96105,67225],{"class":503},[151,96107,16417],{"class":1869},[151,96109,65212],{"class":503},[151,96111,96112],{"class":469,"line":2480},[151,96113,1090],{"emptyLinePlaceholder":609},[151,96115,96116,96118],{"class":469,"line":2489},[151,96117,18280],{"class":1869},[151,96119,14372],{"class":503},[151,96121,96122,96125,96127,96129,96131,96133],{"class":469,"line":2497},[151,96123,96124],{"class":503},"        a ",[151,96126,1876],{"class":1869},[151,96128,16970],{"class":2226},[151,96130,95662],{"class":503},[151,96132,16992],{"class":481},[151,96134,3640],{"class":503},[151,96136,96137,96140,96142],{"class":469,"line":3140},[151,96138,96139],{"class":503},"        b ",[151,96141,1876],{"class":1869},[151,96143,96144],{"class":503}," BeautifulSoup(a)\n",[151,96146,96147],{"class":469,"line":3149},[151,96148,1090],{"emptyLinePlaceholder":609},[151,96150,96151],{"class":469,"line":3158},[151,96152,96153],{"class":1527},"        #Labels: Each build list has labels, for example: Video Card: GeForce GTX 1080\n",[151,96155,96156,96159,96161,96163,96166,96168,96170,96172,96174,96176,96178,96181,96184,96187,96189,96191,96193,96195,96198],{"class":469,"line":3167},[151,96157,96158],{"class":503},"        labels ",[151,96160,1876],{"class":1869},[151,96162,93041],{"class":503},[151,96164,96165],{"class":481},"'ul'",[151,96167,106],{"class":503},[151,96169,65333],{"class":15210},[151,96171,1876],{"class":1869},[151,96173,5729],{"class":503},[151,96175,95706],{"class":481},[151,96177,208],{"class":503},[151,96179,96180],{"class":481},"\"parts\"",[151,96182,96183],{"class":503},"}).findAll(",[151,96185,96186],{"class":481},"'p'",[151,96188,106],{"class":503},[151,96190,65333],{"class":15210},[151,96192,1876],{"class":1869},[151,96194,5729],{"class":503},[151,96196,96197],{"class":481},"\"label\"",[151,96199,19610],{"class":503},[151,96201,96202],{"class":469,"line":3175},[151,96203,1090],{"emptyLinePlaceholder":609},[151,96205,96206],{"class":469,"line":3184},[151,96207,96208],{"class":1527},"        #The number of labels varies between builds\n",[151,96210,96211],{"class":469,"line":3193},[151,96212,96213],{"class":1527},"        #Some builds have multiple parts of the same type\n",[151,96215,96216,96219,96221,96224,96226,96228,96230,96233,96235],{"class":469,"line":3720},[151,96217,96218],{"class":503},"        cols ",[151,96220,1876],{"class":1869},[151,96222,96223],{"class":503}," [label.contents[",[151,96225,9181],{"class":477},[151,96227,16654],{"class":503},[151,96229,16732],{"class":1869},[151,96231,96232],{"class":503}," label ",[151,96234,16417],{"class":1869},[151,96236,96237],{"class":503}," labels]\n",[151,96239,96240],{"class":469,"line":3729},[151,96241,1090],{"emptyLinePlaceholder":609},[151,96243,96244],{"class":469,"line":3735},[151,96245,96246],{"class":1527},"        #This will keep track of how many of each part there are\n",[151,96248,96249,96252,96254,96256,96258,96260,96262,96264,96266,96268,96270],{"class":469,"line":3745},[151,96250,96251],{"class":503},"        count ",[151,96253,1876],{"class":1869},[151,96255,6604],{"class":503},[151,96257,9181],{"class":477},[151,96259,2235],{"class":1869},[151,96261,44552],{"class":503},[151,96263,16417],{"class":1869},[151,96265,2793],{"class":2226},[151,96267,12386],{"class":503},[151,96269,65875],{"class":2226},[151,96271,96272],{"class":503},"(cols))]\n",[151,96274,96275],{"class":469,"line":3754},[151,96276,1090],{"emptyLinePlaceholder":609},[151,96278,96279],{"class":469,"line":3760},[151,96280,96281],{"class":1527},"        #This handles multiple parts of the same type (e.g. 2 or more graphics cards)\n",[151,96283,96284,96286,96288,96290,96292,96294,96296],{"class":469,"line":3773},[151,96285,16616],{"class":1869},[151,96287,44552],{"class":503},[151,96289,16417],{"class":1869},[151,96291,2793],{"class":2226},[151,96293,12386],{"class":503},[151,96295,65875],{"class":2226},[151,96297,96298],{"class":503},"(cols)):\n",[151,96300,96301,96304,96306,96308,96310,96313,96315],{"class":469,"line":3782},[151,96302,96303],{"class":503},"            count[x] ",[151,96305,1876],{"class":1869},[151,96307,12448],{"class":477},[151,96309,23378],{"class":1869},[151,96311,96312],{"class":503}," cols[",[151,96314,9181],{"class":477},[151,96316,96317],{"class":503},":x].count(cols[x])\n",[151,96319,96320],{"class":469,"line":3791},[151,96321,1090],{"emptyLinePlaceholder":609},[151,96323,96324],{"class":469,"line":3803},[151,96325,96326],{"class":1527},"        #Rename the labels with the corresponding count appended to the end\n",[151,96328,96329,96332,96334,96336,96338,96341,96343,96345,96347,96349,96352,96354,96357,96359,96361],{"class":469,"line":3811},[151,96330,96331],{"class":503},"        new_cols ",[151,96333,1876],{"class":1869},[151,96335,6604],{"class":503},[151,96337,15343],{"class":6205},[151,96339,96340],{"class":503},"(x)",[151,96342,22885],{"class":1869},[151,96344,78320],{"class":481},[151,96346,22885],{"class":1869},[151,96348,15343],{"class":6205},[151,96350,96351],{"class":503},"(y) ",[151,96353,16732],{"class":1869},[151,96355,96356],{"class":503}," (x,y) ",[151,96358,16417],{"class":1869},[151,96360,44908],{"class":2226},[151,96362,96363],{"class":503},"(cols, count)]\n",[151,96365,96366],{"class":469,"line":3820},[151,96367,1090],{"emptyLinePlaceholder":609},[151,96369,96370],{"class":469,"line":7084},[151,96371,96372],{"class":1527},"        #Values (Names of parts)\n",[151,96374,96375,96378,96380,96382,96384,96386,96388,96390,96392,96394,96396,96398,96400,96402,96404,96406,96408,96410,96413],{"class":469,"line":7148},[151,96376,96377],{"class":503},"        values ",[151,96379,1876],{"class":1869},[151,96381,93041],{"class":503},[151,96383,96165],{"class":481},[151,96385,106],{"class":503},[151,96387,65333],{"class":15210},[151,96389,1876],{"class":1869},[151,96391,5729],{"class":503},[151,96393,95706],{"class":481},[151,96395,208],{"class":503},[151,96397,96180],{"class":481},[151,96399,96183],{"class":503},[151,96401,65375],{"class":481},[151,96403,106],{"class":503},[151,96405,65333],{"class":15210},[151,96407,1876],{"class":1869},[151,96409,5729],{"class":503},[151,96411,96412],{"class":481},"\"name\"",[151,96414,19610],{"class":503},[151,96416,96417,96420,96422,96425,96427,96429,96431,96434,96436],{"class":469,"line":7211},[151,96418,96419],{"class":503},"        vals ",[151,96421,1876],{"class":1869},[151,96423,96424],{"class":503}," [value.contents[",[151,96426,9181],{"class":477},[151,96428,16654],{"class":503},[151,96430,16732],{"class":1869},[151,96432,96433],{"class":503}," value ",[151,96435,16417],{"class":1869},[151,96437,67567],{"class":503},[151,96439,96440],{"class":469,"line":7273},[151,96441,1090],{"emptyLinePlaceholder":609},[151,96443,96444],{"class":469,"line":7335},[151,96445,96446],{"class":1527},"        #Links\n",[151,96448,96449,96452,96454,96456,96458,96460,96462,96464,96466,96468,96470,96472,96474,96476,96478,96480,96482,96484,96486],{"class":469,"line":7398},[151,96450,96451],{"class":503},"        link_list ",[151,96453,1876],{"class":1869},[151,96455,93041],{"class":503},[151,96457,96165],{"class":481},[151,96459,106],{"class":503},[151,96461,65333],{"class":15210},[151,96463,1876],{"class":1869},[151,96465,5729],{"class":503},[151,96467,95706],{"class":481},[151,96469,208],{"class":503},[151,96471,96180],{"class":481},[151,96473,96183],{"class":503},[151,96475,65375],{"class":481},[151,96477,106],{"class":503},[151,96479,65333],{"class":15210},[151,96481,1876],{"class":1869},[151,96483,5729],{"class":503},[151,96485,96412],{"class":481},[151,96487,19610],{"class":503},[151,96489,96490,96492,96494,96497,96499,96501,96503,96506,96508],{"class":469,"line":7462},[151,96491,96451],{"class":503},[151,96493,1876],{"class":1869},[151,96495,96496],{"class":503}," [link[",[151,96498,95737],{"class":481},[151,96500,16654],{"class":503},[151,96502,16732],{"class":1869},[151,96504,96505],{"class":503}," link ",[151,96507,16417],{"class":1869},[151,96509,96510],{"class":503}," link_list]\n",[151,96512,96513],{"class":469,"line":7467},[151,96514,1090],{"emptyLinePlaceholder":609},[151,96516,96517],{"class":469,"line":7532},[151,96518,96519],{"class":1527},"        #Prices: there is a an individual price for each part and a total price for the build\n",[151,96521,96522,96525,96527,96529,96531,96533,96535,96537,96539,96541,96543,96545,96547,96549,96551,96553,96555,96557,96560],{"class":469,"line":7537},[151,96523,96524],{"class":503},"        prices ",[151,96526,1876],{"class":1869},[151,96528,93041],{"class":503},[151,96530,96165],{"class":481},[151,96532,106],{"class":503},[151,96534,65333],{"class":15210},[151,96536,1876],{"class":1869},[151,96538,5729],{"class":503},[151,96540,95706],{"class":481},[151,96542,208],{"class":503},[151,96544,96180],{"class":481},[151,96546,96183],{"class":503},[151,96548,92151],{"class":481},[151,96550,106],{"class":503},[151,96552,65333],{"class":15210},[151,96554,1876],{"class":1869},[151,96556,5729],{"class":503},[151,96558,96559],{"class":481},"\"price\"",[151,96561,19610],{"class":503},[151,96563,96564,96567,96569,96572,96574,96577,96579,96582,96584],{"class":469,"line":7603},[151,96565,96566],{"class":503},"        price_list ",[151,96568,1876],{"class":1869},[151,96570,96571],{"class":503}," [price.contents[",[151,96573,9181],{"class":477},[151,96575,96576],{"class":503},"].strip() ",[151,96578,16732],{"class":1869},[151,96580,96581],{"class":503}," price ",[151,96583,16417],{"class":1869},[151,96585,96586],{"class":503}," prices]\n",[151,96588,96589],{"class":469,"line":7608},[151,96590,1090],{"emptyLinePlaceholder":609},[151,96592,96593],{"class":469,"line":7673},[151,96594,96595],{"class":1527},"        #This creates a dictionary for each component of the build\n",[151,96597,96598],{"class":469,"line":7678},[151,96599,96600],{"class":1527},"        #Contains the component name, link and price (the price reported by the user)\n",[151,96602,96603,96606,96608,96611,96614,96617,96620,96623,96625,96628,96630,96633,96635,96637],{"class":469,"line":7708},[151,96604,96605],{"class":503},"        build_part_dict ",[151,96607,1876],{"class":1869},[151,96609,96610],{"class":503}," {part: {",[151,96612,96613],{"class":481},"\"Link\"",[151,96615,96616],{"class":503},":link,",[151,96618,96619],{"class":481},"\"Price\"",[151,96621,96622],{"class":503},":price,",[151,96624,71461],{"class":481},[151,96626,96627],{"class":503},":name} ",[151,96629,16732],{"class":1869},[151,96631,96632],{"class":503}," part, link, price, name ",[151,96634,16417],{"class":1869},[151,96636,44908],{"class":2226},[151,96638,96639],{"class":503},"(new_cols, link_list, price_list,vals)}\n",[151,96641,96642],{"class":469,"line":7713},[151,96643,1090],{"emptyLinePlaceholder":609},[151,96645,96646],{"class":469,"line":7746},[151,96647,96648],{"class":1527},"        #Total price\n",[151,96650,96651,96654,96656,96659,96662,96665,96667,96669,96671,96673,96675,96677,96680,96682,96684,96686,96688,96690,96692,96694,96696,96698,96701,96703,96706,96709],{"class":469,"line":7751},[151,96652,96653],{"class":503},"        total_price ",[151,96655,1876],{"class":1869},[151,96657,96658],{"class":6205}," float",[151,96660,96661],{"class":503},"(b.find(",[151,96663,96664],{"class":481},"\"li\"",[151,96666,3634],{"class":503},[151,96668,65333],{"class":15210},[151,96670,1876],{"class":1869},[151,96672,5729],{"class":503},[151,96674,95706],{"class":481},[151,96676,208],{"class":503},[151,96678,96679],{"class":481},"\"partlist-total\"",[151,96681,93101],{"class":503},[151,96683,93044],{"class":481},[151,96685,106],{"class":503},[151,96687,65333],{"class":15210},[151,96689,1876],{"class":1869},[151,96691,5729],{"class":503},[151,96693,95706],{"class":481},[151,96695,208],{"class":503},[151,96697,96559],{"class":481},[151,96699,96700],{"class":503},"}).contents[",[151,96702,9181],{"class":477},[151,96704,96705],{"class":503},"].strip(",[151,96707,96708],{"class":481},"\"$\"",[151,96710,12451],{"class":503},[151,96712,96713,96716,96718,96720,96723],{"class":469,"line":7816},[151,96714,96715],{"class":503},"        total_price_dict ",[151,96717,1876],{"class":1869},[151,96719,52023],{"class":503},[151,96721,96722],{"class":481},"\"total\"",[151,96724,96725],{"class":503},":total_price}\n",[151,96727,96728],{"class":469,"line":7821},[151,96729,1090],{"emptyLinePlaceholder":609},[151,96731,96732],{"class":469,"line":7847},[151,96733,96734],{"class":1527},"        #Description\n",[151,96736,96737,96740,96742,96744,96746,96748,96750,96752,96754,96756,96758,96761,96764,96766],{"class":469,"line":7852},[151,96738,96739],{"class":503},"        description_paragraphs ",[151,96741,1876],{"class":1869},[151,96743,93041],{"class":503},[151,96745,92151],{"class":481},[151,96747,106],{"class":503},[151,96749,65333],{"class":15210},[151,96751,1876],{"class":1869},[151,96753,5729],{"class":503},[151,96755,71242],{"class":481},[151,96757,208],{"class":503},[151,96759,96760],{"class":481},"'description block'",[151,96762,96763],{"class":503},"}).find_all(",[151,96765,96186],{"class":481},[151,96767,3640],{"class":503},[151,96769,96770,96773,96775],{"class":469,"line":7887},[151,96771,96772],{"class":503},"        description_clean ",[151,96774,1876],{"class":1869},[151,96776,96777],{"class":481}," ''\n",[151,96779,96780,96782,96785,96787],{"class":469,"line":7892},[151,96781,16616],{"class":1869},[151,96783,96784],{"class":503}," paragraph ",[151,96786,16417],{"class":1869},[151,96788,96789],{"class":503}," description_paragraphs:\n",[151,96791,96792,96795,96797],{"class":469,"line":7924},[151,96793,96794],{"class":503},"            add ",[151,96796,1876],{"class":1869},[151,96798,96799],{"class":503}," paragraph.text\n",[151,96801,96802,96805,96807,96810,96812],{"class":469,"line":7929},[151,96803,96804],{"class":503},"            description_clean ",[151,96806,24780],{"class":1869},[151,96808,96809],{"class":503}," add ",[151,96811,22885],{"class":1869},[151,96813,96814],{"class":481}," ' '\n",[151,96816,96817,96820,96822,96824,96827],{"class":469,"line":7991},[151,96818,96819],{"class":503},"        description_dict ",[151,96821,1876],{"class":1869},[151,96823,52023],{"class":503},[151,96825,96826],{"class":481},"'Description'",[151,96828,96829],{"class":503},":description_clean}\n",[151,96831,96832],{"class":469,"line":7996},[151,96833,1090],{"emptyLinePlaceholder":609},[151,96835,96836],{"class":469,"line":8078},[151,96837,96838],{"class":1527},"        #Builder\n",[151,96840,96841,96844,96846,96848,96851,96853,96855,96857,96859,96861,96863,96865,96867,96869],{"class":469,"line":8140},[151,96842,96843],{"class":503},"        build_name ",[151,96845,1876],{"class":1869},[151,96847,93041],{"class":503},[151,96849,96850],{"class":481},"'h1'",[151,96852,106],{"class":503},[151,96854,65333],{"class":15210},[151,96856,1876],{"class":1869},[151,96858,5729],{"class":503},[151,96860,95706],{"class":481},[151,96862,208],{"class":503},[151,96864,96412],{"class":481},[151,96866,96700],{"class":503},[151,96868,9181],{"class":477},[151,96870,3691],{"class":503},[151,96872,96873,96876,96878,96880,96882,96884,96886,96888,96890,96892,96894,96897,96899,96901,96903,96905],{"class":469,"line":8145},[151,96874,96875],{"class":503},"        builder_link ",[151,96877,1876],{"class":1869},[151,96879,93041],{"class":503},[151,96881,96186],{"class":481},[151,96883,106],{"class":503},[151,96885,65333],{"class":15210},[151,96887,1876],{"class":1869},[151,96889,5729],{"class":503},[151,96891,95706],{"class":481},[151,96893,208],{"class":503},[151,96895,96896],{"class":481},"\"owner\"",[151,96898,93101],{"class":503},[151,96900,65375],{"class":481},[151,96902,40832],{"class":503},[151,96904,95737],{"class":481},[151,96906,3691],{"class":503},[151,96908,96909,96912,96914,96916,96919],{"class":469,"line":8259},[151,96910,96911],{"class":503},"        builder_dict ",[151,96913,1876],{"class":1869},[151,96915,52023],{"class":503},[151,96917,96918],{"class":481},"\"Builder\"",[151,96920,96921],{"class":503},":build_name}\n",[151,96923,96924,96927,96929,96931,96934],{"class":469,"line":8264},[151,96925,96926],{"class":503},"        owner_dict ",[151,96928,1876],{"class":1869},[151,96930,52023],{"class":503},[151,96932,96933],{"class":481},"\"Owner\"",[151,96935,96936],{"class":503},":builder_link}\n",[151,96938,96939],{"class":469,"line":8613},[151,96940,1090],{"emptyLinePlaceholder":609},[151,96942,96943],{"class":469,"line":8678},[151,96944,96945],{"class":1527},"        #Details: Details include clock speed, temperatures, date built, etc.\n",[151,96947,96948],{"class":469,"line":8742},[151,96949,96950],{"class":1527},"        #The process is similar to sraping the components, but there is no need to worry about duplicate parts\n",[151,96952,96953,96956,96958,96961,96963,96965,96968,96970,96972,96974,96976,96979,96981,96983,96985,96987,96989,96991,96993,96995,96997,96999,97001,97004,97007,97009,97011,97013,97015,97017],{"class":469,"line":8806},[151,96954,96955],{"class":503},"        detail_vals ",[151,96957,1876],{"class":1869},[151,96959,96960],{"class":503}," [x.strip(",[151,96962,68688],{"class":12347},[151,96964,24311],{"class":481},[151,96966,96967],{"class":503},").strip(",[151,96969,68688],{"class":12347},[151,96971,13223],{"class":481},[151,96973,8043],{"class":477},[151,96975,13223],{"class":481},[151,96977,96978],{"class":503},").strip() ",[151,96980,16732],{"class":1869},[151,96982,44552],{"class":503},[151,96984,16417],{"class":1869},[151,96986,93041],{"class":503},[151,96988,92151],{"class":481},[151,96990,106],{"class":503},[151,96992,65333],{"class":15210},[151,96994,1876],{"class":1869},[151,96996,5729],{"class":503},[151,96998,95706],{"class":481},[151,97000,208],{"class":503},[151,97002,97003],{"class":481},"\"part-details\"",[151,97005,97006],{"class":503},"}) ",[151,97008,17218],{"class":1869},[151,97010,16722],{"class":481},[151,97012,8043],{"class":477},[151,97014,16722],{"class":481},[151,97016,2820],{"class":1869},[151,97018,97019],{"class":503}," x]\n",[151,97021,97022,97025,97027,97030,97032,97034,97036,97038,97040,97042,97044,97046,97048,97050,97052,97054,97056,97058,97060,97063],{"class":469,"line":8870},[151,97023,97024],{"class":503},"        details ",[151,97026,1876],{"class":1869},[151,97028,97029],{"class":503}," [x.contents[",[151,97031,9181],{"class":477},[151,97033,16654],{"class":503},[151,97035,16732],{"class":1869},[151,97037,44552],{"class":503},[151,97039,16417],{"class":1869},[151,97041,93041],{"class":503},[151,97043,92151],{"class":481},[151,97045,106],{"class":503},[151,97047,65333],{"class":15210},[151,97049,1876],{"class":1869},[151,97051,5729],{"class":503},[151,97053,95706],{"class":481},[151,97055,208],{"class":503},[151,97057,97003],{"class":481},[151,97059,96763],{"class":503},[151,97061,97062],{"class":481},"'h4'",[151,97064,44576],{"class":503},[151,97066,97067,97070,97072,97075,97077,97080,97082,97084],{"class":469,"line":8875},[151,97068,97069],{"class":503},"        detail_dict ",[151,97071,1876],{"class":1869},[151,97073,97074],{"class":503}," {detail:detail_val ",[151,97076,16732],{"class":1869},[151,97078,97079],{"class":503}," detail_val, detail ",[151,97081,16417],{"class":1869},[151,97083,44908],{"class":2226},[151,97085,97086],{"class":503},"(detail_vals, details)}\n",[151,97088,97089],{"class":469,"line":8881},[151,97090,1090],{"emptyLinePlaceholder":609},[151,97092,97093],{"class":469,"line":8886},[151,97094,97095],{"class":1527},"        #Permalink\n",[151,97097,97098,97101,97103,97105,97108,97110,97112,97114,97116,97118,97120,97123,97126,97129,97131,97134,97136,97138],{"class":469,"line":8892},[151,97099,97100],{"class":503},"        perma_link ",[151,97102,1876],{"class":1869},[151,97104,93041],{"class":503},[151,97106,97107],{"class":481},"'input'",[151,97109,106],{"class":503},[151,97111,65333],{"class":15210},[151,97113,1876],{"class":1869},[151,97115,5729],{"class":503},[151,97117,38850],{"class":481},[151,97119,208],{"class":503},[151,97121,97122],{"class":481},"\"text\"",[151,97124,97125],{"class":503},"})[",[151,97127,97128],{"class":481},"'value'",[151,97130,52005],{"class":503},[151,97132,97133],{"class":481},"\"/b/\"",[151,97135,40832],{"class":503},[151,97137,6760],{"class":477},[151,97139,3691],{"class":503},[151,97141,97142,97145,97147,97149,97152,97155,97157],{"class":469,"line":8963},[151,97143,97144],{"class":503},"        build_dict ",[151,97146,1876],{"class":1869},[151,97148,52023],{"class":503},[151,97150,97151],{"class":481},"\"Permalink\"",[151,97153,97154],{"class":503},": perma_link, ",[151,97156,96918],{"class":481},[151,97158,96936],{"class":503},[151,97160,97161],{"class":469,"line":8969},[151,97162,1090],{"emptyLinePlaceholder":609},[151,97164,97165],{"class":469,"line":15001},[151,97166,97167],{"class":1527},"        #Join all of the above dictionaries into a master build dictionary\n",[151,97169,97170,97173,97175,97177,97180,97182,97185,97187,97190,97192,97195,97197,97200,97202,97205,97207],{"class":469,"line":15009},[151,97171,97172],{"class":503},"        master_dict ",[151,97174,1876],{"class":1869},[151,97176,51817],{"class":6205},[151,97178,97179],{"class":503},"(build_dict.items()",[151,97181,22885],{"class":1869},[151,97183,97184],{"class":503},"builder_dict.items()",[151,97186,22885],{"class":1869},[151,97188,97189],{"class":503},"build_part_dict.items()",[151,97191,22885],{"class":1869},[151,97193,97194],{"class":503},"detail_dict.items()",[151,97196,22885],{"class":1869},[151,97198,97199],{"class":503},"total_price_dict.items()",[151,97201,22885],{"class":1869},[151,97203,97204],{"class":503},"owner_dict.items() ",[151,97206,22885],{"class":1869},[151,97208,97209],{"class":503}," description_dict.items())\n",[151,97211,97212],{"class":469,"line":15019},[151,97213,1090],{"emptyLinePlaceholder":609},[151,97215,97216],{"class":469,"line":15027},[151,97217,97218],{"class":1527},"        #Add the dictionary to build_dict_list defined above\n",[151,97220,97221],{"class":469,"line":15037},[151,97222,97223],{"class":503},"        build_dict_list.append(master_dict)\n",[151,97225,97226],{"class":469,"line":15045},[151,97227,1090],{"emptyLinePlaceholder":609},[151,97229,97230],{"class":469,"line":15055},[151,97231,97232],{"class":1527},"        #This is important for preventing memory leaks in bs4\n",[151,97234,97235],{"class":469,"line":15060},[151,97236,97237],{"class":503},"        b.decompose()\n",[151,97239,97240],{"class":469,"line":15068},[151,97241,97242],{"class":503},"        a.close()\n",[151,97244,97245],{"class":469,"line":15076},[151,97246,1090],{"emptyLinePlaceholder":609},[151,97248,97249],{"class":469,"line":15085},[151,97250,97251],{"class":1527},"    #Some builds have been deleted, even though the links are still listed, so we pass the build if there are any issues scraping data from it\n",[151,97253,97254,97256],{"class":469,"line":15095},[151,97255,18341],{"class":1869},[151,97257,14372],{"class":503},[151,97259,97260],{"class":469,"line":15105},[151,97261,65399],{"class":1869},[151,97263,97264],{"class":469,"line":15110},[151,97265,1090],{"emptyLinePlaceholder":609},[151,97267,97268],{"class":469,"line":15118},[151,97269,97270],{"class":1527},"#Create a pandas DataFrame from build_dict_list\n",[151,97272,97273,97275,97277],{"class":469,"line":15128},[151,97274,70720],{"class":503},[151,97276,1876],{"class":1869},[151,97278,97279],{"class":503}," pd.DataFrame(build_dict_list)\n",[151,97281,97282,97284,97287],{"class":469,"line":15139},[151,97283,92835],{"class":503},[151,97285,97286],{"class":481},"'/Users/andrewcaffey/Documents/Projects/Data/PCPP/builds/'",[151,97288,3640],{"class":503},[151,97290,97291],{"class":469,"line":31954},[151,97292,97293],{"class":1527},"#Save the DataFrame to a csv file\n",[151,97295,97296,97298,97301,97303,97306,97308,97310],{"class":469,"line":31960},[151,97297,86622],{"class":503},[151,97299,97300],{"class":481},"'builds.csv'",[151,97302,106],{"class":503},[151,97304,97305],{"class":15210},"encoding",[151,97307,1876],{"class":1869},[151,97309,68697],{"class":481},[151,97311,3640],{"class":503},[11,97313,97314],{},"At this point we can do a quick visualization of the distribution of PC prices:",[459,97316,97318],{"className":13136,"code":97317,"language":12886,"meta":464,"style":464},"df.total[(df.total>0)&(df.total\u003C7000)].hist(bins=100, figsize=(20,10))\n",[30,97319,97320],{"__ignoreMap":464},[151,97321,97322,97325,97327,97329,97331,97333,97336,97338,97341,97344,97346,97348,97350,97352,97354,97356,97358,97360,97362,97364],{"class":469,"line":470},[151,97323,97324],{"class":503},"df.total[(df.total",[151,97326,3663],{"class":1869},[151,97328,9181],{"class":477},[151,97330,748],{"class":503},[151,97332,54214],{"class":1869},[151,97334,97335],{"class":503},"(df.total",[151,97337,3613],{"class":1869},[151,97339,97340],{"class":477},"7000",[151,97342,97343],{"class":503},")].hist(",[151,97345,87626],{"class":15210},[151,97347,1876],{"class":1869},[151,97349,71821],{"class":477},[151,97351,106],{"class":503},[151,97353,44358],{"class":15210},[151,97355,1876],{"class":1869},[151,97357,12386],{"class":503},[151,97359,9097],{"class":477},[151,97361,3634],{"class":503},[151,97363,12423],{"class":477},[151,97365,12451],{"class":503},[11,97367,97368],{},[2718,97369],{"alt":20386,"src":97370},"/static/pcpp/hist.png",[11,97372,97373,97374,97377,97378,97381],{},"The mean PC price is ",[15,97375,97376],{},"$1,292",". However, the price of PCs is not reflected accurately in ",[30,97379,97380],{},"df.total",". I noticed that some builds include multiple monitors while others don't include any and some builders don't include prices for components from their previous PC builds.",[11,97383,97384],{},"The data frame contains 141 columns for parts. Here they are:",[210,97386,97387],{},[11,97388,97389],{},"All-In-One Monitor/Chassis_1 CPU Cooler_1 CPU Cooler_2 CPU Cooler_3 CPU_1 CPU_2 Case Accessory_1 Case Accessory_2 Case Fan_1 Case Fan_10 Case Fan_11 Case Fan_12 Case Fan_13 Case Fan_14 Case Fan_15 Case Fan_16 Case Fan_17 Case Fan_18 Case Fan_19 Case Fan_2 Case Fan_20 Case Fan_21 Case Fan_22 Case Fan_23 Case Fan_24 Case Fan_25 Case Fan_26 Case Fan_27 Case Fan_28 Case Fan_29 Case Fan_3 Case Fan_30 Case Fan_31 Case Fan_32 Case Fan_4 Case Fan_5 Case Fan_6 Case Fan_7 Case Fan_8 Case Fan_9 Case_1 Coolant_1 External Storage_1 External Storage_2 External Storage_3 External Storage_4 Fan Controller_1 Fan Controller_2 Food_1 Food_2 Food_3 Headphones_1 Headphones_2 Headphones_3 Headphones_4 Keyboard_1 Keyboard_2 Keyboard_3 Keyboard_4 Keyboard_5 Keyboard_6 Keyboard_7 Laptop_1 Memory_1 Memory_2 Memory_3 Memory_4 Memory_5 Memory_6 Memory_7 Memory_8 Monitor_1 Monitor_2 Monitor_3 Monitor_4 Monitor_5 Monitor_6 Motherboard_1 Mouse_1 Mouse_2 Mouse_3 Mouse_4 Mouse_5 Operating System_1 Optical Drive_1 Optical Drive_2 Optical Drive_3 Optical Drive_4 Power Supply_1 Radiator_1 Reservoir_1 Software_1 Software_2 Software_3 Software_4 Software_5 Software_6 Sound Card_1 Sound Card_2 Speakers_1 Speakers_2 Storage_1 Storage_10 Storage_11 Storage_12 Storage_13 Storage_14 Storage_15 Storage_16 Storage_17 Storage_18 Storage_19 Storage_2 Storage_20 Storage_21 Storage_22 Storage_23 Storage_24 Storage_25 Storage_3 Storage_4 Storage_5 Storage_6 Storage_7 Storage_8 Storage_9 Thermal Compound_1 Thermal Compound_2 Thermal Compound_3 Thermal Compound_4 UPS_1 Video Card Cooler_1 Video Card Cooler_2 Video Card_1 Video Card_2 Video Card_3 Video Card_4 Wired Network Adapter_1 Wired Network Adapter_2 Wireless Network Adapter_1 Wireless Network Adapter_2",[11,97391,97392],{},"That's right, somebody included mulitple food items in their PC build! One build listed 29 hard drive disks and another listed 32 case fans. So, it will make more sense to look at individual PC builds by their core components:",[76,97394,97395,97398,97400,97403,97406,97409,97412,97415],{},[79,97396,97397],{},"case",[79,97399,72752],{},[79,97401,97402],{},"GPU (multiple)",[79,97404,97405],{},"motherboard",[79,97407,97408],{},"memory (multiple)",[79,97410,97411],{},"storage (multiple)",[79,97413,97414],{},"CPU cooler",[79,97416,97417],{},"PSU (power supply unit)",[11,97419,97420],{},"Most of the PC builds have at least one of these core components.",[56,97422,97424],{"id":97423},"part-data","Part Data",[11,97426,97427],{},"Collecting data for individual parts followed the same process as collecting data for completed builds. Here are the counts of parts I collected by type:",[1131,97429,97430,97486],{},[1134,97431,97432],{},[1137,97433,97434,97438,97440,97442,97444,97447,97449,97452,97454,97456,97458,97461,97463,97466,97468,97471,97473,97476,97478,97481,97483],{},[1140,97435,97437],{"align":97436},"center","Case",[1140,97439,12445],{},[1140,97441,72752],{"align":97436},[1140,97443,12445],{},[1140,97445,97446],{"align":97436},"CPU Coooler",[1140,97448,12445],{},[1140,97450,97451],{"align":97436},"Case Fan",[1140,97453,12445],{},[1140,97455,73035],{"align":97436},[1140,97457,12445],{},[1140,97459,97460],{"align":97436},"Hard Drive",[1140,97462,12445],{},[1140,97464,97465],{"align":97436},"Memory",[1140,97467,12445],{},[1140,97469,97470],{"align":97436},"Monitor",[1140,97472,12445],{},[1140,97474,97475],{"align":97436},"Motherboard",[1140,97477,12445],{},[1140,97479,97480],{"align":97436},"PSU",[1140,97482,12445],{},[1140,97484,97485],{"align":97436},"UPS",[1153,97487,97488],{},[1137,97489,97490,97493,97495,97498,97500,97502,97504,97507,97509,97512,97514,97517,97519,97522,97524,97526,97528,97531,97533,97536,97538],{},[1158,97491,97492],{"align":97436},"2774",[1158,97494],{},[1158,97496,97497],{"align":97436},"886",[1158,97499],{},[1158,97501,74063],{"align":97436},[1158,97503],{},[1158,97505,97506],{"align":97436},"1192",[1158,97508],{},[1158,97510,97511],{"align":97436},"2996",[1158,97513],{},[1158,97515,97516],{"align":97436},"1736",[1158,97518],{},[1158,97520,97521],{"align":97436},"1700",[1158,97523],{},[1158,97525,44836],{"align":97436},[1158,97527],{},[1158,97529,97530],{"align":97436},"2400",[1158,97532],{},[1158,97534,97535],{"align":97436},"1434",[1158,97537],{},[1158,97539,59584],{"align":97436},[11,97541,97542],{},"Here's a quick look at the features I am interested in for each part:",[76,97544,97545,97553,97561,97569,97577,97585,97593,97601,97609,97617,97625],{},[79,97546,97547,97548],{},"Case\n",[76,97549,97550],{},[79,97551,97552],{},"Color, Manufacturer, Name, Dimensions, Volume, Average Price, Type",[79,97554,97555,97556],{},"CPU\n",[76,97557,97558],{},[79,97559,97560],{},"Name, Manufacturer, Lithography, TDP, Operating Frequency, Boost Frequency, Core Count, Hyperthreading, Maximum Supported Memory, Average Price",[79,97562,97563,97564],{},"CPU Cooler\n",[76,97565,97566],{},[79,97567,97568],{},"Manufacturer, Maximum Noise Level, Maximum RPM, Liquid Cooled, Radiator Size, Bearing Type, Height",[79,97570,97571,97572],{},"Case Fan\n",[76,97573,97574],{},[79,97575,97576],{},"RPM",[79,97578,97579,97580],{},"GPU\n",[76,97581,97582],{},[79,97583,97584],{},"Memory (GB), NVIDIA/AMD, Clock Speed (MHz), Boost Clock Speed (MHz), Chipset, Manufacturer, TDP, Model",[79,97586,97587,97588],{},"Hard Drive\n",[76,97589,97590],{},[79,97591,97592],{},"Storage (GB), RPM, SSD/Spinning, Price/GB, Form Factor, Manufacturer",[79,97594,97595,97596],{},"Memory\n",[76,97597,97598],{},[79,97599,97600],{},"Manufacturer', CAS, Price/GB, DDR3/DDR4, Speed, DIMM, Size (GB), Module Count, Module Size, Voltage",[79,97602,97603,97604],{},"Monitor\n",[76,97605,97606],{},[79,97607,97608],{},"Refresh rate, Response Time (ms), Screen Size, Viewing Angle, Aspect Ratio, Brightness, Display Colors, Manufacturer, LED, Recommended Resolution, Wide Screen, Curved Screen",[79,97610,97611,97612],{},"Motherboard\n",[76,97613,97614],{},[79,97615,97616],{},"Socket, Maximum Supported Memory, Memory Slots, Chipset",[79,97618,97619,97620],{},"Power Supply Unit (PSU)\n",[76,97621,97622],{},[79,97623,97624],{},"Modular, Power, Price/Watt, Manufacturer, Efficiency Certification",[79,97626,97627,97628],{},"UPS\n",[76,97629,97630],{},[79,97631,97632],{},"Charge time",[11,97634,97635,97636,97639,97640,97642],{},"There's a lot of data for each part, some parts are missing a lot of price data. Each part has a list of vendors with prices that are in the same neighborhood. To impute missing prices on the ",[30,97637,97638],{},"builds"," DataFrame, I will be imputing the average price. For parts missing pricing data, where it makes sense, I'll be using a few different methods to predict the average price (linear regression, decision tree, random forest) and then use those predicted values to fill missing on the ",[30,97641,97638],{}," DataFrame.",[11,97644,97645,97646,97650],{},"Here's a link to one of the pages that I scraped with this script: ",[20,97647,97648],{"href":97648,"rel":97649},"https://pcpartpicker.com/product/MYH48d/corsair-memory-cmk16gx4m2b3000c15",[24],". I was able to use this script for each type of part thanks to the consitent stucture and DOM naming scheme.",[459,97652,97654],{"className":13136,"code":97653,"language":12886,"meta":464,"style":464},"import os\nimport pandas as pd\nfrom bs4 import BeautifulSoup\n\n#navigate to the directory containing HTML for PSUs\nos.chdir(\"parts/PSU/parts/\")\npart_list = []\ncomments = []\nfor i in os.listdir(os.getcwd()):\n    a = open(i, 'r')\n    #print a.read()\n    b = BeautifulSoup(a)\n\n    average_rating = b.find('span', attrs={'itemprop': 'ratingValue'})\n    ratings_count = b.find('span', attrs={'itemprop':'ratingCount'})\n    if (average_rating != None) & (ratings_count != None):\n        ratings_dict = {'average_rating':average_rating.text, 'ratings_count':ratings_count.text}\n\n    #part name, kind and link\n    if b.find('h4', attrs={'class':'kind'}) != None:\n        kind = b.find('h4', attrs={'class':'kind'}).text\n        part_name = b.find('h1', attrs={'class':'name'}).text\n        link = b.find('input', attrs={'name':'url'})['value']\n        info_dict = {'Kind':kind, 'Name':part_name, 'Link': link}\n\n        #prices\n        if b.find_all('td', attrs={'class':'base'}) != None:\n            price_list = b.find_all('td', attrs={'class':'base'})\n            price_list = [float(x.text.strip('$')) for x in price_list]\n            #average_price = sum(price_list)/len(price_list)\n            price_dict = {'Prices':price_list,}\n\n        #specs\n        spec_labels = b.find('div', attrs={'class':'specs block'}).find_all('h4')\n        spec_labels = [x.contents[0] for x in spec_labels]\n        spec_values = str(b.find('div', attrs={'class':'specs block'}))\n\n        #this part was a little tricky since the values were placed outside of HTML tags\n        #thankfully I managed to figure out a pattern that would extract everything neatly\n        vals = [x.strip().split('\u003C/h4>')[1].strip('\\n').strip() for x in spec_values.split(\"\u003Ch4>\")[1:]]\n        vals[-1] = vals[-1].split('\\n')[0]\n        spec_values = vals\n\n        spec_values = spec_values[0:len(spec_labels)+1]\n        spec_dict = {spec_label:spec_value for spec_label, spec_value in zip(spec_labels,spec_values)}\n\n        part_dict = dict(spec_dict.items() + info_dict.items() + price_dict.items() + ratings_dict.items())\n        part_list.append(part_dict)\n\n    #ratings\n    if (average_rating != None) & (ratings_count != None):\n        ratings_dict = {'average_rating':average_rating, 'ratings_count':ratings_count}\n    reviews = b.find('div', attrs={'class':'part-reviews'})\n    if reviews != None:\n        reviews = reviews.find_all('div',attrs={'class':'part-review-block'})\n        star_list = [len(reviews[x].find('ul',attrs={'class':'stars'}).find_all('li',attrs={'class':'full-star'})) for x in range(len(reviews))]\n\n        comment_text_list = b.find_all('div', attrs={'class':'comment-message markdown'})\n        comment_text_list = [comment_text_list[x].find_all('p') for x in range(len(comment_text_list))]\n\n        comment_text_list_clean = []\n        for i, x in enumerate(comment_text_list):\n            comment = \"\"\n            for y in x:\n                try:\n                    comment += y.contents[0] + \" \"\n                except:\n                    pass\n            comment_text_list_clean.append(comment)\n\n        review = zip(star_list, comment_text_list_clean)\n        comments.append(review)\n\n    a.close()\n    b.decompose()\n\ndf = pd.DataFrame(part_list)\n",[30,97655,97656,97662,97672,97682,97686,97691,97700,97709,97718,97728,97742,97747,97755,97759,97788,97816,97840,97861,97865,97870,97901,97928,97956,97988,98015,98019,98024,98056,98083,98110,98115,98130,98134,98139,98171,98192,98222,98226,98231,98236,98281,98315,98324,98328,98352,98374,98378,98405,98410,98414,98419,98441,98459,98487,98500,98529,98598,98602,98630,98658,98662,98671,98685,98694,98705,98712,98731,98738,98743,98748,98752,98764,98769,98773,98778,98782,98786],{"__ignoreMap":464},[151,97657,97658,97660],{"class":469,"line":470},[151,97659,16859],{"class":1869},[151,97661,24070],{"class":503},[151,97663,97664,97666,97668,97670],{"class":469,"line":488},[151,97665,16859],{"class":1869},[151,97667,83790],{"class":503},[151,97669,16998],{"class":1869},[151,97671,83795],{"class":503},[151,97673,97674,97676,97678,97680],{"class":469,"line":500},[151,97675,16853],{"class":1869},[151,97677,92070],{"class":503},[151,97679,16859],{"class":1869},[151,97681,92075],{"class":503},[151,97683,97684],{"class":469,"line":509},[151,97685,1090],{"emptyLinePlaceholder":609},[151,97687,97688],{"class":469,"line":517},[151,97689,97690],{"class":1527},"#navigate to the directory containing HTML for PSUs\n",[151,97692,97693,97695,97698],{"class":469,"line":534},[151,97694,92835],{"class":503},[151,97696,97697],{"class":481},"\"parts/PSU/parts/\"",[151,97699,3640],{"class":503},[151,97701,97702,97705,97707],{"class":469,"line":1413},[151,97703,97704],{"class":503},"part_list ",[151,97706,1876],{"class":1869},[151,97708,16606],{"class":503},[151,97710,97711,97714,97716],{"class":469,"line":1418},[151,97712,97713],{"class":503},"comments ",[151,97715,1876],{"class":1869},[151,97717,16606],{"class":503},[151,97719,97720,97722,97724,97726],{"class":469,"line":2462},[151,97721,16732],{"class":1869},[151,97723,67225],{"class":503},[151,97725,16417],{"class":1869},[151,97727,95651],{"class":503},[151,97729,97730,97732,97734,97736,97738,97740],{"class":469,"line":2471},[151,97731,63538],{"class":503},[151,97733,1876],{"class":1869},[151,97735,16970],{"class":2226},[151,97737,95662],{"class":503},[151,97739,44149],{"class":481},[151,97741,3640],{"class":503},[151,97743,97744],{"class":469,"line":2480},[151,97745,97746],{"class":1527},"    #print a.read()\n",[151,97748,97749,97751,97753],{"class":469,"line":2489},[151,97750,92963],{"class":503},[151,97752,1876],{"class":1869},[151,97754,96144],{"class":503},[151,97756,97757],{"class":469,"line":2497},[151,97758,1090],{"emptyLinePlaceholder":609},[151,97760,97761,97764,97766,97768,97770,97772,97774,97776,97778,97781,97783,97786],{"class":469,"line":3140},[151,97762,97763],{"class":503},"    average_rating ",[151,97765,1876],{"class":1869},[151,97767,93041],{"class":503},[151,97769,93044],{"class":481},[151,97771,106],{"class":503},[151,97773,65333],{"class":15210},[151,97775,1876],{"class":1869},[151,97777,5729],{"class":503},[151,97779,97780],{"class":481},"'itemprop'",[151,97782,6208],{"class":503},[151,97784,97785],{"class":481},"'ratingValue'",[151,97787,19610],{"class":503},[151,97789,97790,97793,97795,97797,97799,97801,97803,97805,97807,97809,97811,97814],{"class":469,"line":3149},[151,97791,97792],{"class":503},"    ratings_count ",[151,97794,1876],{"class":1869},[151,97796,93041],{"class":503},[151,97798,93044],{"class":481},[151,97800,106],{"class":503},[151,97802,65333],{"class":15210},[151,97804,1876],{"class":1869},[151,97806,5729],{"class":503},[151,97808,97780],{"class":481},[151,97810,208],{"class":503},[151,97812,97813],{"class":481},"'ratingCount'",[151,97815,19610],{"class":503},[151,97817,97818,97820,97823,97825,97827,97829,97831,97834,97836,97838],{"class":469,"line":3158},[151,97819,23327],{"class":1869},[151,97821,97822],{"class":503}," (average_rating ",[151,97824,58602],{"class":1869},[151,97826,40451],{"class":477},[151,97828,16995],{"class":503},[151,97830,54214],{"class":1869},[151,97832,97833],{"class":503}," (ratings_count ",[151,97835,58602],{"class":1869},[151,97837,40451],{"class":477},[151,97839,15264],{"class":503},[151,97841,97842,97845,97847,97849,97852,97855,97858],{"class":469,"line":3167},[151,97843,97844],{"class":503},"        ratings_dict ",[151,97846,1876],{"class":1869},[151,97848,52023],{"class":503},[151,97850,97851],{"class":481},"'average_rating'",[151,97853,97854],{"class":503},":average_rating.text, ",[151,97856,97857],{"class":481},"'ratings_count'",[151,97859,97860],{"class":503},":ratings_count.text}\n",[151,97862,97863],{"class":469,"line":3175},[151,97864,1090],{"emptyLinePlaceholder":609},[151,97866,97867],{"class":469,"line":3184},[151,97868,97869],{"class":1527},"    #part name, kind and link\n",[151,97871,97872,97874,97876,97878,97880,97882,97884,97886,97888,97890,97893,97895,97897,97899],{"class":469,"line":3193},[151,97873,23327],{"class":1869},[151,97875,93041],{"class":503},[151,97877,97062],{"class":481},[151,97879,106],{"class":503},[151,97881,65333],{"class":15210},[151,97883,1876],{"class":1869},[151,97885,5729],{"class":503},[151,97887,71242],{"class":481},[151,97889,208],{"class":503},[151,97891,97892],{"class":481},"'kind'",[151,97894,97006],{"class":503},[151,97896,58602],{"class":1869},[151,97898,40451],{"class":477},[151,97900,14372],{"class":503},[151,97902,97903,97906,97908,97910,97912,97914,97916,97918,97920,97922,97924,97926],{"class":469,"line":3720},[151,97904,97905],{"class":503},"        kind ",[151,97907,1876],{"class":1869},[151,97909,93041],{"class":503},[151,97911,97062],{"class":481},[151,97913,106],{"class":503},[151,97915,65333],{"class":15210},[151,97917,1876],{"class":1869},[151,97919,5729],{"class":503},[151,97921,71242],{"class":481},[151,97923,208],{"class":503},[151,97925,97892],{"class":481},[151,97927,93252],{"class":503},[151,97929,97930,97933,97935,97937,97939,97941,97943,97945,97947,97949,97951,97954],{"class":469,"line":3729},[151,97931,97932],{"class":503},"        part_name ",[151,97934,1876],{"class":1869},[151,97936,93041],{"class":503},[151,97938,96850],{"class":481},[151,97940,106],{"class":503},[151,97942,65333],{"class":15210},[151,97944,1876],{"class":1869},[151,97946,5729],{"class":503},[151,97948,71242],{"class":481},[151,97950,208],{"class":503},[151,97952,97953],{"class":481},"'name'",[151,97955,93252],{"class":503},[151,97957,97958,97961,97963,97965,97967,97969,97971,97973,97975,97977,97979,97982,97984,97986],{"class":469,"line":3735},[151,97959,97960],{"class":503},"        link ",[151,97962,1876],{"class":1869},[151,97964,93041],{"class":503},[151,97966,97107],{"class":481},[151,97968,106],{"class":503},[151,97970,65333],{"class":15210},[151,97972,1876],{"class":1869},[151,97974,5729],{"class":503},[151,97976,97953],{"class":481},[151,97978,208],{"class":503},[151,97980,97981],{"class":481},"'url'",[151,97983,97125],{"class":503},[151,97985,97128],{"class":481},[151,97987,3691],{"class":503},[151,97989,97990,97993,97995,97997,98000,98003,98006,98009,98012],{"class":469,"line":3745},[151,97991,97992],{"class":503},"        info_dict ",[151,97994,1876],{"class":1869},[151,97996,52023],{"class":503},[151,97998,97999],{"class":481},"'Kind'",[151,98001,98002],{"class":503},":kind, ",[151,98004,98005],{"class":481},"'Name'",[151,98007,98008],{"class":503},":part_name, ",[151,98010,98011],{"class":481},"'Link'",[151,98013,98014],{"class":503},": link}\n",[151,98016,98017],{"class":469,"line":3754},[151,98018,1090],{"emptyLinePlaceholder":609},[151,98020,98021],{"class":469,"line":3760},[151,98022,98023],{"class":1527},"        #prices\n",[151,98025,98026,98028,98031,98033,98035,98037,98039,98041,98043,98045,98048,98050,98052,98054],{"class":469,"line":3773},[151,98027,23357],{"class":1869},[151,98029,98030],{"class":503}," b.find_all(",[151,98032,64868],{"class":481},[151,98034,106],{"class":503},[151,98036,65333],{"class":15210},[151,98038,1876],{"class":1869},[151,98040,5729],{"class":503},[151,98042,71242],{"class":481},[151,98044,208],{"class":503},[151,98046,98047],{"class":481},"'base'",[151,98049,97006],{"class":503},[151,98051,58602],{"class":1869},[151,98053,40451],{"class":477},[151,98055,14372],{"class":503},[151,98057,98058,98061,98063,98065,98067,98069,98071,98073,98075,98077,98079,98081],{"class":469,"line":3782},[151,98059,98060],{"class":503},"            price_list ",[151,98062,1876],{"class":1869},[151,98064,98030],{"class":503},[151,98066,64868],{"class":481},[151,98068,106],{"class":503},[151,98070,65333],{"class":15210},[151,98072,1876],{"class":1869},[151,98074,5729],{"class":503},[151,98076,71242],{"class":481},[151,98078,208],{"class":503},[151,98080,98047],{"class":481},[151,98082,19610],{"class":503},[151,98084,98085,98087,98089,98091,98093,98096,98099,98101,98103,98105,98107],{"class":469,"line":3791},[151,98086,98060],{"class":503},[151,98088,1876],{"class":1869},[151,98090,6604],{"class":503},[151,98092,59805],{"class":6205},[151,98094,98095],{"class":503},"(x.text.strip(",[151,98097,98098],{"class":481},"'$'",[151,98100,34074],{"class":503},[151,98102,16732],{"class":1869},[151,98104,44552],{"class":503},[151,98106,16417],{"class":1869},[151,98108,98109],{"class":503}," price_list]\n",[151,98111,98112],{"class":469,"line":3803},[151,98113,98114],{"class":1527},"            #average_price = sum(price_list)/len(price_list)\n",[151,98116,98117,98120,98122,98124,98127],{"class":469,"line":3811},[151,98118,98119],{"class":503},"            price_dict ",[151,98121,1876],{"class":1869},[151,98123,52023],{"class":503},[151,98125,98126],{"class":481},"'Prices'",[151,98128,98129],{"class":503},":price_list,}\n",[151,98131,98132],{"class":469,"line":3820},[151,98133,1090],{"emptyLinePlaceholder":609},[151,98135,98136],{"class":469,"line":7084},[151,98137,98138],{"class":1527},"        #specs\n",[151,98140,98141,98144,98146,98148,98150,98152,98154,98156,98158,98160,98162,98165,98167,98169],{"class":469,"line":7148},[151,98142,98143],{"class":503},"        spec_labels ",[151,98145,1876],{"class":1869},[151,98147,93041],{"class":503},[151,98149,92151],{"class":481},[151,98151,106],{"class":503},[151,98153,65333],{"class":15210},[151,98155,1876],{"class":1869},[151,98157,5729],{"class":503},[151,98159,71242],{"class":481},[151,98161,208],{"class":503},[151,98163,98164],{"class":481},"'specs block'",[151,98166,96763],{"class":503},[151,98168,97062],{"class":481},[151,98170,3640],{"class":503},[151,98172,98173,98175,98177,98179,98181,98183,98185,98187,98189],{"class":469,"line":7211},[151,98174,98143],{"class":503},[151,98176,1876],{"class":1869},[151,98178,97029],{"class":503},[151,98180,9181],{"class":477},[151,98182,16654],{"class":503},[151,98184,16732],{"class":1869},[151,98186,44552],{"class":503},[151,98188,16417],{"class":1869},[151,98190,98191],{"class":503}," spec_labels]\n",[151,98193,98194,98197,98199,98201,98203,98205,98207,98209,98211,98213,98215,98217,98219],{"class":469,"line":7273},[151,98195,98196],{"class":503},"        spec_values ",[151,98198,1876],{"class":1869},[151,98200,84112],{"class":6205},[151,98202,96661],{"class":503},[151,98204,92151],{"class":481},[151,98206,106],{"class":503},[151,98208,65333],{"class":15210},[151,98210,1876],{"class":1869},[151,98212,5729],{"class":503},[151,98214,71242],{"class":481},[151,98216,208],{"class":503},[151,98218,98164],{"class":481},[151,98220,98221],{"class":503},"}))\n",[151,98223,98224],{"class":469,"line":7335},[151,98225,1090],{"emptyLinePlaceholder":609},[151,98227,98228],{"class":469,"line":7398},[151,98229,98230],{"class":1527},"        #this part was a little tricky since the values were placed outside of HTML tags\n",[151,98232,98233],{"class":469,"line":7462},[151,98234,98235],{"class":1527},"        #thankfully I managed to figure out a pattern that would extract everything neatly\n",[151,98237,98238,98240,98242,98245,98248,98250,98252,98254,98256,98258,98260,98262,98264,98266,98268,98271,98274,98276,98278],{"class":469,"line":7467},[151,98239,96419],{"class":503},[151,98241,1876],{"class":1869},[151,98243,98244],{"class":503}," [x.strip().split(",[151,98246,98247],{"class":481},"'\u003C/h4>'",[151,98249,40832],{"class":503},[151,98251,6760],{"class":477},[151,98253,96705],{"class":503},[151,98255,13223],{"class":481},[151,98257,8043],{"class":477},[151,98259,13223],{"class":481},[151,98261,96978],{"class":503},[151,98263,16732],{"class":1869},[151,98265,44552],{"class":503},[151,98267,16417],{"class":1869},[151,98269,98270],{"class":503}," spec_values.split(",[151,98272,98273],{"class":481},"\"\u003Ch4>\"",[151,98275,40832],{"class":503},[151,98277,6760],{"class":477},[151,98279,98280],{"class":503},":]]\n",[151,98282,98283,98286,98288,98290,98292,98294,98297,98299,98301,98303,98305,98307,98309,98311,98313],{"class":469,"line":7532},[151,98284,98285],{"class":503},"        vals[",[151,98287,12445],{"class":1869},[151,98289,6760],{"class":477},[151,98291,16654],{"class":503},[151,98293,1876],{"class":1869},[151,98295,98296],{"class":503}," vals[",[151,98298,12445],{"class":1869},[151,98300,6760],{"class":477},[151,98302,52005],{"class":503},[151,98304,13223],{"class":481},[151,98306,8043],{"class":477},[151,98308,13223],{"class":481},[151,98310,40832],{"class":503},[151,98312,9181],{"class":477},[151,98314,3691],{"class":503},[151,98316,98317,98319,98321],{"class":469,"line":7537},[151,98318,98196],{"class":503},[151,98320,1876],{"class":1869},[151,98322,98323],{"class":503}," vals\n",[151,98325,98326],{"class":469,"line":7603},[151,98327,1090],{"emptyLinePlaceholder":609},[151,98329,98330,98332,98334,98337,98339,98341,98343,98346,98348,98350],{"class":469,"line":7608},[151,98331,98196],{"class":503},[151,98333,1876],{"class":1869},[151,98335,98336],{"class":503}," spec_values[",[151,98338,9181],{"class":477},[151,98340,208],{"class":503},[151,98342,65875],{"class":2226},[151,98344,98345],{"class":503},"(spec_labels)",[151,98347,22885],{"class":1869},[151,98349,6760],{"class":477},[151,98351,3691],{"class":503},[151,98353,98354,98357,98359,98362,98364,98367,98369,98371],{"class":469,"line":7673},[151,98355,98356],{"class":503},"        spec_dict ",[151,98358,1876],{"class":1869},[151,98360,98361],{"class":503}," {spec_label:spec_value ",[151,98363,16732],{"class":1869},[151,98365,98366],{"class":503}," spec_label, spec_value ",[151,98368,16417],{"class":1869},[151,98370,44908],{"class":2226},[151,98372,98373],{"class":503},"(spec_labels,spec_values)}\n",[151,98375,98376],{"class":469,"line":7678},[151,98377,1090],{"emptyLinePlaceholder":609},[151,98379,98380,98383,98385,98387,98390,98392,98395,98397,98400,98402],{"class":469,"line":7708},[151,98381,98382],{"class":503},"        part_dict ",[151,98384,1876],{"class":1869},[151,98386,51817],{"class":6205},[151,98388,98389],{"class":503},"(spec_dict.items() ",[151,98391,22885],{"class":1869},[151,98393,98394],{"class":503}," info_dict.items() ",[151,98396,22885],{"class":1869},[151,98398,98399],{"class":503}," price_dict.items() ",[151,98401,22885],{"class":1869},[151,98403,98404],{"class":503}," ratings_dict.items())\n",[151,98406,98407],{"class":469,"line":7713},[151,98408,98409],{"class":503},"        part_list.append(part_dict)\n",[151,98411,98412],{"class":469,"line":7746},[151,98413,1090],{"emptyLinePlaceholder":609},[151,98415,98416],{"class":469,"line":7751},[151,98417,98418],{"class":1527},"    #ratings\n",[151,98420,98421,98423,98425,98427,98429,98431,98433,98435,98437,98439],{"class":469,"line":7816},[151,98422,23327],{"class":1869},[151,98424,97822],{"class":503},[151,98426,58602],{"class":1869},[151,98428,40451],{"class":477},[151,98430,16995],{"class":503},[151,98432,54214],{"class":1869},[151,98434,97833],{"class":503},[151,98436,58602],{"class":1869},[151,98438,40451],{"class":477},[151,98440,15264],{"class":503},[151,98442,98443,98445,98447,98449,98451,98454,98456],{"class":469,"line":7821},[151,98444,97844],{"class":503},[151,98446,1876],{"class":1869},[151,98448,52023],{"class":503},[151,98450,97851],{"class":481},[151,98452,98453],{"class":503},":average_rating, ",[151,98455,97857],{"class":481},[151,98457,98458],{"class":503},":ratings_count}\n",[151,98460,98461,98464,98466,98468,98470,98472,98474,98476,98478,98480,98482,98485],{"class":469,"line":7847},[151,98462,98463],{"class":503},"    reviews ",[151,98465,1876],{"class":1869},[151,98467,93041],{"class":503},[151,98469,92151],{"class":481},[151,98471,106],{"class":503},[151,98473,65333],{"class":15210},[151,98475,1876],{"class":1869},[151,98477,5729],{"class":503},[151,98479,71242],{"class":481},[151,98481,208],{"class":503},[151,98483,98484],{"class":481},"'part-reviews'",[151,98486,19610],{"class":503},[151,98488,98489,98491,98494,98496,98498],{"class":469,"line":7852},[151,98490,23327],{"class":1869},[151,98492,98493],{"class":503}," reviews ",[151,98495,58602],{"class":1869},[151,98497,40451],{"class":477},[151,98499,14372],{"class":503},[151,98501,98502,98505,98507,98510,98512,98514,98516,98518,98520,98522,98524,98527],{"class":469,"line":7887},[151,98503,98504],{"class":503},"        reviews ",[151,98506,1876],{"class":1869},[151,98508,98509],{"class":503}," reviews.find_all(",[151,98511,92151],{"class":481},[151,98513,3634],{"class":503},[151,98515,65333],{"class":15210},[151,98517,1876],{"class":1869},[151,98519,5729],{"class":503},[151,98521,71242],{"class":481},[151,98523,208],{"class":503},[151,98525,98526],{"class":481},"'part-review-block'",[151,98528,19610],{"class":503},[151,98530,98531,98534,98536,98538,98540,98543,98545,98547,98549,98551,98553,98555,98557,98560,98562,98565,98567,98569,98571,98573,98575,98577,98580,98583,98585,98587,98589,98591,98593,98595],{"class":469,"line":7892},[151,98532,98533],{"class":503},"        star_list ",[151,98535,1876],{"class":1869},[151,98537,6604],{"class":503},[151,98539,65875],{"class":2226},[151,98541,98542],{"class":503},"(reviews[x].find(",[151,98544,96165],{"class":481},[151,98546,3634],{"class":503},[151,98548,65333],{"class":15210},[151,98550,1876],{"class":1869},[151,98552,5729],{"class":503},[151,98554,71242],{"class":481},[151,98556,208],{"class":503},[151,98558,98559],{"class":481},"'stars'",[151,98561,96763],{"class":503},[151,98563,98564],{"class":481},"'li'",[151,98566,3634],{"class":503},[151,98568,65333],{"class":15210},[151,98570,1876],{"class":1869},[151,98572,5729],{"class":503},[151,98574,71242],{"class":481},[151,98576,208],{"class":503},[151,98578,98579],{"class":481},"'full-star'",[151,98581,98582],{"class":503},"})) ",[151,98584,16732],{"class":1869},[151,98586,44552],{"class":503},[151,98588,16417],{"class":1869},[151,98590,2793],{"class":2226},[151,98592,12386],{"class":503},[151,98594,65875],{"class":2226},[151,98596,98597],{"class":503},"(reviews))]\n",[151,98599,98600],{"class":469,"line":7924},[151,98601,1090],{"emptyLinePlaceholder":609},[151,98603,98604,98607,98609,98611,98613,98615,98617,98619,98621,98623,98625,98628],{"class":469,"line":7929},[151,98605,98606],{"class":503},"        comment_text_list ",[151,98608,1876],{"class":1869},[151,98610,98030],{"class":503},[151,98612,92151],{"class":481},[151,98614,106],{"class":503},[151,98616,65333],{"class":15210},[151,98618,1876],{"class":1869},[151,98620,5729],{"class":503},[151,98622,71242],{"class":481},[151,98624,208],{"class":503},[151,98626,98627],{"class":481},"'comment-message markdown'",[151,98629,19610],{"class":503},[151,98631,98632,98634,98636,98639,98641,98643,98645,98647,98649,98651,98653,98655],{"class":469,"line":7991},[151,98633,98606],{"class":503},[151,98635,1876],{"class":1869},[151,98637,98638],{"class":503}," [comment_text_list[x].find_all(",[151,98640,96186],{"class":481},[151,98642,16995],{"class":503},[151,98644,16732],{"class":1869},[151,98646,44552],{"class":503},[151,98648,16417],{"class":1869},[151,98650,2793],{"class":2226},[151,98652,12386],{"class":503},[151,98654,65875],{"class":2226},[151,98656,98657],{"class":503},"(comment_text_list))]\n",[151,98659,98660],{"class":469,"line":7996},[151,98661,1090],{"emptyLinePlaceholder":609},[151,98663,98664,98667,98669],{"class":469,"line":8078},[151,98665,98666],{"class":503},"        comment_text_list_clean ",[151,98668,1876],{"class":1869},[151,98670,16606],{"class":503},[151,98672,98673,98675,98678,98680,98682],{"class":469,"line":8140},[151,98674,16616],{"class":1869},[151,98676,98677],{"class":503}," i, x ",[151,98679,16417],{"class":1869},[151,98681,17042],{"class":2226},[151,98683,98684],{"class":503},"(comment_text_list):\n",[151,98686,98687,98690,98692],{"class":469,"line":8145},[151,98688,98689],{"class":503},"            comment ",[151,98691,1876],{"class":1869},[151,98693,38981],{"class":481},[151,98695,98696,98698,98700,98702],{"class":469,"line":8259},[151,98697,17050],{"class":1869},[151,98699,85487],{"class":503},[151,98701,16417],{"class":1869},[151,98703,98704],{"class":503}," x:\n",[151,98706,98707,98710],{"class":469,"line":8264},[151,98708,98709],{"class":1869},"                try",[151,98711,14372],{"class":503},[151,98713,98714,98717,98719,98722,98724,98726,98728],{"class":469,"line":8613},[151,98715,98716],{"class":503},"                    comment ",[151,98718,24780],{"class":1869},[151,98720,98721],{"class":503}," y.contents[",[151,98723,9181],{"class":477},[151,98725,16654],{"class":503},[151,98727,22885],{"class":1869},[151,98729,98730],{"class":481}," \" \"\n",[151,98732,98733,98736],{"class":469,"line":8678},[151,98734,98735],{"class":1869},"                except",[151,98737,14372],{"class":503},[151,98739,98740],{"class":469,"line":8742},[151,98741,98742],{"class":1869},"                    pass\n",[151,98744,98745],{"class":469,"line":8806},[151,98746,98747],{"class":503},"            comment_text_list_clean.append(comment)\n",[151,98749,98750],{"class":469,"line":8870},[151,98751,1090],{"emptyLinePlaceholder":609},[151,98753,98754,98757,98759,98761],{"class":469,"line":8875},[151,98755,98756],{"class":503},"        review ",[151,98758,1876],{"class":1869},[151,98760,44908],{"class":2226},[151,98762,98763],{"class":503},"(star_list, comment_text_list_clean)\n",[151,98765,98766],{"class":469,"line":8881},[151,98767,98768],{"class":503},"        comments.append(review)\n",[151,98770,98771],{"class":469,"line":8886},[151,98772,1090],{"emptyLinePlaceholder":609},[151,98774,98775],{"class":469,"line":8892},[151,98776,98777],{"class":503},"    a.close()\n",[151,98779,98780],{"class":469,"line":8963},[151,98781,93555],{"class":503},[151,98783,98784],{"class":469,"line":8969},[151,98785,1090],{"emptyLinePlaceholder":609},[151,98787,98788,98790,98792],{"class":469,"line":15001},[151,98789,70720],{"class":503},[151,98791,1876],{"class":1869},[151,98793,98794],{"class":503}," pd.DataFrame(part_list)\n",[56,98796,98798],{"id":98797},"power-supply-unit-psu","Power Supply Unit (PSU)",[11,98800,98801],{},"Each type of part required a bit of formatting, especially things measured in MHz/GHz and MB/GB, and specs with lots of missing values. Here's a little bit of the cleaning I did for power supply units (PSUs).",[459,98803,98805],{"className":13136,"code":98804,"language":12886,"meta":464,"style":464},"#strip the word Watts from the Wattage specs\ndf['power'] = [float(x.strip('Watts')) if x != '' else 0 for x in df.Wattage]\n#price per watt\ndf['ppw'] = [x/y for x,y in zip(df.avg, df.power)]\n#dictionary to translate efficiency ratings from text to integers\neff_rank_mapping = {u'80+ Bronze':2, u'80+ Gold':4, u'80+':1, u'80+ Titanium':6, u'80+ Platinum':5, u'-':0, u'80+ Silver':3}\n#map efficiency ratings to integers\ndf['eff_rank'] = df['Efficiency Certification'].map(eff_rank_mapping)\n",[30,98806,98807,98812,98856,98861,98891,98896,98982,98987],{"__ignoreMap":464},[151,98808,98809],{"class":469,"line":470},[151,98810,98811],{"class":1527},"#strip the word Watts from the Wattage specs\n",[151,98813,98814,98816,98819,98821,98823,98825,98827,98830,98833,98835,98837,98839,98841,98843,98845,98847,98849,98851,98853],{"class":469,"line":488},[151,98815,70736],{"class":503},[151,98817,98818],{"class":481},"'power'",[151,98820,16654],{"class":503},[151,98822,1876],{"class":1869},[151,98824,6604],{"class":503},[151,98826,59805],{"class":6205},[151,98828,98829],{"class":503},"(x.strip(",[151,98831,98832],{"class":481},"'Watts'",[151,98834,34074],{"class":503},[151,98836,17218],{"class":1869},[151,98838,44552],{"class":503},[151,98840,58602],{"class":1869},[151,98842,58969],{"class":481},[151,98844,17229],{"class":1869},[151,98846,57890],{"class":477},[151,98848,2235],{"class":1869},[151,98850,44552],{"class":503},[151,98852,16417],{"class":1869},[151,98854,98855],{"class":503}," df.Wattage]\n",[151,98857,98858],{"class":469,"line":500},[151,98859,98860],{"class":1527},"#price per watt\n",[151,98862,98863,98865,98868,98870,98872,98874,98876,98879,98881,98884,98886,98888],{"class":469,"line":509},[151,98864,70736],{"class":503},[151,98866,98867],{"class":481},"'ppw'",[151,98869,16654],{"class":503},[151,98871,1876],{"class":1869},[151,98873,87180],{"class":503},[151,98875,19883],{"class":1869},[151,98877,98878],{"class":503},"y ",[151,98880,16732],{"class":1869},[151,98882,98883],{"class":503}," x,y ",[151,98885,16417],{"class":1869},[151,98887,44908],{"class":2226},[151,98889,98890],{"class":503},"(df.avg, df.power)]\n",[151,98892,98893],{"class":469,"line":517},[151,98894,98895],{"class":1527},"#dictionary to translate efficiency ratings from text to integers\n",[151,98897,98898,98901,98903,98905,98907,98910,98912,98914,98916,98918,98921,98923,98925,98927,98929,98932,98934,98936,98938,98940,98943,98945,98947,98949,98951,98954,98956,98958,98960,98962,98965,98967,98969,98971,98973,98976,98978,98980],{"class":469,"line":534},[151,98899,98900],{"class":503},"eff_rank_mapping ",[151,98902,1876],{"class":1869},[151,98904,52023],{"class":503},[151,98906,68688],{"class":12347},[151,98908,98909],{"class":481},"'80+ Bronze'",[151,98911,208],{"class":503},[151,98913,6619],{"class":477},[151,98915,106],{"class":503},[151,98917,68688],{"class":12347},[151,98919,98920],{"class":481},"'80+ Gold'",[151,98922,208],{"class":503},[151,98924,9187],{"class":477},[151,98926,106],{"class":503},[151,98928,68688],{"class":12347},[151,98930,98931],{"class":481},"'80+'",[151,98933,208],{"class":503},[151,98935,6760],{"class":477},[151,98937,106],{"class":503},[151,98939,68688],{"class":12347},[151,98941,98942],{"class":481},"'80+ Titanium'",[151,98944,208],{"class":503},[151,98946,25038],{"class":477},[151,98948,106],{"class":503},[151,98950,68688],{"class":12347},[151,98952,98953],{"class":481},"'80+ Platinum'",[151,98955,208],{"class":503},[151,98957,24380],{"class":477},[151,98959,106],{"class":503},[151,98961,68688],{"class":12347},[151,98963,98964],{"class":481},"'-'",[151,98966,208],{"class":503},[151,98968,9181],{"class":477},[151,98970,106],{"class":503},[151,98972,68688],{"class":12347},[151,98974,98975],{"class":481},"'80+ Silver'",[151,98977,208],{"class":503},[151,98979,6557],{"class":477},[151,98981,6274],{"class":503},[151,98983,98984],{"class":469,"line":1413},[151,98985,98986],{"class":1527},"#map efficiency ratings to integers\n",[151,98988,98989,98991,98994,98996,98998,99000,99003],{"class":469,"line":1418},[151,98990,70736],{"class":503},[151,98992,98993],{"class":481},"'eff_rank'",[151,98995,16654],{"class":503},[151,98997,1876],{"class":1869},[151,98999,70760],{"class":503},[151,99001,99002],{"class":481},"'Efficiency Certification'",[151,99004,99005],{"class":503},"].map(eff_rank_mapping)\n",[11,99007,99008],{},"Let's take a look at some of the data we have so far:",[459,99010,99012],{"className":13136,"code":99011,"language":12886,"meta":464,"style":464},"plt.figure(figsize=(12,8))\nplt.axis([0,1800,0,500])\nplt.title('PSU Wattages vs. PSU Prices', fontsize=16)\nplt.xlabel('Wattage', fontsize=14)\nplt.ylabel('Price', fontsize=14)\n#colors = ['black', 'bronze', 'silver', 'gold', 'green', 'red']\ncolors = ['#000000', '#cd7f32', '#CCCCCC', '#ffd700', '#00ff00', '#ff0000']\n\nef1 = plt.scatter(df[df.eff_rank==1].power, df[df.eff_rank==1].avg, color = colors[0], s=50)\nef2 = plt.scatter(df[df.eff_rank==2].power, df[df.eff_rank==2].avg, color = colors[1], s=50)\nef3 = plt.scatter(df[df.eff_rank==3].power, df[df.eff_rank==3].avg, color = colors[2], s=50)\nef4 = plt.scatter(df[df.eff_rank==4].power, df[df.eff_rank==4].avg, color = colors[3], s=50)\nef5 = plt.scatter(df[df.eff_rank==5].power, df[df.eff_rank==5].avg, color = colors[4], s=50)\nef6 = plt.scatter(df[df.eff_rank==6].power, df[df.eff_rank==6].avg, color = colors[5], s=50)\n\nplt.legend((ef1, ef2, ef3, ef4, ef5, ef6),\n           ('80+', '80+ Bronze', '80+ Silver', '80+ Gold', '80+ Platinum', '80+ Titanium'),\n           title = 'Efficiency Rating',\n           scatterpoints=3,\n           loc='upper left',\n           ncol=2,\n           fontsize=14)\nplt.xticks(fontsize=14)\nplt.yticks(fontsize=14)\nplt.savefig(os.path.expanduser('~/Documents/GitHub/briancaffey.github.io/img/psu/watts_vs_price.png'))\n",[30,99013,99014,99032,99054,99072,99089,99106,99111,99150,99154,99197,99236,99275,99314,99353,99392,99396,99401,99430,99441,99452,99464,99475,99486,99498,99511],{"__ignoreMap":464},[151,99015,99016,99018,99020,99022,99024,99026,99028,99030],{"class":469,"line":470},[151,99017,44355],{"class":503},[151,99019,44358],{"class":15210},[151,99021,1876],{"class":1869},[151,99023,12386],{"class":503},[151,99025,42360],{"class":477},[151,99027,3634],{"class":503},[151,99029,24369],{"class":477},[151,99031,12451],{"class":503},[151,99033,99034,99037,99039,99041,99044,99046,99048,99050,99052],{"class":469,"line":488},[151,99035,99036],{"class":503},"plt.axis([",[151,99038,9181],{"class":477},[151,99040,3634],{"class":503},[151,99042,99043],{"class":477},"1800",[151,99045,3634],{"class":503},[151,99047,9181],{"class":477},[151,99049,3634],{"class":503},[151,99051,12208],{"class":477},[151,99053,38820],{"class":503},[151,99055,99056,99058,99061,99063,99066,99068,99070],{"class":469,"line":500},[151,99057,65123],{"class":503},[151,99059,99060],{"class":481},"'PSU Wattages vs. PSU Prices'",[151,99062,106],{"class":503},[151,99064,99065],{"class":15210},"fontsize",[151,99067,1876],{"class":1869},[151,99069,87061],{"class":477},[151,99071,3640],{"class":503},[151,99073,99074,99076,99079,99081,99083,99085,99087],{"class":469,"line":509},[151,99075,65133],{"class":503},[151,99077,99078],{"class":481},"'Wattage'",[151,99080,106],{"class":503},[151,99082,99065],{"class":15210},[151,99084,1876],{"class":1869},[151,99086,67140],{"class":477},[151,99088,3640],{"class":503},[151,99090,99091,99093,99096,99098,99100,99102,99104],{"class":469,"line":517},[151,99092,65143],{"class":503},[151,99094,99095],{"class":481},"'Price'",[151,99097,106],{"class":503},[151,99099,99065],{"class":15210},[151,99101,1876],{"class":1869},[151,99103,67140],{"class":477},[151,99105,3640],{"class":503},[151,99107,99108],{"class":469,"line":534},[151,99109,99110],{"class":1527},"#colors = ['black', 'bronze', 'silver', 'gold', 'green', 'red']\n",[151,99112,99113,99116,99118,99120,99123,99125,99128,99130,99133,99135,99138,99140,99143,99145,99148],{"class":469,"line":1413},[151,99114,99115],{"class":503},"colors ",[151,99117,1876],{"class":1869},[151,99119,6604],{"class":503},[151,99121,99122],{"class":481},"'#000000'",[151,99124,106],{"class":503},[151,99126,99127],{"class":481},"'#cd7f32'",[151,99129,106],{"class":503},[151,99131,99132],{"class":481},"'#CCCCCC'",[151,99134,106],{"class":503},[151,99136,99137],{"class":481},"'#ffd700'",[151,99139,106],{"class":503},[151,99141,99142],{"class":481},"'#00ff00'",[151,99144,106],{"class":503},[151,99146,99147],{"class":481},"'#ff0000'",[151,99149,3691],{"class":503},[151,99151,99152],{"class":469,"line":1418},[151,99153,1090],{"emptyLinePlaceholder":609},[151,99155,99156,99159,99161,99164,99166,99168,99171,99173,99175,99178,99180,99182,99185,99187,99189,99191,99193,99195],{"class":469,"line":2462},[151,99157,99158],{"class":503},"ef1 ",[151,99160,1876],{"class":1869},[151,99162,99163],{"class":503}," plt.scatter(df[df.eff_rank",[151,99165,17223],{"class":1869},[151,99167,6760],{"class":477},[151,99169,99170],{"class":503},"].power, df[df.eff_rank",[151,99172,17223],{"class":1869},[151,99174,6760],{"class":477},[151,99176,99177],{"class":503},"].avg, ",[151,99179,79362],{"class":15210},[151,99181,19865],{"class":1869},[151,99183,99184],{"class":503}," colors[",[151,99186,9181],{"class":477},[151,99188,60308],{"class":503},[151,99190,55630],{"class":15210},[151,99192,1876],{"class":1869},[151,99194,73146],{"class":477},[151,99196,3640],{"class":503},[151,99198,99199,99202,99204,99206,99208,99210,99212,99214,99216,99218,99220,99222,99224,99226,99228,99230,99232,99234],{"class":469,"line":2471},[151,99200,99201],{"class":503},"ef2 ",[151,99203,1876],{"class":1869},[151,99205,99163],{"class":503},[151,99207,17223],{"class":1869},[151,99209,6619],{"class":477},[151,99211,99170],{"class":503},[151,99213,17223],{"class":1869},[151,99215,6619],{"class":477},[151,99217,99177],{"class":503},[151,99219,79362],{"class":15210},[151,99221,19865],{"class":1869},[151,99223,99184],{"class":503},[151,99225,6760],{"class":477},[151,99227,60308],{"class":503},[151,99229,55630],{"class":15210},[151,99231,1876],{"class":1869},[151,99233,73146],{"class":477},[151,99235,3640],{"class":503},[151,99237,99238,99241,99243,99245,99247,99249,99251,99253,99255,99257,99259,99261,99263,99265,99267,99269,99271,99273],{"class":469,"line":2480},[151,99239,99240],{"class":503},"ef3 ",[151,99242,1876],{"class":1869},[151,99244,99163],{"class":503},[151,99246,17223],{"class":1869},[151,99248,6557],{"class":477},[151,99250,99170],{"class":503},[151,99252,17223],{"class":1869},[151,99254,6557],{"class":477},[151,99256,99177],{"class":503},[151,99258,79362],{"class":15210},[151,99260,19865],{"class":1869},[151,99262,99184],{"class":503},[151,99264,6619],{"class":477},[151,99266,60308],{"class":503},[151,99268,55630],{"class":15210},[151,99270,1876],{"class":1869},[151,99272,73146],{"class":477},[151,99274,3640],{"class":503},[151,99276,99277,99280,99282,99284,99286,99288,99290,99292,99294,99296,99298,99300,99302,99304,99306,99308,99310,99312],{"class":469,"line":2489},[151,99278,99279],{"class":503},"ef4 ",[151,99281,1876],{"class":1869},[151,99283,99163],{"class":503},[151,99285,17223],{"class":1869},[151,99287,9187],{"class":477},[151,99289,99170],{"class":503},[151,99291,17223],{"class":1869},[151,99293,9187],{"class":477},[151,99295,99177],{"class":503},[151,99297,79362],{"class":15210},[151,99299,19865],{"class":1869},[151,99301,99184],{"class":503},[151,99303,6557],{"class":477},[151,99305,60308],{"class":503},[151,99307,55630],{"class":15210},[151,99309,1876],{"class":1869},[151,99311,73146],{"class":477},[151,99313,3640],{"class":503},[151,99315,99316,99319,99321,99323,99325,99327,99329,99331,99333,99335,99337,99339,99341,99343,99345,99347,99349,99351],{"class":469,"line":2497},[151,99317,99318],{"class":503},"ef5 ",[151,99320,1876],{"class":1869},[151,99322,99163],{"class":503},[151,99324,17223],{"class":1869},[151,99326,24380],{"class":477},[151,99328,99170],{"class":503},[151,99330,17223],{"class":1869},[151,99332,24380],{"class":477},[151,99334,99177],{"class":503},[151,99336,79362],{"class":15210},[151,99338,19865],{"class":1869},[151,99340,99184],{"class":503},[151,99342,9187],{"class":477},[151,99344,60308],{"class":503},[151,99346,55630],{"class":15210},[151,99348,1876],{"class":1869},[151,99350,73146],{"class":477},[151,99352,3640],{"class":503},[151,99354,99355,99358,99360,99362,99364,99366,99368,99370,99372,99374,99376,99378,99380,99382,99384,99386,99388,99390],{"class":469,"line":3140},[151,99356,99357],{"class":503},"ef6 ",[151,99359,1876],{"class":1869},[151,99361,99163],{"class":503},[151,99363,17223],{"class":1869},[151,99365,25038],{"class":477},[151,99367,99170],{"class":503},[151,99369,17223],{"class":1869},[151,99371,25038],{"class":477},[151,99373,99177],{"class":503},[151,99375,79362],{"class":15210},[151,99377,19865],{"class":1869},[151,99379,99184],{"class":503},[151,99381,24380],{"class":477},[151,99383,60308],{"class":503},[151,99385,55630],{"class":15210},[151,99387,1876],{"class":1869},[151,99389,73146],{"class":477},[151,99391,3640],{"class":503},[151,99393,99394],{"class":469,"line":3149},[151,99395,1090],{"emptyLinePlaceholder":609},[151,99397,99398],{"class":469,"line":3158},[151,99399,99400],{"class":503},"plt.legend((ef1, ef2, ef3, ef4, ef5, ef6),\n",[151,99402,99403,99406,99408,99410,99412,99414,99416,99418,99420,99422,99424,99426,99428],{"class":469,"line":3167},[151,99404,99405],{"class":503},"           (",[151,99407,98931],{"class":481},[151,99409,106],{"class":503},[151,99411,98909],{"class":481},[151,99413,106],{"class":503},[151,99415,98975],{"class":481},[151,99417,106],{"class":503},[151,99419,98920],{"class":481},[151,99421,106],{"class":503},[151,99423,98953],{"class":481},[151,99425,106],{"class":503},[151,99427,98942],{"class":481},[151,99429,37985],{"class":503},[151,99431,99432,99434,99436,99439],{"class":469,"line":3175},[151,99433,71582],{"class":15210},[151,99435,19865],{"class":1869},[151,99437,99438],{"class":481}," 'Efficiency Rating'",[151,99440,9417],{"class":503},[151,99442,99443,99446,99448,99450],{"class":469,"line":3184},[151,99444,99445],{"class":15210},"           scatterpoints",[151,99447,1876],{"class":1869},[151,99449,6557],{"class":477},[151,99451,9417],{"class":503},[151,99453,99454,99457,99459,99462],{"class":469,"line":3193},[151,99455,99456],{"class":15210},"           loc",[151,99458,1876],{"class":1869},[151,99460,99461],{"class":481},"'upper left'",[151,99463,9417],{"class":503},[151,99465,99466,99469,99471,99473],{"class":469,"line":3720},[151,99467,99468],{"class":15210},"           ncol",[151,99470,1876],{"class":1869},[151,99472,6619],{"class":477},[151,99474,9417],{"class":503},[151,99476,99477,99480,99482,99484],{"class":469,"line":3729},[151,99478,99479],{"class":15210},"           fontsize",[151,99481,1876],{"class":1869},[151,99483,67140],{"class":477},[151,99485,3640],{"class":503},[151,99487,99488,99490,99492,99494,99496],{"class":469,"line":3735},[151,99489,65163],{"class":503},[151,99491,99065],{"class":15210},[151,99493,1876],{"class":1869},[151,99495,67140],{"class":477},[151,99497,3640],{"class":503},[151,99499,99500,99503,99505,99507,99509],{"class":469,"line":3745},[151,99501,99502],{"class":503},"plt.yticks(",[151,99504,99065],{"class":15210},[151,99506,1876],{"class":1869},[151,99508,67140],{"class":477},[151,99510,3640],{"class":503},[151,99512,99513,99515,99518],{"class":469,"line":3754},[151,99514,93826],{"class":503},[151,99516,99517],{"class":481},"'~/Documents/GitHub/briancaffey.github.io/img/psu/watts_vs_price.png'",[151,99519,12451],{"class":503},[11,99521,99522],{},[2718,99523],{"alt":20386,"src":99524},"/static/pcpp/psu/watts_vs_price.png",[11,99526,99527],{},"The data points along the x-axis are PSUs with no price data. There's a pretty nice correlation between watts and price, and Efficiency Rating should help make price predictions even more accurate than using wattage alone. A quick and easy way to determine how influential each feature of our data is on the target variable (price) is to train the data on machine learning algorithm called a random forest.",[11,99529,99530],{},"Before we run the random forest, categorical variable values must be replaced with integer values and we also need to remove PSUs with missing values. Here's a quick way to do that:",[459,99532,99534],{"className":13136,"code":99533,"language":12886,"meta":464,"style":464},"#all of the columns we will be working with\ncols = [u'power', u'eff_rank', u'ppw', u'Manufacturer', u'Modular', u'Type', u'avg']\n#feature columns (doesn't include average price or price per watt)\nfeature_cols = [u'power', u'eff_rank', u'Manufacturer', u'Modular', u'Type']\n#drop null values and seperate X and y\nX = df[cols].dropna()[feature_cols]\ny = df[cols].dropna().avg\n",[30,99535,99536,99541,99596,99601,99640,99645,99654],{"__ignoreMap":464},[151,99537,99538],{"class":469,"line":470},[151,99539,99540],{"class":1527},"#all of the columns we will be working with\n",[151,99542,99543,99546,99548,99550,99552,99554,99556,99558,99560,99562,99564,99566,99568,99570,99573,99575,99577,99580,99582,99584,99587,99589,99591,99594],{"class":469,"line":488},[151,99544,99545],{"class":503},"cols ",[151,99547,1876],{"class":1869},[151,99549,6604],{"class":503},[151,99551,68688],{"class":12347},[151,99553,98818],{"class":481},[151,99555,106],{"class":503},[151,99557,68688],{"class":12347},[151,99559,98993],{"class":481},[151,99561,106],{"class":503},[151,99563,68688],{"class":12347},[151,99565,98867],{"class":481},[151,99567,106],{"class":503},[151,99569,68688],{"class":12347},[151,99571,99572],{"class":481},"'Manufacturer'",[151,99574,106],{"class":503},[151,99576,68688],{"class":12347},[151,99578,99579],{"class":481},"'Modular'",[151,99581,106],{"class":503},[151,99583,68688],{"class":12347},[151,99585,99586],{"class":481},"'Type'",[151,99588,106],{"class":503},[151,99590,68688],{"class":12347},[151,99592,99593],{"class":481},"'avg'",[151,99595,3691],{"class":503},[151,99597,99598],{"class":469,"line":500},[151,99599,99600],{"class":1527},"#feature columns (doesn't include average price or price per watt)\n",[151,99602,99603,99606,99608,99610,99612,99614,99616,99618,99620,99622,99624,99626,99628,99630,99632,99634,99636,99638],{"class":469,"line":509},[151,99604,99605],{"class":503},"feature_cols ",[151,99607,1876],{"class":1869},[151,99609,6604],{"class":503},[151,99611,68688],{"class":12347},[151,99613,98818],{"class":481},[151,99615,106],{"class":503},[151,99617,68688],{"class":12347},[151,99619,98993],{"class":481},[151,99621,106],{"class":503},[151,99623,68688],{"class":12347},[151,99625,99572],{"class":481},[151,99627,106],{"class":503},[151,99629,68688],{"class":12347},[151,99631,99579],{"class":481},[151,99633,106],{"class":503},[151,99635,68688],{"class":12347},[151,99637,99586],{"class":481},[151,99639,3691],{"class":503},[151,99641,99642],{"class":469,"line":517},[151,99643,99644],{"class":1527},"#drop null values and seperate X and y\n",[151,99646,99647,99649,99651],{"class":469,"line":534},[151,99648,87698],{"class":503},[151,99650,1876],{"class":1869},[151,99652,99653],{"class":503}," df[cols].dropna()[feature_cols]\n",[151,99655,99656,99658,99660],{"class":469,"line":1413},[151,99657,98878],{"class":503},[151,99659,1876],{"class":1869},[151,99661,99662],{"class":503}," df[cols].dropna().avg\n",[11,99664,50000,99665,99668,99669,99672],{},[30,99666,99667],{},"print X.shape, y.shape"," returns ",[30,99670,99671],{},"((1100, 5), (1100,))",", so we have 1100 observations of PSUs with complete data. We started with 1434 observations of PSUs, so my goal is to make predictions on PSU prices for the values with missing price data. (There may not be good enough feature data to make these predictions, but we won't worry about that for now). The next step is to map categorical variable values from strings to integers:",[459,99674,99676],{"className":13136,"code":99675,"language":12886,"meta":464,"style":464},"type_mapping = {x:y for x,y in zip(df.Type.unique(),range(len(df.Type.unique())))}\ndf.Type = df.Type.map(type_mapping)\n\nmodular_mapping = {x:y for x,y in zip(df.Modular.unique(),range(len(df.Modular.unique())))}\ndf.Modular = df.Modular.map(modular_mapping)\n\nmanufacturer_mapping = {x:y for x,y in zip(df.Manufacturer.unique(),range(len(df.Manufacturer.unique())))}\ndf.Manufacturer = df.Manufacturer.map(manufacturer_mapping)\n",[30,99677,99678,99708,99718,99722,99751,99761,99765,99794],{"__ignoreMap":464},[151,99679,99680,99683,99685,99687,99689,99691,99693,99695,99698,99701,99703,99705],{"class":469,"line":470},[151,99681,99682],{"class":503},"type_mapping ",[151,99684,1876],{"class":1869},[151,99686,93913],{"class":503},[151,99688,16732],{"class":1869},[151,99690,98883],{"class":503},[151,99692,16417],{"class":1869},[151,99694,44908],{"class":2226},[151,99696,99697],{"class":503},"(df.Type.unique(),",[151,99699,99700],{"class":2226},"range",[151,99702,12386],{"class":503},[151,99704,65875],{"class":2226},[151,99706,99707],{"class":503},"(df.Type.unique())))}\n",[151,99709,99710,99713,99715],{"class":469,"line":488},[151,99711,99712],{"class":503},"df.Type ",[151,99714,1876],{"class":1869},[151,99716,99717],{"class":503}," df.Type.map(type_mapping)\n",[151,99719,99720],{"class":469,"line":500},[151,99721,1090],{"emptyLinePlaceholder":609},[151,99723,99724,99727,99729,99731,99733,99735,99737,99739,99742,99744,99746,99748],{"class":469,"line":509},[151,99725,99726],{"class":503},"modular_mapping ",[151,99728,1876],{"class":1869},[151,99730,93913],{"class":503},[151,99732,16732],{"class":1869},[151,99734,98883],{"class":503},[151,99736,16417],{"class":1869},[151,99738,44908],{"class":2226},[151,99740,99741],{"class":503},"(df.Modular.unique(),",[151,99743,99700],{"class":2226},[151,99745,12386],{"class":503},[151,99747,65875],{"class":2226},[151,99749,99750],{"class":503},"(df.Modular.unique())))}\n",[151,99752,99753,99756,99758],{"class":469,"line":517},[151,99754,99755],{"class":503},"df.Modular ",[151,99757,1876],{"class":1869},[151,99759,99760],{"class":503}," df.Modular.map(modular_mapping)\n",[151,99762,99763],{"class":469,"line":534},[151,99764,1090],{"emptyLinePlaceholder":609},[151,99766,99767,99770,99772,99774,99776,99778,99780,99782,99785,99787,99789,99791],{"class":469,"line":1413},[151,99768,99769],{"class":503},"manufacturer_mapping ",[151,99771,1876],{"class":1869},[151,99773,93913],{"class":503},[151,99775,16732],{"class":1869},[151,99777,98883],{"class":503},[151,99779,16417],{"class":1869},[151,99781,44908],{"class":2226},[151,99783,99784],{"class":503},"(df.Manufacturer.unique(),",[151,99786,99700],{"class":2226},[151,99788,12386],{"class":503},[151,99790,65875],{"class":2226},[151,99792,99793],{"class":503},"(df.Manufacturer.unique())))}\n",[151,99795,99796,99799,99801],{"class":469,"line":1418},[151,99797,99798],{"class":503},"df.Manufacturer ",[151,99800,1876],{"class":1869},[151,99802,99803],{"class":503}," df.Manufacturer.map(manufacturer_mapping)\n",[11,99805,99806,99807,208],{},"Now we are ready to setup a random forest model. Here's the description of a random forest regressor from ",[20,99808,99811],{"href":99809,"rel":99810},"http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html",[24],"scikit-learn.org",[210,99813,99814],{},[11,99815,99816],{},"A random forest is a meta estimator that fits a number of classifying decision trees on various sub-samples of the dataset and use averaging to improve the predictive accuracy and control over-fitting.",[459,99818,99820],{"className":13136,"code":99819,"language":12886,"meta":464,"style":464},"from sklearn.ensemble import RandomForestRegressor\nrfreg = RandomForestRegressor(n_estimators=150, max_features=4, oob_score=True, random_state=3)\nrfreg.fit(X, y)\nfeature_importance = pd.DataFrame({'feature':feature_cols, 'importance':rfreg.feature_importances_}).sort_values(by='importance', ascending=False)\n",[30,99821,99822,99833,99878,99883],{"__ignoreMap":464},[151,99823,99824,99826,99828,99830],{"class":469,"line":470},[151,99825,16853],{"class":1869},[151,99827,89932],{"class":503},[151,99829,16859],{"class":1869},[151,99831,99832],{"class":503}," RandomForestRegressor\n",[151,99834,99835,99838,99840,99843,99846,99848,99850,99852,99855,99857,99859,99861,99864,99866,99868,99870,99872,99874,99876],{"class":469,"line":488},[151,99836,99837],{"class":503},"rfreg ",[151,99839,1876],{"class":1869},[151,99841,99842],{"class":503}," RandomForestRegressor(",[151,99844,99845],{"class":15210},"n_estimators",[151,99847,1876],{"class":1869},[151,99849,45949],{"class":477},[151,99851,106],{"class":503},[151,99853,99854],{"class":15210},"max_features",[151,99856,1876],{"class":1869},[151,99858,9187],{"class":477},[151,99860,106],{"class":503},[151,99862,99863],{"class":15210},"oob_score",[151,99865,1876],{"class":1869},[151,99867,36962],{"class":477},[151,99869,106],{"class":503},[151,99871,71775],{"class":15210},[151,99873,1876],{"class":1869},[151,99875,6557],{"class":477},[151,99877,3640],{"class":503},[151,99879,99880],{"class":469,"line":500},[151,99881,99882],{"class":503},"rfreg.fit(X, y)\n",[151,99884,99885,99888,99890,99893,99896,99899,99902,99905,99907,99909,99911,99913,99915,99917,99919],{"class":469,"line":509},[151,99886,99887],{"class":503},"feature_importance ",[151,99889,1876],{"class":1869},[151,99891,99892],{"class":503}," pd.DataFrame({",[151,99894,99895],{"class":481},"'feature'",[151,99897,99898],{"class":503},":feature_cols, ",[151,99900,99901],{"class":481},"'importance'",[151,99903,99904],{"class":503},":rfreg.feature_importances_}).sort_values(",[151,99906,65808],{"class":15210},[151,99908,1876],{"class":1869},[151,99910,99901],{"class":481},[151,99912,106],{"class":503},[151,99914,65817],{"class":15210},[151,99916,1876],{"class":1869},[151,99918,39461],{"class":477},[151,99920,3640],{"class":503},[11,99922,19225,99923,99926],{},[30,99924,99925],{},"feature_importance"," dataframe assigns a percentage representing how important each feature is in predicting the target variable, price.",[1131,99928,99929,99939],{},[1134,99930,99931],{},[1137,99932,99933,99936],{},[1140,99934,99935],{},"feature",[1140,99937,99938],{},"importance",[1153,99940,99941,99949,99957,99965,99973],{},[1137,99942,99943,99946],{},[1158,99944,99945],{},"power",[1158,99947,99948],{},"0.674090",[1137,99950,99951,99954],{},[1158,99952,99953],{},"eff_rank",[1158,99955,99956],{},"0.170238",[1137,99958,99959,99962],{},[1158,99960,99961],{},"Manufacturer",[1158,99963,99964],{},"0.101809",[1137,99966,99967,99970],{},[1158,99968,99969],{},"Modular",[1158,99971,99972],{},"0.033512",[1137,99974,99975,99978],{},[1158,99976,99977],{},"Type",[1158,99979,99980],{},"0.020352",[11,99982,99983,99984,99986,99987,99989],{},"Looking at feature importance is a quick way evaluate the relative strength of each feature in a model. The results here aren't surprising: ",[30,99985,99945],{}," is by far the most important feature, but ",[30,99988,99953],{}," also has significant pull on the target variable (PSU price). The manufacturer, whether or not the PSU is modular and the type (form factor) are less important and could be ignored altogether in the next model.",[11,99991,99992,99993,99995,99996,99998],{},"We can dig a little bit deeper by searching for a ",[30,99994,99845],{}," value that will minimize RMSE. ",[30,99997,99845],{}," represents the number of decision trees used in the random forest regressor.",[459,100000,100002],{"className":13136,"code":100001,"language":12886,"meta":464,"style":464},"from sklearn.cross_validation import cross_val_score\n\n#list of values to try for n_estimators\nestimator_range = range(10, 310, 10)\n\n#list to store the average RMSE for each value of n_estimators\nRMSE_scores = []\n\n#use 5-fold cross-validation with each value of n_estimators\nfor estimator in estimator_range:\n    rfreg = RandomForestRegressor(n_estimators=estimator, random_state=1)\n    MSE_scores = cross_val_score(rfreg, X, y, cv=5, scoring='mean_squared_error')\n    RMSE_scores.append(np.mean(np.sqrt(-MSE_scores)))\n\n# plot n_estimators (x-axis) versus RMSE (y-axis)\nplt.plot(estimator_range, RMSE_scores)\nplt.xlabel('n_estimators')\nplt.ylabel('RMSE (lower is better)')\n",[30,100003,100004,100016,100020,100025,100048,100052,100057,100066,100070,100075,100087,100111,100140,100156,100160,100165,100174,100183],{"__ignoreMap":464},[151,100005,100006,100008,100011,100013],{"class":469,"line":470},[151,100007,16853],{"class":1869},[151,100009,100010],{"class":503}," sklearn.cross_validation ",[151,100012,16859],{"class":1869},[151,100014,100015],{"class":503}," cross_val_score\n",[151,100017,100018],{"class":469,"line":488},[151,100019,1090],{"emptyLinePlaceholder":609},[151,100021,100022],{"class":469,"line":500},[151,100023,100024],{"class":1527},"#list of values to try for n_estimators\n",[151,100026,100027,100030,100032,100034,100036,100038,100040,100042,100044,100046],{"class":469,"line":509},[151,100028,100029],{"class":503},"estimator_range ",[151,100031,1876],{"class":1869},[151,100033,2793],{"class":2226},[151,100035,12386],{"class":503},[151,100037,12423],{"class":477},[151,100039,106],{"class":503},[151,100041,73527],{"class":477},[151,100043,106],{"class":503},[151,100045,12423],{"class":477},[151,100047,3640],{"class":503},[151,100049,100050],{"class":469,"line":517},[151,100051,1090],{"emptyLinePlaceholder":609},[151,100053,100054],{"class":469,"line":534},[151,100055,100056],{"class":1527},"#list to store the average RMSE for each value of n_estimators\n",[151,100058,100059,100062,100064],{"class":469,"line":1413},[151,100060,100061],{"class":477},"RMSE_scores",[151,100063,19865],{"class":1869},[151,100065,16606],{"class":503},[151,100067,100068],{"class":469,"line":1418},[151,100069,1090],{"emptyLinePlaceholder":609},[151,100071,100072],{"class":469,"line":2462},[151,100073,100074],{"class":1527},"#use 5-fold cross-validation with each value of n_estimators\n",[151,100076,100077,100079,100082,100084],{"class":469,"line":2471},[151,100078,16732],{"class":1869},[151,100080,100081],{"class":503}," estimator ",[151,100083,16417],{"class":1869},[151,100085,100086],{"class":503}," estimator_range:\n",[151,100088,100089,100092,100094,100096,100098,100100,100103,100105,100107,100109],{"class":469,"line":2480},[151,100090,100091],{"class":503},"    rfreg ",[151,100093,1876],{"class":1869},[151,100095,99842],{"class":503},[151,100097,99845],{"class":15210},[151,100099,1876],{"class":1869},[151,100101,100102],{"class":503},"estimator, ",[151,100104,71775],{"class":15210},[151,100106,1876],{"class":1869},[151,100108,6760],{"class":477},[151,100110,3640],{"class":503},[151,100112,100113,100116,100118,100121,100124,100126,100128,100130,100133,100135,100138],{"class":469,"line":2489},[151,100114,100115],{"class":477},"    MSE_scores",[151,100117,19865],{"class":1869},[151,100119,100120],{"class":503}," cross_val_score(rfreg, X, y, ",[151,100122,100123],{"class":15210},"cv",[151,100125,1876],{"class":1869},[151,100127,24380],{"class":477},[151,100129,106],{"class":503},[151,100131,100132],{"class":15210},"scoring",[151,100134,1876],{"class":1869},[151,100136,100137],{"class":481},"'mean_squared_error'",[151,100139,3640],{"class":503},[151,100141,100142,100145,100148,100150,100153],{"class":469,"line":2497},[151,100143,100144],{"class":477},"    RMSE_scores",[151,100146,100147],{"class":503},".append(np.mean(np.sqrt(",[151,100149,12445],{"class":1869},[151,100151,100152],{"class":477},"MSE_scores",[151,100154,100155],{"class":503},")))\n",[151,100157,100158],{"class":469,"line":3140},[151,100159,1090],{"emptyLinePlaceholder":609},[151,100161,100162],{"class":469,"line":3149},[151,100163,100164],{"class":1527},"# plot n_estimators (x-axis) versus RMSE (y-axis)\n",[151,100166,100167,100170,100172],{"class":469,"line":3158},[151,100168,100169],{"class":503},"plt.plot(estimator_range, ",[151,100171,100061],{"class":477},[151,100173,3640],{"class":503},[151,100175,100176,100178,100181],{"class":469,"line":3167},[151,100177,65133],{"class":503},[151,100179,100180],{"class":481},"'n_estimators'",[151,100182,3640],{"class":503},[151,100184,100185,100187,100190],{"class":469,"line":3175},[151,100186,65143],{"class":503},[151,100188,100189],{"class":481},"'RMSE (lower is better)'",[151,100191,3640],{"class":503},[11,100193,100194],{},[2718,100195],{"alt":20386,"src":100196},"/static/pcpp/psu/n_est_vs_rmse.png",[11,100198,100199,100200,100202,100203,100205,100206,100209,100210,100213,100214,100216],{},"This graph shows the error in the model for 30 different settings of the parameter ",[30,100201,99845],{},". However, each time we test a new ",[30,100204,99845],{}," value we are caluculating the error using cross-validation. Cross-validataion, or K-fold cross-validation helps improve the accuracy of error estimation by averaging the results of ",[30,100207,100208],{},"k"," models. In the model above, we see that ",[30,100211,100212],{},"cv=5",", so we are running the model 5 times for every value of ",[30,100215,99845],{},", for a total of 150 times.",[11,100218,100219,100220,100222],{},"To find the optimal value of ",[30,100221,99845],{}," we search for the value with the lowest error:",[459,100224,100226],{"className":13136,"code":100225,"language":12886,"meta":464,"style":464},"sorted(zip(RMSE_scores, estimator_range))[0]\n",[30,100227,100228],{"__ignoreMap":464},[151,100229,100230,100232,100234,100237,100239,100241,100244,100246],{"class":469,"line":470},[151,100231,43956],{"class":2226},[151,100233,12386],{"class":503},[151,100235,100236],{"class":2226},"zip",[151,100238,12386],{"class":503},[151,100240,100061],{"class":477},[151,100242,100243],{"class":503},", estimator_range))[",[151,100245,9181],{"class":477},[151,100247,3691],{"class":503},[11,100249,100250],{},[30,100251,100252],{},"(28.100834969072217, 160)",[11,100254,100255,100256,100258,100259,100261],{},"So, we get a slightly lower root mean squared error of 28.1 when we choose an ",[30,100257,99845],{}," value of 160 (we started with ",[30,100260,99845],{}," equal to 150).",[11,100263,100264],{},"We can do a quick test of the model by comparing the model's price predictions for certain PSUs to prices on Amazon.",[11,100266,100267],{},"Notice that there is a red point on the x-axis just beyond 1200 Watts. Let's predict the price for that PSU:",[459,100269,100271],{"className":13136,"code":100270,"language":12886,"meta":464,"style":464},"#filter the dataframe by 80+ Titanium rated PSUs that have a price of 0 and power over 1200 W\nX_ = np.array(df[(df['Efficiency Certification']=='80+ Titanium')&(df.power>1200)&(df.avg==0)][feature_cols])\n",[30,100272,100273,100278],{"__ignoreMap":464},[151,100274,100275],{"class":469,"line":470},[151,100276,100277],{"class":1527},"#filter the dataframe by 80+ Titanium rated PSUs that have a price of 0 and power over 1200 W\n",[151,100279,100280,100282,100284,100287,100289,100291,100293,100295,100297,100299,100302,100304,100307,100309,100311,100314,100316,100318],{"class":469,"line":488},[151,100281,89947],{"class":503},[151,100283,1876],{"class":1869},[151,100285,100286],{"class":503}," np.array(df[(df[",[151,100288,99002],{"class":481},[151,100290,8582],{"class":503},[151,100292,17223],{"class":1869},[151,100294,98942],{"class":481},[151,100296,748],{"class":503},[151,100298,54214],{"class":1869},[151,100300,100301],{"class":503},"(df.power",[151,100303,3663],{"class":1869},[151,100305,100306],{"class":477},"1200",[151,100308,748],{"class":503},[151,100310,54214],{"class":1869},[151,100312,100313],{"class":503},"(df.avg",[151,100315,17223],{"class":1869},[151,100317,9181],{"class":477},[151,100319,100320],{"class":503},")][feature_cols])\n",[11,100322,100323],{},"Let's get the index for this PSU:",[459,100325,100327],{"className":13136,"code":100326,"language":12886,"meta":464,"style":464},"df[(df['Efficiency Certification']=='80+ Titanium')&(df.power>1200)&(df.avg==0)].index\n",[30,100328,100329],{"__ignoreMap":464},[151,100330,100331,100334,100336,100338,100340,100342,100344,100346,100348,100350,100352,100354,100356,100358,100360,100362],{"class":469,"line":470},[151,100332,100333],{"class":503},"df[(df[",[151,100335,99002],{"class":481},[151,100337,8582],{"class":503},[151,100339,17223],{"class":1869},[151,100341,98942],{"class":481},[151,100343,748],{"class":503},[151,100345,54214],{"class":1869},[151,100347,100301],{"class":503},[151,100349,3663],{"class":1869},[151,100351,100306],{"class":477},[151,100353,748],{"class":503},[151,100355,54214],{"class":1869},[151,100357,100313],{"class":503},[151,100359,17223],{"class":1869},[151,100361,9181],{"class":477},[151,100363,100364],{"class":503},")].index\n",[11,100366,100367],{},[30,100368,100369],{},"Int64Index([1260], dtype='int64')",[11,100371,100372],{},"Now we can make a prediction for price of this PSU:",[459,100374,100376],{"className":13136,"code":100375,"language":12886,"meta":464,"style":464},"rfreg.predict(df.ix[1260][feature_cols])\n",[30,100377,100378],{"__ignoreMap":464},[151,100379,100380,100383,100386],{"class":469,"line":470},[151,100381,100382],{"class":503},"rfreg.predict(df.ix[",[151,100384,100385],{"class":477},"1260",[151,100387,100388],{"class":503},"][feature_cols])\n",[11,100390,100391,100392,187,100397,100402],{},"And the prediction we get for this model is $278.09. This product is listed ",[20,100393,100396],{"href":100394,"rel":100395},"https://www.amazon.com/Thermaltake-ToughPower-TITANIUM-256-colors-Management/dp/B019JKM20W",[24],"on Amazon",[20,100398,100401],{"href":100399,"rel":100400},"https://www.newegg.com/Product/Product.aspx?Item=N82E16817153270",[24],"NewEgg"," for $349.99, which means that our prediction fell short of the actual price by quite a bit (by $71.90).",[11,100404,100405],{},"A more practical approach for modeling PSU prices might be to simply make individual linear regressions for each Efficiency Certification. Here's a prediction for the same PSU using a linear regression of only a handful of 80+ Titanium rated PSUs:",[459,100407,100409],{"className":13136,"code":100408,"language":12886,"meta":464,"style":464},"df4 = df[(df.avg>0)&(df['Efficiency Certification']=='80+ Titanium')]\nX = df4[['power']]\ny = df4[['avg']]\n\nfrom sklearn.linear_model import LinearRegression\nreg = LinearRegression()\nreg.fit(X,y)\nprint reg.predict([1250])\n",[30,100410,100411,100442,100455,100467,100471,100483,100493,100498],{"__ignoreMap":464},[151,100412,100413,100416,100418,100421,100423,100425,100427,100429,100432,100434,100436,100438,100440],{"class":469,"line":470},[151,100414,100415],{"class":503},"df4 ",[151,100417,1876],{"class":1869},[151,100419,100420],{"class":503}," df[(df.avg",[151,100422,3663],{"class":1869},[151,100424,9181],{"class":477},[151,100426,748],{"class":503},[151,100428,54214],{"class":1869},[151,100430,100431],{"class":503},"(df[",[151,100433,99002],{"class":481},[151,100435,8582],{"class":503},[151,100437,17223],{"class":1869},[151,100439,98942],{"class":481},[151,100441,44576],{"class":503},[151,100443,100444,100446,100448,100451,100453],{"class":469,"line":488},[151,100445,87698],{"class":503},[151,100447,1876],{"class":1869},[151,100449,100450],{"class":503}," df4[[",[151,100452,98818],{"class":481},[151,100454,87779],{"class":503},[151,100456,100457,100459,100461,100463,100465],{"class":469,"line":500},[151,100458,98878],{"class":503},[151,100460,1876],{"class":1869},[151,100462,100450],{"class":503},[151,100464,99593],{"class":481},[151,100466,87779],{"class":503},[151,100468,100469],{"class":469,"line":509},[151,100470,1090],{"emptyLinePlaceholder":609},[151,100472,100473,100475,100478,100480],{"class":469,"line":517},[151,100474,16853],{"class":1869},[151,100476,100477],{"class":503}," sklearn.linear_model ",[151,100479,16859],{"class":1869},[151,100481,100482],{"class":503}," LinearRegression\n",[151,100484,100485,100488,100490],{"class":469,"line":534},[151,100486,100487],{"class":503},"reg ",[151,100489,1876],{"class":1869},[151,100491,100492],{"class":503}," LinearRegression()\n",[151,100494,100495],{"class":469,"line":1413},[151,100496,100497],{"class":503},"reg.fit(X,y)\n",[151,100499,100500,100502,100505,100508],{"class":469,"line":1418},[151,100501,18513],{"class":2226},[151,100503,100504],{"class":503}," reg.predict([",[151,100506,100507],{"class":477},"1250",[151,100509,38820],{"class":503},[11,100511,100512],{},[30,100513,100514],{},"[[ 332.5585859]]",[11,100516,100517],{},"This prediction is much more accurate, and it falls right in line with a line-of-best-fit for the red points on the scatter plot above.",[11,100519,100520],{},"By visualizing power and efficiency rating vs. price in the graph above, we can see the strong correlation between wattage and price, and we can observe some general trends between different efficiency certifications: for any given power rating 80+ Titanium is generally more expensive than 80+ Platinum, and 80+ Platinum is generally more expensive than 80+ Bronze. 80+ Gold PSU prices seem to range quite a bit, so let's look at the distribution of 80+ Gold prices by manufacturer and form factor:",[459,100522,100524],{"className":13136,"code":100523,"language":12886,"meta":464,"style":464},"sns.plt.figure(figsize=(12,8))\nplt.title('Average Price per Watt for 80+ Gold PSUs by Manufacturer', fontsize=14)\nplt.xlabel('Manufacturer', fontsize=14)\nplt.ylabel('Average Price per Watt', fontsize=14)\ndf[(df['Efficiency Certification']=='80+ Gold')&(df.power>0)&(df.avg>0)].groupby('Manufacturer').ppw.mean().sort_values().plot(kind='bar')\nplt.savefig(os.path.expanduser('~/Documents/GitHub/briancaffey.github.io/img/psu/average_price_by_manufacturer.png'))\n",[30,100525,100526,100545,100562,100578,100595,100645],{"__ignoreMap":464},[151,100527,100528,100531,100533,100535,100537,100539,100541,100543],{"class":469,"line":470},[151,100529,100530],{"class":503},"sns.plt.figure(",[151,100532,44358],{"class":15210},[151,100534,1876],{"class":1869},[151,100536,12386],{"class":503},[151,100538,42360],{"class":477},[151,100540,3634],{"class":503},[151,100542,24369],{"class":477},[151,100544,12451],{"class":503},[151,100546,100547,100549,100552,100554,100556,100558,100560],{"class":469,"line":488},[151,100548,65123],{"class":503},[151,100550,100551],{"class":481},"'Average Price per Watt for 80+ Gold PSUs by Manufacturer'",[151,100553,106],{"class":503},[151,100555,99065],{"class":15210},[151,100557,1876],{"class":1869},[151,100559,67140],{"class":477},[151,100561,3640],{"class":503},[151,100563,100564,100566,100568,100570,100572,100574,100576],{"class":469,"line":500},[151,100565,65133],{"class":503},[151,100567,99572],{"class":481},[151,100569,106],{"class":503},[151,100571,99065],{"class":15210},[151,100573,1876],{"class":1869},[151,100575,67140],{"class":477},[151,100577,3640],{"class":503},[151,100579,100580,100582,100585,100587,100589,100591,100593],{"class":469,"line":509},[151,100581,65143],{"class":503},[151,100583,100584],{"class":481},"'Average Price per Watt'",[151,100586,106],{"class":503},[151,100588,99065],{"class":15210},[151,100590,1876],{"class":1869},[151,100592,67140],{"class":477},[151,100594,3640],{"class":503},[151,100596,100597,100599,100601,100603,100605,100607,100609,100611,100613,100615,100617,100619,100621,100623,100625,100627,100630,100632,100635,100638,100640,100643],{"class":469,"line":517},[151,100598,100333],{"class":503},[151,100600,99002],{"class":481},[151,100602,8582],{"class":503},[151,100604,17223],{"class":1869},[151,100606,98920],{"class":481},[151,100608,748],{"class":503},[151,100610,54214],{"class":1869},[151,100612,100301],{"class":503},[151,100614,3663],{"class":1869},[151,100616,9181],{"class":477},[151,100618,748],{"class":503},[151,100620,54214],{"class":1869},[151,100622,100313],{"class":503},[151,100624,3663],{"class":1869},[151,100626,9181],{"class":477},[151,100628,100629],{"class":503},")].groupby(",[151,100631,99572],{"class":481},[151,100633,100634],{"class":503},").ppw.mean().sort_values().plot(",[151,100636,100637],{"class":15210},"kind",[151,100639,1876],{"class":1869},[151,100641,100642],{"class":481},"'bar'",[151,100644,3640],{"class":503},[151,100646,100647,100649,100652],{"class":469,"line":534},[151,100648,93826],{"class":503},[151,100650,100651],{"class":481},"'~/Documents/GitHub/briancaffey.github.io/img/psu/average_price_by_manufacturer.png'",[151,100653,12451],{"class":503},[11,100655,100656],{},[2718,100657],{"alt":20386,"src":100658},"/static/pcpp/psu/average_price_by_manufacturer.png",[11,100660,100661],{},"The most expensive 80+ Gold PSUs are more than twice as expensive as the cheapest PSUs by price per watt. This could explain the significance that the random forest regressor attached to this feature.",[11,100663,100664],{},"Whether or not a PSU is modular refers to the connectivity of power cables that come out of the PSUs. Fully modular means you can unplug all of the cables from the back and plug in only what you need for your PC. Semi modular means that there is one cable you can't unplug from the back (it is usually a 24-pin connector that plugs in to the motherboard), and other cables can be plugged in for graphics cards or other devices. 'No' means that all of the cables that you will need are permanently fixed to the PSU and you can't unplug anything. Here's a graph showing showing the distributions of price per watt by modular type (full, semi and none):",[459,100666,100668],{"className":13136,"code":100667,"language":12886,"meta":464,"style":464},"df[(df['Efficiency Certification']=='80+ Gold')&(df.avg>0)].boxplot(column='ppw', by='Modular', figsize=(12,8))\nplt.ylim([0.07,0.3])\nplt.suptitle('')\nplt.ylabel('Price per Watt', fontsize=14)\nplt.title('Price per Watt distributions of 80+ Gold PSUs by Modular Type', fontsize=14)\nplt.xticks(fontsize=13)\nplt.yticks(fontsize=13)\nplt.savefig(os.path.expanduser('~/Documents/GitHub/briancaffey.github.io/img/psu/price_by_modular.png'))\n",[30,100669,100670,100726,100741,100750,100767,100784,100796,100808],{"__ignoreMap":464},[151,100671,100672,100674,100676,100678,100680,100682,100684,100686,100688,100690,100692,100695,100698,100700,100702,100704,100706,100708,100710,100712,100714,100716,100718,100720,100722,100724],{"class":469,"line":470},[151,100673,100333],{"class":503},[151,100675,99002],{"class":481},[151,100677,8582],{"class":503},[151,100679,17223],{"class":1869},[151,100681,98920],{"class":481},[151,100683,748],{"class":503},[151,100685,54214],{"class":1869},[151,100687,100313],{"class":503},[151,100689,3663],{"class":1869},[151,100691,9181],{"class":477},[151,100693,100694],{"class":503},")].boxplot(",[151,100696,100697],{"class":15210},"column",[151,100699,1876],{"class":1869},[151,100701,98867],{"class":481},[151,100703,106],{"class":503},[151,100705,65808],{"class":15210},[151,100707,1876],{"class":1869},[151,100709,99579],{"class":481},[151,100711,106],{"class":503},[151,100713,44358],{"class":15210},[151,100715,1876],{"class":1869},[151,100717,12386],{"class":503},[151,100719,42360],{"class":477},[151,100721,3634],{"class":503},[151,100723,24369],{"class":477},[151,100725,12451],{"class":503},[151,100727,100728,100731,100734,100736,100739],{"class":469,"line":488},[151,100729,100730],{"class":503},"plt.ylim([",[151,100732,100733],{"class":477},"0.07",[151,100735,3634],{"class":503},[151,100737,100738],{"class":477},"0.3",[151,100740,38820],{"class":503},[151,100742,100743,100746,100748],{"class":469,"line":500},[151,100744,100745],{"class":503},"plt.suptitle(",[151,100747,2301],{"class":481},[151,100749,3640],{"class":503},[151,100751,100752,100754,100757,100759,100761,100763,100765],{"class":469,"line":509},[151,100753,65143],{"class":503},[151,100755,100756],{"class":481},"'Price per Watt'",[151,100758,106],{"class":503},[151,100760,99065],{"class":15210},[151,100762,1876],{"class":1869},[151,100764,67140],{"class":477},[151,100766,3640],{"class":503},[151,100768,100769,100771,100774,100776,100778,100780,100782],{"class":469,"line":517},[151,100770,65123],{"class":503},[151,100772,100773],{"class":481},"'Price per Watt distributions of 80+ Gold PSUs by Modular Type'",[151,100775,106],{"class":503},[151,100777,99065],{"class":15210},[151,100779,1876],{"class":1869},[151,100781,67140],{"class":477},[151,100783,3640],{"class":503},[151,100785,100786,100788,100790,100792,100794],{"class":469,"line":534},[151,100787,65163],{"class":503},[151,100789,99065],{"class":15210},[151,100791,1876],{"class":1869},[151,100793,42327],{"class":477},[151,100795,3640],{"class":503},[151,100797,100798,100800,100802,100804,100806],{"class":469,"line":1413},[151,100799,99502],{"class":503},[151,100801,99065],{"class":15210},[151,100803,1876],{"class":1869},[151,100805,42327],{"class":477},[151,100807,3640],{"class":503},[151,100809,100810,100812,100815],{"class":469,"line":1418},[151,100811,93826],{"class":503},[151,100813,100814],{"class":481},"'~/Documents/GitHub/briancaffey.github.io/img/psu/price_by_modular.png'",[151,100816,12451],{"class":503},[11,100818,100819],{},[2718,100820],{"alt":20386,"src":100821},"/static/pcpp/psu/price_by_modular.png",[11,100823,100824],{},"The distributions aren't surprising: fully-modular is more expensive than semi-modular, and semi-modular PSUs are more expensive than PSUs that are not modular. However, there is quite a bit of overlap in the prices of each modular type, so this feature only contributes roughly 3% of importance for predicting the price of PSUs with the random forest regressor model.",[56,100826,97475],{"id":97405},[11,100828,100829],{},"Motherboards are a critical part of any PC build. This component determines the compatability for most of the other components in a PC build, such as memory type, CPU socket, SSD drives types and SLI/CrossFire configurations (we'll get to what these mean soon).",[11,100831,100832,100833,100836,100837,100840],{},"There are a lot of features to choose from, and most of the features in ",[30,100834,100835],{},"motherboard_csv.csv"," are categorical variables, some containing over 100 unique values. We can use ",[30,100838,100839],{},"df.groupby()"," to look popular combinations of motherboard features. There are 26 CPU socket types, 11 different form factors and 14 memory slot types. There are a total of 60 unique combinations for these three motherboard features, the combinations with the most motherboards are shown below along with average prices:",[459,100842,100844],{"className":13136,"code":100843,"language":12886,"meta":464,"style":464},"df[(df.avg>0)].groupby(['Memory Slots','Form Factor', 'socket']).avg.agg(['mean', 'count']).sort_values(by='count', ascending=False)[:30].plot(kind='bar', figsize=(10,4))\nplt.title('Prices and counts for top 30 Memory Slot, Form Factor and Socket combinations')\nplt.xlabel('Memory Slot, Form Factor and Socket combinations')\nplt.figure()\n",[30,100845,100846,100928,100937,100946],{"__ignoreMap":464},[151,100847,100848,100851,100853,100855,100858,100861,100863,100866,100868,100871,100874,100877,100879,100882,100885,100887,100889,100891,100893,100895,100897,100899,100901,100903,100906,100908,100910,100912,100914,100916,100918,100920,100922,100924,100926],{"class":469,"line":470},[151,100849,100850],{"class":503},"df[(df.avg",[151,100852,3663],{"class":1869},[151,100854,9181],{"class":477},[151,100856,100857],{"class":503},")].groupby([",[151,100859,100860],{"class":481},"'Memory Slots'",[151,100862,3634],{"class":503},[151,100864,100865],{"class":481},"'Form Factor'",[151,100867,106],{"class":503},[151,100869,100870],{"class":481},"'socket'",[151,100872,100873],{"class":503},"]).avg.agg([",[151,100875,100876],{"class":481},"'mean'",[151,100878,106],{"class":503},[151,100880,100881],{"class":481},"'count'",[151,100883,100884],{"class":503},"]).sort_values(",[151,100886,65808],{"class":15210},[151,100888,1876],{"class":1869},[151,100890,100881],{"class":481},[151,100892,106],{"class":503},[151,100894,65817],{"class":15210},[151,100896,1876],{"class":1869},[151,100898,39461],{"class":477},[151,100900,94822],{"class":503},[151,100902,42017],{"class":477},[151,100904,100905],{"class":503},"].plot(",[151,100907,100637],{"class":15210},[151,100909,1876],{"class":1869},[151,100911,100642],{"class":481},[151,100913,106],{"class":503},[151,100915,44358],{"class":15210},[151,100917,1876],{"class":1869},[151,100919,12386],{"class":503},[151,100921,12423],{"class":477},[151,100923,3634],{"class":503},[151,100925,9187],{"class":477},[151,100927,12451],{"class":503},[151,100929,100930,100932,100935],{"class":469,"line":488},[151,100931,65123],{"class":503},[151,100933,100934],{"class":481},"'Prices and counts for top 30 Memory Slot, Form Factor and Socket combinations'",[151,100936,3640],{"class":503},[151,100938,100939,100941,100944],{"class":469,"line":500},[151,100940,65133],{"class":503},[151,100942,100943],{"class":481},"'Memory Slot, Form Factor and Socket combinations'",[151,100945,3640],{"class":503},[151,100947,100948],{"class":469,"line":509},[151,100949,100950],{"class":503},"plt.figure()\n",[11,100952,100953],{},[2718,100954],{"alt":20386,"src":100955},"/static/pcpp/motherboard/features_vs_price.png",[11,100957,100958],{},"Of the 2400 motherboards in the dataset, there is price information for only 618 motherboards. The average motherboard price is $157.50. Here's a visualizations of motherboard prices:",[459,100960,100962],{"className":13136,"code":100961,"language":12886,"meta":464,"style":464},"plt.figure(figsize=(12,8))\ndf[(df.avg!=0)&(df.avg\u003C750)].avg.hist(bins=30)\nplt.title('Motherboard prices', fontsize=14)\nplt.xlabel('Price', fontsize=14)\nplt.ylabel('Count', fontsize=14)\nplt.xticks(fontsize=13)\nplt.yticks(fontsize=13)\nplt.savefig(os.path.expanduser('~/Documents/GitHub/briancaffey.github.io/img/motherboard/price_histogram.png'))\n",[30,100963,100964,100982,101011,101028,101044,101060,101072,101084],{"__ignoreMap":464},[151,100965,100966,100968,100970,100972,100974,100976,100978,100980],{"class":469,"line":470},[151,100967,44355],{"class":503},[151,100969,44358],{"class":15210},[151,100971,1876],{"class":1869},[151,100973,12386],{"class":503},[151,100975,42360],{"class":477},[151,100977,3634],{"class":503},[151,100979,24369],{"class":477},[151,100981,12451],{"class":503},[151,100983,100984,100986,100988,100990,100992,100994,100996,100998,101000,101003,101005,101007,101009],{"class":469,"line":488},[151,100985,100850],{"class":503},[151,100987,58602],{"class":1869},[151,100989,9181],{"class":477},[151,100991,748],{"class":503},[151,100993,54214],{"class":1869},[151,100995,100313],{"class":503},[151,100997,3613],{"class":1869},[151,100999,74087],{"class":477},[151,101001,101002],{"class":503},")].avg.hist(",[151,101004,87626],{"class":15210},[151,101006,1876],{"class":1869},[151,101008,42017],{"class":477},[151,101010,3640],{"class":503},[151,101012,101013,101015,101018,101020,101022,101024,101026],{"class":469,"line":500},[151,101014,65123],{"class":503},[151,101016,101017],{"class":481},"'Motherboard prices'",[151,101019,106],{"class":503},[151,101021,99065],{"class":15210},[151,101023,1876],{"class":1869},[151,101025,67140],{"class":477},[151,101027,3640],{"class":503},[151,101029,101030,101032,101034,101036,101038,101040,101042],{"class":469,"line":509},[151,101031,65133],{"class":503},[151,101033,99095],{"class":481},[151,101035,106],{"class":503},[151,101037,99065],{"class":15210},[151,101039,1876],{"class":1869},[151,101041,67140],{"class":477},[151,101043,3640],{"class":503},[151,101045,101046,101048,101050,101052,101054,101056,101058],{"class":469,"line":517},[151,101047,65143],{"class":503},[151,101049,87648],{"class":481},[151,101051,106],{"class":503},[151,101053,99065],{"class":15210},[151,101055,1876],{"class":1869},[151,101057,67140],{"class":477},[151,101059,3640],{"class":503},[151,101061,101062,101064,101066,101068,101070],{"class":469,"line":534},[151,101063,65163],{"class":503},[151,101065,99065],{"class":15210},[151,101067,1876],{"class":1869},[151,101069,42327],{"class":477},[151,101071,3640],{"class":503},[151,101073,101074,101076,101078,101080,101082],{"class":469,"line":1413},[151,101075,99502],{"class":503},[151,101077,99065],{"class":15210},[151,101079,1876],{"class":1869},[151,101081,42327],{"class":477},[151,101083,3640],{"class":503},[151,101085,101086,101088,101091],{"class":469,"line":1418},[151,101087,93826],{"class":503},[151,101089,101090],{"class":481},"'~/Documents/GitHub/briancaffey.github.io/img/motherboard/price_histogram.png'",[151,101092,12451],{"class":503},[11,101094,101095],{},[2718,101096],{"alt":20386,"src":101097},"/static/pcpp/motherboard/price_histogram.png",[11,101099,101100,101101,101106],{},"Another feature that may be indicative of price is SLI support. Here's a description of SLI from a ",[20,101102,101105],{"href":101103,"rel":101104},"http://superuser.com/questions/562631/what-does-sli-ready-mean-and-how-do-i-use-it",[24],"superuser"," forum post:",[210,101108,101109],{},[11,101110,101111],{},"Scalable Link Interface (SLI) is a brand name for a multi-GPU solution developed by NVIDIA for linking two or more video cards together to produce a single output. SLI is an application of parallel processing for computer graphics, meant to increase the processing power available for graphics.",[11,101113,19225,101114,101117],{},[30,101115,101116],{},"SLI Support"," feature values include NaN, Yes, 3 and 4. Intuition tells me that a motherboard supporting 4 graphics cards will be more expensive than motherboard supporing only graphics cards.",[11,101119,101120,101121,208],{},"Here's a look at motherboard prices by ",[30,101122,101116],{},[459,101124,101126],{"className":13136,"code":101125,"language":12886,"meta":464,"style":464},"df[df.avg>0].boxplot(column='avg', by='SLI Support', figsize=(12,8))\nplt.ylim([0,650])\nplt.suptitle('')\nplt.title('Motherboard Prices by SLI Support',fontsize=14)\nplt.xlabel('SLI Support',fontsize=14)\nplt.ylabel('Price', fontsize=14)\nplt.xticks(fontsize=13)\nplt.yticks(fontsize=13)\nplt.savefig(os.path.expanduser('~/Documents/GitHub/briancaffey.github.io/img/motherboard/SLI_prices.png'))\n",[30,101127,101128,101171,101183,101191,101208,101224,101240,101252,101264],{"__ignoreMap":464},[151,101129,101130,101133,101135,101137,101140,101142,101144,101146,101148,101150,101152,101155,101157,101159,101161,101163,101165,101167,101169],{"class":469,"line":470},[151,101131,101132],{"class":503},"df[df.avg",[151,101134,3663],{"class":1869},[151,101136,9181],{"class":477},[151,101138,101139],{"class":503},"].boxplot(",[151,101141,100697],{"class":15210},[151,101143,1876],{"class":1869},[151,101145,99593],{"class":481},[151,101147,106],{"class":503},[151,101149,65808],{"class":15210},[151,101151,1876],{"class":1869},[151,101153,101154],{"class":481},"'SLI Support'",[151,101156,106],{"class":503},[151,101158,44358],{"class":15210},[151,101160,1876],{"class":1869},[151,101162,12386],{"class":503},[151,101164,42360],{"class":477},[151,101166,3634],{"class":503},[151,101168,24369],{"class":477},[151,101170,12451],{"class":503},[151,101172,101173,101175,101177,101179,101181],{"class":469,"line":488},[151,101174,100730],{"class":503},[151,101176,9181],{"class":477},[151,101178,3634],{"class":503},[151,101180,73958],{"class":477},[151,101182,38820],{"class":503},[151,101184,101185,101187,101189],{"class":469,"line":500},[151,101186,100745],{"class":503},[151,101188,2301],{"class":481},[151,101190,3640],{"class":503},[151,101192,101193,101195,101198,101200,101202,101204,101206],{"class":469,"line":509},[151,101194,65123],{"class":503},[151,101196,101197],{"class":481},"'Motherboard Prices by SLI Support'",[151,101199,3634],{"class":503},[151,101201,99065],{"class":15210},[151,101203,1876],{"class":1869},[151,101205,67140],{"class":477},[151,101207,3640],{"class":503},[151,101209,101210,101212,101214,101216,101218,101220,101222],{"class":469,"line":517},[151,101211,65133],{"class":503},[151,101213,101154],{"class":481},[151,101215,3634],{"class":503},[151,101217,99065],{"class":15210},[151,101219,1876],{"class":1869},[151,101221,67140],{"class":477},[151,101223,3640],{"class":503},[151,101225,101226,101228,101230,101232,101234,101236,101238],{"class":469,"line":534},[151,101227,65143],{"class":503},[151,101229,99095],{"class":481},[151,101231,106],{"class":503},[151,101233,99065],{"class":15210},[151,101235,1876],{"class":1869},[151,101237,67140],{"class":477},[151,101239,3640],{"class":503},[151,101241,101242,101244,101246,101248,101250],{"class":469,"line":1413},[151,101243,65163],{"class":503},[151,101245,99065],{"class":15210},[151,101247,1876],{"class":1869},[151,101249,42327],{"class":477},[151,101251,3640],{"class":503},[151,101253,101254,101256,101258,101260,101262],{"class":469,"line":1418},[151,101255,99502],{"class":503},[151,101257,99065],{"class":15210},[151,101259,1876],{"class":1869},[151,101261,42327],{"class":477},[151,101263,3640],{"class":503},[151,101265,101266,101268,101271],{"class":469,"line":2462},[151,101267,93826],{"class":503},[151,101269,101270],{"class":481},"'~/Documents/GitHub/briancaffey.github.io/img/motherboard/SLI_prices.png'",[151,101272,12451],{"class":503},[11,101274,101275],{},[2718,101276],{"alt":20386,"src":101277},"/static/pcpp/motherboard/SLI_prices.png",[11,101279,101280],{},"Memory slots on motherboard determine what type of memory and how much memory can be included in a PC. The amount of memory is also limited by the maximum supported memory of the CPU. Here'a a breakdown of the count of motherboards by memory type:",[459,101282,101284],{"className":13136,"code":101283,"language":12886,"meta":464,"style":464},"df['Memory Slots'].value_counts()[:8].plot(kind='bar', rot=45, figsize=(12,8))\nplt.title('Top Eight Motherboard Memory Slot types', fontsize=13)\nplt.ylabel('Count', fontsize=13)\nplt.xlabel('Memory Slot types', fontsize=13)\nplt.xticks(fontsize=13)\nplt.yticks(fontsize=13)\nplt.savefig(os.path.expanduser('~/Documents/GitHub/briancaffey.github.io/img/motherboard/motherboard_count_by_mem_type.png'))\n",[30,101285,101286,101330,101347,101363,101380,101392,101404],{"__ignoreMap":464},[151,101287,101288,101290,101292,101295,101297,101299,101301,101303,101305,101307,101310,101312,101314,101316,101318,101320,101322,101324,101326,101328],{"class":469,"line":470},[151,101289,70736],{"class":503},[151,101291,100860],{"class":481},[151,101293,101294],{"class":503},"].value_counts()[:",[151,101296,24369],{"class":477},[151,101298,100905],{"class":503},[151,101300,100637],{"class":15210},[151,101302,1876],{"class":1869},[151,101304,100642],{"class":481},[151,101306,106],{"class":503},[151,101308,101309],{"class":15210},"rot",[151,101311,1876],{"class":1869},[151,101313,87323],{"class":477},[151,101315,106],{"class":503},[151,101317,44358],{"class":15210},[151,101319,1876],{"class":1869},[151,101321,12386],{"class":503},[151,101323,42360],{"class":477},[151,101325,3634],{"class":503},[151,101327,24369],{"class":477},[151,101329,12451],{"class":503},[151,101331,101332,101334,101337,101339,101341,101343,101345],{"class":469,"line":488},[151,101333,65123],{"class":503},[151,101335,101336],{"class":481},"'Top Eight Motherboard Memory Slot types'",[151,101338,106],{"class":503},[151,101340,99065],{"class":15210},[151,101342,1876],{"class":1869},[151,101344,42327],{"class":477},[151,101346,3640],{"class":503},[151,101348,101349,101351,101353,101355,101357,101359,101361],{"class":469,"line":500},[151,101350,65143],{"class":503},[151,101352,87648],{"class":481},[151,101354,106],{"class":503},[151,101356,99065],{"class":15210},[151,101358,1876],{"class":1869},[151,101360,42327],{"class":477},[151,101362,3640],{"class":503},[151,101364,101365,101367,101370,101372,101374,101376,101378],{"class":469,"line":509},[151,101366,65133],{"class":503},[151,101368,101369],{"class":481},"'Memory Slot types'",[151,101371,106],{"class":503},[151,101373,99065],{"class":15210},[151,101375,1876],{"class":1869},[151,101377,42327],{"class":477},[151,101379,3640],{"class":503},[151,101381,101382,101384,101386,101388,101390],{"class":469,"line":517},[151,101383,65163],{"class":503},[151,101385,99065],{"class":15210},[151,101387,1876],{"class":1869},[151,101389,42327],{"class":477},[151,101391,3640],{"class":503},[151,101393,101394,101396,101398,101400,101402],{"class":469,"line":534},[151,101395,99502],{"class":503},[151,101397,99065],{"class":15210},[151,101399,1876],{"class":1869},[151,101401,42327],{"class":477},[151,101403,3640],{"class":503},[151,101405,101406,101408,101411],{"class":469,"line":1413},[151,101407,93826],{"class":503},[151,101409,101410],{"class":481},"'~/Documents/GitHub/briancaffey.github.io/img/motherboard/motherboard_count_by_mem_type.png'",[151,101412,12451],{"class":503},[11,101414,101415],{},[2718,101416],{"alt":20386,"src":101417},"/static/pcpp/motherboard/motherboard_count_by_mem_type.png",[11,101419,101420],{},"And here are price boxplots for the top three most common memory slot types:",[459,101422,101424],{"className":13136,"code":101423,"language":12886,"meta":464,"style":464},"df[(df.avg>0)&((df['Memory Slots']=='4 x 240-pin DIMM')|(df['Memory Slots']=='2 x 240-pin DIMM')|\\\n              (df['Memory Slots']=='4 x 288-pin DIMM'))].boxplot(column='avg', by='Memory Slots', figsize=(12,8))\nplt.ylim([0,400])\nplt.suptitle('')\nplt.title('Motherboard price distributions by top three Memory Slot types', fontsize=14)\nplt.xlabel('Memory Slot Type',fontsize=14)\nplt.ylabel('Price', fontsize=14)\nplt.xticks(fontsize=13)\nplt.yticks(fontsize=13)\nplt.savefig(os.path.expanduser('~/Documents/GitHub/briancaffey.github.io/img/motherboard/prices_by_mem_slot.png'))\n",[30,101425,101426,101471,101518,101530,101538,101555,101572,101588,101600,101612],{"__ignoreMap":464},[151,101427,101428,101430,101432,101434,101436,101438,101441,101443,101445,101447,101450,101452,101454,101456,101458,101460,101462,101465,101467,101469],{"class":469,"line":470},[151,101429,100850],{"class":503},[151,101431,3663],{"class":1869},[151,101433,9181],{"class":477},[151,101435,748],{"class":503},[151,101437,54214],{"class":1869},[151,101439,101440],{"class":503},"((df[",[151,101442,100860],{"class":481},[151,101444,8582],{"class":503},[151,101446,17223],{"class":1869},[151,101448,101449],{"class":481},"'4 x 240-pin DIMM'",[151,101451,748],{"class":503},[151,101453,3947],{"class":1869},[151,101455,100431],{"class":503},[151,101457,100860],{"class":481},[151,101459,8582],{"class":503},[151,101461,17223],{"class":1869},[151,101463,101464],{"class":481},"'2 x 240-pin DIMM'",[151,101466,748],{"class":503},[151,101468,3947],{"class":1869},[151,101470,497],{"class":503},[151,101472,101473,101476,101478,101480,101482,101485,101488,101490,101492,101494,101496,101498,101500,101502,101504,101506,101508,101510,101512,101514,101516],{"class":469,"line":488},[151,101474,101475],{"class":503},"              (df[",[151,101477,100860],{"class":481},[151,101479,8582],{"class":503},[151,101481,17223],{"class":1869},[151,101483,101484],{"class":481},"'4 x 288-pin DIMM'",[151,101486,101487],{"class":503},"))].boxplot(",[151,101489,100697],{"class":15210},[151,101491,1876],{"class":1869},[151,101493,99593],{"class":481},[151,101495,106],{"class":503},[151,101497,65808],{"class":15210},[151,101499,1876],{"class":1869},[151,101501,100860],{"class":481},[151,101503,106],{"class":503},[151,101505,44358],{"class":15210},[151,101507,1876],{"class":1869},[151,101509,12386],{"class":503},[151,101511,42360],{"class":477},[151,101513,3634],{"class":503},[151,101515,24369],{"class":477},[151,101517,12451],{"class":503},[151,101519,101520,101522,101524,101526,101528],{"class":469,"line":500},[151,101521,100730],{"class":503},[151,101523,9181],{"class":477},[151,101525,3634],{"class":503},[151,101527,71554],{"class":477},[151,101529,38820],{"class":503},[151,101531,101532,101534,101536],{"class":469,"line":509},[151,101533,100745],{"class":503},[151,101535,2301],{"class":481},[151,101537,3640],{"class":503},[151,101539,101540,101542,101545,101547,101549,101551,101553],{"class":469,"line":517},[151,101541,65123],{"class":503},[151,101543,101544],{"class":481},"'Motherboard price distributions by top three Memory Slot types'",[151,101546,106],{"class":503},[151,101548,99065],{"class":15210},[151,101550,1876],{"class":1869},[151,101552,67140],{"class":477},[151,101554,3640],{"class":503},[151,101556,101557,101559,101562,101564,101566,101568,101570],{"class":469,"line":534},[151,101558,65133],{"class":503},[151,101560,101561],{"class":481},"'Memory Slot Type'",[151,101563,3634],{"class":503},[151,101565,99065],{"class":15210},[151,101567,1876],{"class":1869},[151,101569,67140],{"class":477},[151,101571,3640],{"class":503},[151,101573,101574,101576,101578,101580,101582,101584,101586],{"class":469,"line":1413},[151,101575,65143],{"class":503},[151,101577,99095],{"class":481},[151,101579,106],{"class":503},[151,101581,99065],{"class":15210},[151,101583,1876],{"class":1869},[151,101585,67140],{"class":477},[151,101587,3640],{"class":503},[151,101589,101590,101592,101594,101596,101598],{"class":469,"line":1418},[151,101591,65163],{"class":503},[151,101593,99065],{"class":15210},[151,101595,1876],{"class":1869},[151,101597,42327],{"class":477},[151,101599,3640],{"class":503},[151,101601,101602,101604,101606,101608,101610],{"class":469,"line":2462},[151,101603,99502],{"class":503},[151,101605,99065],{"class":15210},[151,101607,1876],{"class":1869},[151,101609,42327],{"class":477},[151,101611,3640],{"class":503},[151,101613,101614,101616,101619],{"class":469,"line":2471},[151,101615,93826],{"class":503},[151,101617,101618],{"class":481},"'~/Documents/GitHub/briancaffey.github.io/img/motherboard/prices_by_mem_slot.png'",[151,101620,12451],{"class":503},[11,101622,101623],{},[2718,101624],{"alt":20386,"src":101625},"/static/pcpp/motherboard/prices_by_mem_slot.png",[11,101627,101628],{},"Now let's perform a random forest regression for motherboards in the same way we did for PSUs to rank the importance of these features.",[11,101630,101631],{},"First we need to map categorical variable values to integers:",[459,101633,101635],{"className":13136,"code":101634,"language":12886,"meta":464,"style":464},"#map categorical variable values to integers\nchipset_mapping = {x:y for x,y in zip(df.Chipset.unique(),range(len(df.Chipset.unique())))}\ndf['Chipset_int'] = df.Chipset.map(chipset_mapping)\n\nsocket_mapping = {x:y for x,y in zip(df.socket.unique(),range(len(df.socket.unique())))}\ndf['socket_int'] = df.socket.map(socket_mapping)\n\ndf['Form_Factor'] = df['Form Factor']\nform_factor_mapping = {x:y for x,y in zip(df.Form_Factor.unique(),range(len(df.Form_Factor.unique())))}\ndf['Form_Factor_int'] = df.Form_Factor.map(form_factor_mapping)\n\ndf['Memory_Type'] = df['Memory Type']\nmemory_type_mapping = {x:y for x,y in zip(df.Memory_Type.unique(),range(len(df.Memory_Type.unique())))}\ndf['Memory_Type_int'] = df.Memory_Type.map(memory_type_mapping)\n\ndf['Memory_Slots'] = df['Memory Slots']\nmemory_slots_mapping = {x:y for x,y in zip(df.Memory_Slots.unique(),range(len(df.Memory_Slots.unique())))}\ndf['Memory_Slots_int'] = df.Memory_Slots.map(memory_slots_mapping)\n\nmanufacturer_mapping = {x:y for x,y in zip(df.Manufacturer.unique(),range(len(df.Manufacturer.unique())))}\ndf['Manufacturer_int'] = df.Manufacturer.map(manufacturer_mapping)\n\ndf['SLI_Support'] = df['SLI Support']\nsli_mapping = {x:y for x,y in zip(df.SLI_Support.unique(),range(len(df.SLI_Support.unique())))}\ndf['SLI_Support_int'] = df.SLI_Support.map(sli_mapping)\n",[30,101636,101637,101642,101671,101685,101689,101718,101732,101736,101753,101782,101796,101800,101818,101847,101861,101865,101882,101911,101925,101929,101955,101968,101972,101989,102028],{"__ignoreMap":464},[151,101638,101639],{"class":469,"line":470},[151,101640,101641],{"class":1527},"#map categorical variable values to integers\n",[151,101643,101644,101647,101649,101651,101653,101655,101657,101659,101662,101664,101666,101668],{"class":469,"line":488},[151,101645,101646],{"class":503},"chipset_mapping ",[151,101648,1876],{"class":1869},[151,101650,93913],{"class":503},[151,101652,16732],{"class":1869},[151,101654,98883],{"class":503},[151,101656,16417],{"class":1869},[151,101658,44908],{"class":2226},[151,101660,101661],{"class":503},"(df.Chipset.unique(),",[151,101663,99700],{"class":2226},[151,101665,12386],{"class":503},[151,101667,65875],{"class":2226},[151,101669,101670],{"class":503},"(df.Chipset.unique())))}\n",[151,101672,101673,101675,101678,101680,101682],{"class":469,"line":500},[151,101674,70736],{"class":503},[151,101676,101677],{"class":481},"'Chipset_int'",[151,101679,16654],{"class":503},[151,101681,1876],{"class":1869},[151,101683,101684],{"class":503}," df.Chipset.map(chipset_mapping)\n",[151,101686,101687],{"class":469,"line":509},[151,101688,1090],{"emptyLinePlaceholder":609},[151,101690,101691,101694,101696,101698,101700,101702,101704,101706,101709,101711,101713,101715],{"class":469,"line":517},[151,101692,101693],{"class":503},"socket_mapping ",[151,101695,1876],{"class":1869},[151,101697,93913],{"class":503},[151,101699,16732],{"class":1869},[151,101701,98883],{"class":503},[151,101703,16417],{"class":1869},[151,101705,44908],{"class":2226},[151,101707,101708],{"class":503},"(df.socket.unique(),",[151,101710,99700],{"class":2226},[151,101712,12386],{"class":503},[151,101714,65875],{"class":2226},[151,101716,101717],{"class":503},"(df.socket.unique())))}\n",[151,101719,101720,101722,101725,101727,101729],{"class":469,"line":534},[151,101721,70736],{"class":503},[151,101723,101724],{"class":481},"'socket_int'",[151,101726,16654],{"class":503},[151,101728,1876],{"class":1869},[151,101730,101731],{"class":503}," df.socket.map(socket_mapping)\n",[151,101733,101734],{"class":469,"line":1413},[151,101735,1090],{"emptyLinePlaceholder":609},[151,101737,101738,101740,101743,101745,101747,101749,101751],{"class":469,"line":1418},[151,101739,70736],{"class":503},[151,101741,101742],{"class":481},"'Form_Factor'",[151,101744,16654],{"class":503},[151,101746,1876],{"class":1869},[151,101748,70760],{"class":503},[151,101750,100865],{"class":481},[151,101752,3691],{"class":503},[151,101754,101755,101758,101760,101762,101764,101766,101768,101770,101773,101775,101777,101779],{"class":469,"line":2462},[151,101756,101757],{"class":503},"form_factor_mapping ",[151,101759,1876],{"class":1869},[151,101761,93913],{"class":503},[151,101763,16732],{"class":1869},[151,101765,98883],{"class":503},[151,101767,16417],{"class":1869},[151,101769,44908],{"class":2226},[151,101771,101772],{"class":503},"(df.Form_Factor.unique(),",[151,101774,99700],{"class":2226},[151,101776,12386],{"class":503},[151,101778,65875],{"class":2226},[151,101780,101781],{"class":503},"(df.Form_Factor.unique())))}\n",[151,101783,101784,101786,101789,101791,101793],{"class":469,"line":2471},[151,101785,70736],{"class":503},[151,101787,101788],{"class":481},"'Form_Factor_int'",[151,101790,16654],{"class":503},[151,101792,1876],{"class":1869},[151,101794,101795],{"class":503}," df.Form_Factor.map(form_factor_mapping)\n",[151,101797,101798],{"class":469,"line":2480},[151,101799,1090],{"emptyLinePlaceholder":609},[151,101801,101802,101804,101807,101809,101811,101813,101816],{"class":469,"line":2489},[151,101803,70736],{"class":503},[151,101805,101806],{"class":481},"'Memory_Type'",[151,101808,16654],{"class":503},[151,101810,1876],{"class":1869},[151,101812,70760],{"class":503},[151,101814,101815],{"class":481},"'Memory Type'",[151,101817,3691],{"class":503},[151,101819,101820,101823,101825,101827,101829,101831,101833,101835,101838,101840,101842,101844],{"class":469,"line":2497},[151,101821,101822],{"class":503},"memory_type_mapping ",[151,101824,1876],{"class":1869},[151,101826,93913],{"class":503},[151,101828,16732],{"class":1869},[151,101830,98883],{"class":503},[151,101832,16417],{"class":1869},[151,101834,44908],{"class":2226},[151,101836,101837],{"class":503},"(df.Memory_Type.unique(),",[151,101839,99700],{"class":2226},[151,101841,12386],{"class":503},[151,101843,65875],{"class":2226},[151,101845,101846],{"class":503},"(df.Memory_Type.unique())))}\n",[151,101848,101849,101851,101854,101856,101858],{"class":469,"line":3140},[151,101850,70736],{"class":503},[151,101852,101853],{"class":481},"'Memory_Type_int'",[151,101855,16654],{"class":503},[151,101857,1876],{"class":1869},[151,101859,101860],{"class":503}," df.Memory_Type.map(memory_type_mapping)\n",[151,101862,101863],{"class":469,"line":3149},[151,101864,1090],{"emptyLinePlaceholder":609},[151,101866,101867,101869,101872,101874,101876,101878,101880],{"class":469,"line":3158},[151,101868,70736],{"class":503},[151,101870,101871],{"class":481},"'Memory_Slots'",[151,101873,16654],{"class":503},[151,101875,1876],{"class":1869},[151,101877,70760],{"class":503},[151,101879,100860],{"class":481},[151,101881,3691],{"class":503},[151,101883,101884,101887,101889,101891,101893,101895,101897,101899,101902,101904,101906,101908],{"class":469,"line":3167},[151,101885,101886],{"class":503},"memory_slots_mapping ",[151,101888,1876],{"class":1869},[151,101890,93913],{"class":503},[151,101892,16732],{"class":1869},[151,101894,98883],{"class":503},[151,101896,16417],{"class":1869},[151,101898,44908],{"class":2226},[151,101900,101901],{"class":503},"(df.Memory_Slots.unique(),",[151,101903,99700],{"class":2226},[151,101905,12386],{"class":503},[151,101907,65875],{"class":2226},[151,101909,101910],{"class":503},"(df.Memory_Slots.unique())))}\n",[151,101912,101913,101915,101918,101920,101922],{"class":469,"line":3175},[151,101914,70736],{"class":503},[151,101916,101917],{"class":481},"'Memory_Slots_int'",[151,101919,16654],{"class":503},[151,101921,1876],{"class":1869},[151,101923,101924],{"class":503}," df.Memory_Slots.map(memory_slots_mapping)\n",[151,101926,101927],{"class":469,"line":3184},[151,101928,1090],{"emptyLinePlaceholder":609},[151,101930,101931,101933,101935,101937,101939,101941,101943,101945,101947,101949,101951,101953],{"class":469,"line":3193},[151,101932,99769],{"class":503},[151,101934,1876],{"class":1869},[151,101936,93913],{"class":503},[151,101938,16732],{"class":1869},[151,101940,98883],{"class":503},[151,101942,16417],{"class":1869},[151,101944,44908],{"class":2226},[151,101946,99784],{"class":503},[151,101948,99700],{"class":2226},[151,101950,12386],{"class":503},[151,101952,65875],{"class":2226},[151,101954,99793],{"class":503},[151,101956,101957,101959,101962,101964,101966],{"class":469,"line":3720},[151,101958,70736],{"class":503},[151,101960,101961],{"class":481},"'Manufacturer_int'",[151,101963,16654],{"class":503},[151,101965,1876],{"class":1869},[151,101967,99803],{"class":503},[151,101969,101970],{"class":469,"line":3729},[151,101971,1090],{"emptyLinePlaceholder":609},[151,101973,101974,101976,101979,101981,101983,101985,101987],{"class":469,"line":3735},[151,101975,70736],{"class":503},[151,101977,101978],{"class":481},"'SLI_Support'",[151,101980,16654],{"class":503},[151,101982,1876],{"class":1869},[151,101984,70760],{"class":503},[151,101986,101154],{"class":481},[151,101988,3691],{"class":503},[151,101990,101991,101994,101996,101998,102000,102002,102004,102006,102009,102012,102015,102017,102019,102021,102023,102025],{"class":469,"line":3745},[151,101992,101993],{"class":503},"sli_mapping ",[151,101995,1876],{"class":1869},[151,101997,93913],{"class":503},[151,101999,16732],{"class":1869},[151,102001,98883],{"class":503},[151,102003,16417],{"class":1869},[151,102005,44908],{"class":2226},[151,102007,102008],{"class":503},"(df.",[151,102010,102011],{"class":477},"SLI_Support",[151,102013,102014],{"class":503},".unique(),",[151,102016,99700],{"class":2226},[151,102018,12386],{"class":503},[151,102020,65875],{"class":2226},[151,102022,102008],{"class":503},[151,102024,102011],{"class":477},[151,102026,102027],{"class":503},".unique())))}\n",[151,102029,102030,102032,102035,102037,102039,102042,102044],{"class":469,"line":3754},[151,102031,70736],{"class":503},[151,102033,102034],{"class":481},"'SLI_Support_int'",[151,102036,16654],{"class":503},[151,102038,1876],{"class":1869},[151,102040,102041],{"class":503}," df.",[151,102043,102011],{"class":477},[151,102045,102046],{"class":503},".map(sli_mapping)\n",[11,102048,102049],{},"And then drop null values and separate X and y:",[459,102051,102053],{"className":13136,"code":102052,"language":12886,"meta":464,"style":464},"cols = ['avg', 'max_mem', 'Memory_Slots_int', 'SLI_Support_int', 'Memory_Type_int', 'Form_Factor_int', 'socket_int','Chipset_int']\nfeature_cols = ['max_mem', 'Memory_Slots_int', 'SLI_Support_int','Memory_Type_int', 'Form_Factor_int', 'socket_int','Chipset_int']\nX = df[cols][df.avg>0].dropna()[feature_cols]\ny = df[cols][df.avg>0].avg\n",[30,102054,102055,102096,102132,102148],{"__ignoreMap":464},[151,102056,102057,102059,102061,102063,102065,102067,102070,102072,102074,102076,102078,102080,102082,102084,102086,102088,102090,102092,102094],{"class":469,"line":470},[151,102058,99545],{"class":503},[151,102060,1876],{"class":1869},[151,102062,6604],{"class":503},[151,102064,99593],{"class":481},[151,102066,106],{"class":503},[151,102068,102069],{"class":481},"'max_mem'",[151,102071,106],{"class":503},[151,102073,101917],{"class":481},[151,102075,106],{"class":503},[151,102077,102034],{"class":481},[151,102079,106],{"class":503},[151,102081,101853],{"class":481},[151,102083,106],{"class":503},[151,102085,101788],{"class":481},[151,102087,106],{"class":503},[151,102089,101724],{"class":481},[151,102091,3634],{"class":503},[151,102093,101677],{"class":481},[151,102095,3691],{"class":503},[151,102097,102098,102100,102102,102104,102106,102108,102110,102112,102114,102116,102118,102120,102122,102124,102126,102128,102130],{"class":469,"line":488},[151,102099,99605],{"class":503},[151,102101,1876],{"class":1869},[151,102103,6604],{"class":503},[151,102105,102069],{"class":481},[151,102107,106],{"class":503},[151,102109,101917],{"class":481},[151,102111,106],{"class":503},[151,102113,102034],{"class":481},[151,102115,3634],{"class":503},[151,102117,101853],{"class":481},[151,102119,106],{"class":503},[151,102121,101788],{"class":481},[151,102123,106],{"class":503},[151,102125,101724],{"class":481},[151,102127,3634],{"class":503},[151,102129,101677],{"class":481},[151,102131,3691],{"class":503},[151,102133,102134,102136,102138,102141,102143,102145],{"class":469,"line":500},[151,102135,87698],{"class":503},[151,102137,1876],{"class":1869},[151,102139,102140],{"class":503}," df[cols][df.avg",[151,102142,3663],{"class":1869},[151,102144,9181],{"class":477},[151,102146,102147],{"class":503},"].dropna()[feature_cols]\n",[151,102149,102150,102152,102154,102156,102158,102160],{"class":469,"line":509},[151,102151,98878],{"class":503},[151,102153,1876],{"class":1869},[151,102155,102140],{"class":503},[151,102157,3663],{"class":1869},[151,102159,9181],{"class":477},[151,102161,102162],{"class":503},"].avg\n",[11,102164,102165,102166,208],{},"Now we setup the model and take a look at the results ",[30,102167,102168],{},".feature_importances_",[459,102170,102172],{"className":13136,"code":102171,"language":12886,"meta":464,"style":464},"from sklearn.ensemble import RandomForestRegressor\n# max_features=8 is best and n_estimators=150 is sufficiently large\nrfreg = RandomForestRegressor(n_estimators=80, max_features=6, oob_score=True, random_state=5)\nrfreg.fit(X, y)\nfeature_importance = pd.DataFrame({'feature':feature_cols, 'importance':rfreg.feature_importances_}).sort_values(by='importance', ascending=False)\nprint feature_importance\n",[30,102173,102174,102184,102189,102229,102233,102265],{"__ignoreMap":464},[151,102175,102176,102178,102180,102182],{"class":469,"line":470},[151,102177,16853],{"class":1869},[151,102179,89932],{"class":503},[151,102181,16859],{"class":1869},[151,102183,99832],{"class":503},[151,102185,102186],{"class":469,"line":488},[151,102187,102188],{"class":1527},"# max_features=8 is best and n_estimators=150 is sufficiently large\n",[151,102190,102191,102193,102195,102197,102199,102201,102203,102205,102207,102209,102211,102213,102215,102217,102219,102221,102223,102225,102227],{"class":469,"line":500},[151,102192,99837],{"class":503},[151,102194,1876],{"class":1869},[151,102196,99842],{"class":503},[151,102198,99845],{"class":15210},[151,102200,1876],{"class":1869},[151,102202,27033],{"class":477},[151,102204,106],{"class":503},[151,102206,99854],{"class":15210},[151,102208,1876],{"class":1869},[151,102210,25038],{"class":477},[151,102212,106],{"class":503},[151,102214,99863],{"class":15210},[151,102216,1876],{"class":1869},[151,102218,36962],{"class":477},[151,102220,106],{"class":503},[151,102222,71775],{"class":15210},[151,102224,1876],{"class":1869},[151,102226,24380],{"class":477},[151,102228,3640],{"class":503},[151,102230,102231],{"class":469,"line":509},[151,102232,99882],{"class":503},[151,102234,102235,102237,102239,102241,102243,102245,102247,102249,102251,102253,102255,102257,102259,102261,102263],{"class":469,"line":517},[151,102236,99887],{"class":503},[151,102238,1876],{"class":1869},[151,102240,99892],{"class":503},[151,102242,99895],{"class":481},[151,102244,99898],{"class":503},[151,102246,99901],{"class":481},[151,102248,99904],{"class":503},[151,102250,65808],{"class":15210},[151,102252,1876],{"class":1869},[151,102254,99901],{"class":481},[151,102256,106],{"class":503},[151,102258,65817],{"class":15210},[151,102260,1876],{"class":1869},[151,102262,39461],{"class":477},[151,102264,3640],{"class":503},[151,102266,102267,102269],{"class":469,"line":534},[151,102268,18513],{"class":2226},[151,102270,102271],{"class":503}," feature_importance\n",[1131,102273,102274,102282],{},[1134,102275,102276],{},[1137,102277,102278,102280],{},[1140,102279,99935],{},[1140,102281,99938],{},[1153,102283,102284,102292,102300,102308,102316,102324,102332],{},[1137,102285,102286,102289],{},[1158,102287,102288],{},"Memory_Slots_int",[1158,102290,102291],{},"0.352454",[1137,102293,102294,102297],{},[1158,102295,102296],{},"SLI_Support_int",[1158,102298,102299],{},"0.155018",[1137,102301,102302,102305],{},[1158,102303,102304],{},"Memory_Type_int",[1158,102306,102307],{},"0.127274",[1137,102309,102310,102313],{},[1158,102311,102312],{},"Form_Factor_int",[1158,102314,102315],{},"0.113944",[1137,102317,102318,102321],{},[1158,102319,102320],{},"Chipset_int",[1158,102322,102323],{},"0.088664",[1137,102325,102326,102329],{},[1158,102327,102328],{},"max_mem",[1158,102330,102331],{},"0.085077",[1137,102333,102334,102337],{},[1158,102335,102336],{},"socket_int",[1158,102338,102339],{},"0.077571",[11,102341,102342,102343,102345],{},"And finally we can test the accuracy of the model by searching for an optimal ",[30,102344,99845],{}," number of decision trees with 5-fold cross validation:",[459,102347,102349],{"className":13136,"code":102348,"language":12886,"meta":464,"style":464},"from sklearn import metrics\nfrom sklearn.cross_validation import cross_val_score\n\n# list of values to try for n_estimators\nestimator_range = range(10, 310, 10)\n\n# list to store the average RMSE for each value of n_estimators\nRMSE_scores1 = []\n\n# use 5-fold cross-validation with each value of n_estimators (WARNING: SLOW!)\nfor estimator in estimator_range:\n    rfreg = RandomForestRegressor(n_estimators=estimator, random_state=1)\n    MSE_scores = cross_val_score(rfreg, X, y, cv=5, scoring='mean_squared_error')\n    RMSE_scores1.append(np.mean(np.sqrt(-MSE_scores)))\n\n\nplt.figure(figsize=(12,8))\nplt.xlabel('n_estimators', fontsize=14)\nplt.ylabel('RMSE (root-mean-sqaure error)', fontsize=14)\nplt.plot(estimator_range, RMSE_scores1)\nplt.xticks(fontsize=13)\nplt.yticks(fontsize=13)\nplt.savefig(os.path.expanduser('~/Documents/GitHub/briancaffey.github.io/img/motherboard/n_est_vs_rmse.png'))\n",[30,102350,102351,102362,102372,102376,102381,102403,102407,102412,102421,102425,102430,102440,102462,102486,102499,102503,102507,102525,102541,102558,102566,102578,102590],{"__ignoreMap":464},[151,102352,102353,102355,102357,102359],{"class":469,"line":470},[151,102354,16853],{"class":1869},[151,102356,83841],{"class":503},[151,102358,16859],{"class":1869},[151,102360,102361],{"class":503}," metrics\n",[151,102363,102364,102366,102368,102370],{"class":469,"line":488},[151,102365,16853],{"class":1869},[151,102367,100010],{"class":503},[151,102369,16859],{"class":1869},[151,102371,100015],{"class":503},[151,102373,102374],{"class":469,"line":500},[151,102375,1090],{"emptyLinePlaceholder":609},[151,102377,102378],{"class":469,"line":509},[151,102379,102380],{"class":1527},"# list of values to try for n_estimators\n",[151,102382,102383,102385,102387,102389,102391,102393,102395,102397,102399,102401],{"class":469,"line":517},[151,102384,100029],{"class":503},[151,102386,1876],{"class":1869},[151,102388,2793],{"class":2226},[151,102390,12386],{"class":503},[151,102392,12423],{"class":477},[151,102394,106],{"class":503},[151,102396,73527],{"class":477},[151,102398,106],{"class":503},[151,102400,12423],{"class":477},[151,102402,3640],{"class":503},[151,102404,102405],{"class":469,"line":534},[151,102406,1090],{"emptyLinePlaceholder":609},[151,102408,102409],{"class":469,"line":1413},[151,102410,102411],{"class":1527},"# list to store the average RMSE for each value of n_estimators\n",[151,102413,102414,102417,102419],{"class":469,"line":1418},[151,102415,102416],{"class":477},"RMSE_scores1",[151,102418,19865],{"class":1869},[151,102420,16606],{"class":503},[151,102422,102423],{"class":469,"line":2462},[151,102424,1090],{"emptyLinePlaceholder":609},[151,102426,102427],{"class":469,"line":2471},[151,102428,102429],{"class":1527},"# use 5-fold cross-validation with each value of n_estimators (WARNING: SLOW!)\n",[151,102431,102432,102434,102436,102438],{"class":469,"line":2480},[151,102433,16732],{"class":1869},[151,102435,100081],{"class":503},[151,102437,16417],{"class":1869},[151,102439,100086],{"class":503},[151,102441,102442,102444,102446,102448,102450,102452,102454,102456,102458,102460],{"class":469,"line":2489},[151,102443,100091],{"class":503},[151,102445,1876],{"class":1869},[151,102447,99842],{"class":503},[151,102449,99845],{"class":15210},[151,102451,1876],{"class":1869},[151,102453,100102],{"class":503},[151,102455,71775],{"class":15210},[151,102457,1876],{"class":1869},[151,102459,6760],{"class":477},[151,102461,3640],{"class":503},[151,102463,102464,102466,102468,102470,102472,102474,102476,102478,102480,102482,102484],{"class":469,"line":2497},[151,102465,100115],{"class":477},[151,102467,19865],{"class":1869},[151,102469,100120],{"class":503},[151,102471,100123],{"class":15210},[151,102473,1876],{"class":1869},[151,102475,24380],{"class":477},[151,102477,106],{"class":503},[151,102479,100132],{"class":15210},[151,102481,1876],{"class":1869},[151,102483,100137],{"class":481},[151,102485,3640],{"class":503},[151,102487,102488,102491,102493,102495,102497],{"class":469,"line":3140},[151,102489,102490],{"class":477},"    RMSE_scores1",[151,102492,100147],{"class":503},[151,102494,12445],{"class":1869},[151,102496,100152],{"class":477},[151,102498,100155],{"class":503},[151,102500,102501],{"class":469,"line":3149},[151,102502,1090],{"emptyLinePlaceholder":609},[151,102504,102505],{"class":469,"line":3158},[151,102506,1090],{"emptyLinePlaceholder":609},[151,102508,102509,102511,102513,102515,102517,102519,102521,102523],{"class":469,"line":3167},[151,102510,44355],{"class":503},[151,102512,44358],{"class":15210},[151,102514,1876],{"class":1869},[151,102516,12386],{"class":503},[151,102518,42360],{"class":477},[151,102520,3634],{"class":503},[151,102522,24369],{"class":477},[151,102524,12451],{"class":503},[151,102526,102527,102529,102531,102533,102535,102537,102539],{"class":469,"line":3175},[151,102528,65133],{"class":503},[151,102530,100180],{"class":481},[151,102532,106],{"class":503},[151,102534,99065],{"class":15210},[151,102536,1876],{"class":1869},[151,102538,67140],{"class":477},[151,102540,3640],{"class":503},[151,102542,102543,102545,102548,102550,102552,102554,102556],{"class":469,"line":3184},[151,102544,65143],{"class":503},[151,102546,102547],{"class":481},"'RMSE (root-mean-sqaure error)'",[151,102549,106],{"class":503},[151,102551,99065],{"class":15210},[151,102553,1876],{"class":1869},[151,102555,67140],{"class":477},[151,102557,3640],{"class":503},[151,102559,102560,102562,102564],{"class":469,"line":3193},[151,102561,100169],{"class":503},[151,102563,102416],{"class":477},[151,102565,3640],{"class":503},[151,102567,102568,102570,102572,102574,102576],{"class":469,"line":3720},[151,102569,65163],{"class":503},[151,102571,99065],{"class":15210},[151,102573,1876],{"class":1869},[151,102575,42327],{"class":477},[151,102577,3640],{"class":503},[151,102579,102580,102582,102584,102586,102588],{"class":469,"line":3729},[151,102581,99502],{"class":503},[151,102583,99065],{"class":15210},[151,102585,1876],{"class":1869},[151,102587,42327],{"class":477},[151,102589,3640],{"class":503},[151,102591,102592,102594,102597],{"class":469,"line":3735},[151,102593,93826],{"class":503},[151,102595,102596],{"class":481},"'~/Documents/GitHub/briancaffey.github.io/img/motherboard/n_est_vs_rmse.png'",[151,102598,12451],{"class":503},[11,102600,102601],{},[2718,102602],{"alt":20386,"src":102603},"/static/pcpp/motherboard/n_est_vs_rmse.png",[11,102605,102606],{},"The predictive accuracy is not great. Let's look at a few random motherboards with no pricing data and compare the model's prediction to prices on Amazon.",[459,102608,102610],{"className":13136,"code":102609,"language":12886,"meta":464,"style":464},"df[df.avg==0].sample(1)['Part #']\n",[30,102611,102612],{"__ignoreMap":464},[151,102613,102614,102616,102618,102620,102623,102625,102627,102630],{"class":469,"line":470},[151,102615,101132],{"class":503},[151,102617,17223],{"class":1869},[151,102619,9181],{"class":477},[151,102621,102622],{"class":503},"].sample(",[151,102624,6760],{"class":477},[151,102626,40832],{"class":503},[151,102628,102629],{"class":481},"'Part #'",[151,102631,3691],{"class":503},[11,102633,102634],{},[30,102635,102636],{},"1710 GA-Z97X-SOC FORCE",[11,102638,102639],{},"This is a random sample with no pricing data, and its index value is 1710.",[11,102641,102642],{},"The following will give us a prediction based on our model:",[459,102644,102646],{"className":13136,"code":102645,"language":12886,"meta":464,"style":464},"rfreg.predict(np.array(df.ix[1710][feature_cols]))\n",[30,102647,102648],{"__ignoreMap":464},[151,102649,102650,102653,102656],{"class":469,"line":470},[151,102651,102652],{"class":503},"rfreg.predict(np.array(df.ix[",[151,102654,102655],{"class":477},"1710",[151,102657,102658],{"class":503},"][feature_cols]))\n",[11,102660,102661],{},[30,102662,102663],{},"array([ 203.90425])",[11,102665,102666,102671],{},[20,102667,102670],{"href":102668,"rel":102669},"https://www.amazon.com/gp/offer-listing/B00JKCHEPS/ref=dp_olp_used_mbc?ie=UTF8&condition=used",[24],"This product"," is available used on Amazon for $249.00, so we have fairly significant error in this one prediction.",[11,102673,102674],{},"Here's one more example that I will cherry-pick:",[459,102676,102677],{"className":13136,"code":102609,"language":12886,"meta":464,"style":464},[30,102678,102679],{"__ignoreMap":464},[151,102680,102681,102683,102685,102687,102689,102691,102693,102695],{"class":469,"line":470},[151,102682,101132],{"class":503},[151,102684,17223],{"class":1869},[151,102686,9181],{"class":477},[151,102688,102622],{"class":503},[151,102690,6760],{"class":477},[151,102692,40832],{"class":503},[151,102694,102629],{"class":481},[151,102696,3691],{"class":503},[11,102698,102699],{},[30,102700,102701],{},"234 A76ML-K 3.0",[459,102703,102705],{"className":13136,"code":102704,"language":12886,"meta":464,"style":464},"rfreg.predict(np.array(df.ix[234][feature_cols]))\n",[30,102706,102707],{"__ignoreMap":464},[151,102708,102709,102711,102713],{"class":469,"line":470},[151,102710,102652],{"class":503},[151,102712,87013],{"class":477},[151,102714,102658],{"class":503},[11,102716,102717],{},[30,102718,102719],{},"array([ 82.61848925])",[11,102721,102722,102726,102727,102732],{},[20,102723,102670],{"href":102724,"rel":102725},"https://www.newegg.com/Product/Product.aspx?Item=N82E16813186215",[24]," is available on ",[20,102728,102731],{"href":102729,"rel":102730},"https://www.newegg.com",[24],"NewEgg.com"," for $57.95 (currently on sale for $39.95). Our model's prediction for this product price is slightly more accurate, but it also reveals a problem with trying to fill the missing data in this data set.",[11,102734,102735],{},"Most of the motherboards with no pricing data in the dataset are out of stock on all major online retailers and are also quite old. In the next part of this project we will be able to see how much pricing data is missing in the collection of 25,000 PC builds on PCPartPicker. I am anticipating that there will be very little missing data since people are building with new hardware.",[56,102737,72752],{"id":102738},"cpu",[11,102740,102741],{},"CPU price and performance are determined by several features, so I'll go through them one by one and show interesting relationships among different features and between features and price.",[14063,102743,102745],{"id":102744},"lithography","Lithography",[11,102747,102748,102749,102751],{},"One interesting feature of CPUs that distinguishes Intel and AMD (the two CPU manufacturers) is ",[51,102750,102744],{},". Lithography can be described as the average space between a processor's transistors, and it is an important factor that determines clock speed and power consumption. This graph shows lithography vs. CPU price for CPUs priced under $750:",[459,102753,102755],{"className":13136,"code":102754,"language":12886,"meta":464,"style":464},"plt.figure(figsize=(12,4))\nsns.set_style('whitegrid')\nplt.axis([0,70,0,750])\nplt.xlabel('Lithography', fontsize=14)\nplt.title('CPU Lithography and Price', fontsize=14)\nplt.ylabel('Price', fontsize=14)\nplt.scatter(df[df.Manufacturer==\"Intel\"].Lithography, df[df.Manufacturer==\"Intel\"].avg, alpha=.2, color='blue', s=50)\nplt.scatter(df[df.Manufacturer==\"AMD\"].Lithography, df[df.Manufacturer==\"AMD\"].avg, alpha=.2, color='red', s=50 )\nplt.legend(['Intel','AMD'], fontsize=14)\n",[30,102756,102757,102775,102783,102803,102820,102837,102853,102898,102940],{"__ignoreMap":464},[151,102758,102759,102761,102763,102765,102767,102769,102771,102773],{"class":469,"line":470},[151,102760,44355],{"class":503},[151,102762,44358],{"class":15210},[151,102764,1876],{"class":1869},[151,102766,12386],{"class":503},[151,102768,42360],{"class":477},[151,102770,3634],{"class":503},[151,102772,9187],{"class":477},[151,102774,12451],{"class":503},[151,102776,102777,102779,102781],{"class":469,"line":488},[151,102778,87588],{"class":503},[151,102780,87591],{"class":481},[151,102782,3640],{"class":503},[151,102784,102785,102787,102789,102791,102793,102795,102797,102799,102801],{"class":469,"line":500},[151,102786,99036],{"class":503},[151,102788,9181],{"class":477},[151,102790,3634],{"class":503},[151,102792,73169],{"class":477},[151,102794,3634],{"class":503},[151,102796,9181],{"class":477},[151,102798,3634],{"class":503},[151,102800,74087],{"class":477},[151,102802,38820],{"class":503},[151,102804,102805,102807,102810,102812,102814,102816,102818],{"class":469,"line":509},[151,102806,65133],{"class":503},[151,102808,102809],{"class":481},"'Lithography'",[151,102811,106],{"class":503},[151,102813,99065],{"class":15210},[151,102815,1876],{"class":1869},[151,102817,67140],{"class":477},[151,102819,3640],{"class":503},[151,102821,102822,102824,102827,102829,102831,102833,102835],{"class":469,"line":517},[151,102823,65123],{"class":503},[151,102825,102826],{"class":481},"'CPU Lithography and Price'",[151,102828,106],{"class":503},[151,102830,99065],{"class":15210},[151,102832,1876],{"class":1869},[151,102834,67140],{"class":477},[151,102836,3640],{"class":503},[151,102838,102839,102841,102843,102845,102847,102849,102851],{"class":469,"line":534},[151,102840,65143],{"class":503},[151,102842,99095],{"class":481},[151,102844,106],{"class":503},[151,102846,99065],{"class":15210},[151,102848,1876],{"class":1869},[151,102850,67140],{"class":477},[151,102852,3640],{"class":503},[151,102854,102855,102858,102860,102863,102866,102868,102870,102872,102874,102876,102879,102881,102883,102885,102888,102890,102892,102894,102896],{"class":469,"line":1413},[151,102856,102857],{"class":503},"plt.scatter(df[df.Manufacturer",[151,102859,17223],{"class":1869},[151,102861,102862],{"class":481},"\"Intel\"",[151,102864,102865],{"class":503},"].Lithography, df[df.Manufacturer",[151,102867,17223],{"class":1869},[151,102869,102862],{"class":481},[151,102871,99177],{"class":503},[151,102873,26256],{"class":15210},[151,102875,1876],{"class":1869},[151,102877,102878],{"class":477},".2",[151,102880,106],{"class":503},[151,102882,79362],{"class":15210},[151,102884,1876],{"class":1869},[151,102886,102887],{"class":481},"'blue'",[151,102889,106],{"class":503},[151,102891,55630],{"class":15210},[151,102893,1876],{"class":1869},[151,102895,73146],{"class":477},[151,102897,3640],{"class":503},[151,102899,102900,102902,102904,102907,102909,102911,102913,102915,102917,102919,102921,102923,102925,102927,102929,102931,102933,102935,102937],{"class":469,"line":1418},[151,102901,102857],{"class":503},[151,102903,17223],{"class":1869},[151,102905,102906],{"class":481},"\"AMD\"",[151,102908,102865],{"class":503},[151,102910,17223],{"class":1869},[151,102912,102906],{"class":481},[151,102914,99177],{"class":503},[151,102916,26256],{"class":15210},[151,102918,1876],{"class":1869},[151,102920,102878],{"class":477},[151,102922,106],{"class":503},[151,102924,79362],{"class":15210},[151,102926,1876],{"class":1869},[151,102928,80832],{"class":481},[151,102930,106],{"class":503},[151,102932,55630],{"class":15210},[151,102934,1876],{"class":1869},[151,102936,73146],{"class":477},[151,102938,102939],{"class":503}," )\n",[151,102941,102942,102945,102948,102950,102953,102955,102957,102959,102961],{"class":469,"line":2462},[151,102943,102944],{"class":503},"plt.legend([",[151,102946,102947],{"class":481},"'Intel'",[151,102949,3634],{"class":503},[151,102951,102952],{"class":481},"'AMD'",[151,102954,60308],{"class":503},[151,102956,99065],{"class":15210},[151,102958,1876],{"class":1869},[151,102960,67140],{"class":477},[151,102962,3640],{"class":503},[11,102964,102965],{},[2718,102966],{"alt":20386,"src":102967},"/static/pcpp/cpu/lith_vs_price.png",[11,102969,102970],{},"Intel's 22 nm and 14 nm lithography set it apart from AMD. From what I understand, AMD contracts its semi-conducter fabrication to other companies and Intel has its own advanced fabrication processes.",[14063,102972,102974],{"id":102973},"clock-speed-cpu-cores","Clock speed, CPU cores",[11,102976,102977],{},"CPU speed is measured in clock cycles. A clock cycle is the amount of time between two pulses of an oscillator (something that goes back and forth), and it helps determine the amount of instructions that the CPU can execute per second. Clock cycles used to be the best way to measure the speed of a CPU, but developments around 2005 in CPU architecture (and computer applications) have added other important features that must be considered alongside clock speed when determining the power of a PC.",[11,102979,102980],{},"Engineers had difficulty increasing clock speeds, so they started to develop multi-core processors. It's helpful to think of CPUs and CPUs cores like a kitchen. A single-core CPU is like a CPU with one cook, and is limited by how fast the cook can make food. Adding cores to a CPU can be thought of as adding cooks to the kitchen. The cooks don't get faster, but the productivity of the kitchen increases. Here's a look at the distribution of clock speeds by CPU:",[459,102982,102984],{"className":13136,"code":102983,"language":12886,"meta":464,"style":464},"df[(df.cores>0)&(df.avg>0)].boxplot(column='opfreq', by='cores', figsize=(12,8))\nplt.suptitle('')\nplt.title('CPU Operating Frequency by Core count', fontsize=14)\nplt.xlabel('CPU Core count', fontsize=14)\nplt.ylabel('Clock speed (MHz)', fontsize=14)\nplt.xticks(fontsize=13)\nplt.yticks(fontsize=13)\nplt.savefig(os.path.expanduser('~/Documents/GitHub/briancaffey.github.io/img/cpu/speed_vs_cores.png'))\n",[30,102985,102986,103039,103047,103064,103081,103098,103110,103122],{"__ignoreMap":464},[151,102987,102988,102991,102993,102995,102997,102999,103001,103003,103005,103007,103009,103011,103014,103016,103018,103020,103023,103025,103027,103029,103031,103033,103035,103037],{"class":469,"line":470},[151,102989,102990],{"class":503},"df[(df.cores",[151,102992,3663],{"class":1869},[151,102994,9181],{"class":477},[151,102996,748],{"class":503},[151,102998,54214],{"class":1869},[151,103000,100313],{"class":503},[151,103002,3663],{"class":1869},[151,103004,9181],{"class":477},[151,103006,100694],{"class":503},[151,103008,100697],{"class":15210},[151,103010,1876],{"class":1869},[151,103012,103013],{"class":481},"'opfreq'",[151,103015,106],{"class":503},[151,103017,65808],{"class":15210},[151,103019,1876],{"class":1869},[151,103021,103022],{"class":481},"'cores'",[151,103024,106],{"class":503},[151,103026,44358],{"class":15210},[151,103028,1876],{"class":1869},[151,103030,12386],{"class":503},[151,103032,42360],{"class":477},[151,103034,3634],{"class":503},[151,103036,24369],{"class":477},[151,103038,12451],{"class":503},[151,103040,103041,103043,103045],{"class":469,"line":488},[151,103042,100745],{"class":503},[151,103044,2301],{"class":481},[151,103046,3640],{"class":503},[151,103048,103049,103051,103054,103056,103058,103060,103062],{"class":469,"line":500},[151,103050,65123],{"class":503},[151,103052,103053],{"class":481},"'CPU Operating Frequency by Core count'",[151,103055,106],{"class":503},[151,103057,99065],{"class":15210},[151,103059,1876],{"class":1869},[151,103061,67140],{"class":477},[151,103063,3640],{"class":503},[151,103065,103066,103068,103071,103073,103075,103077,103079],{"class":469,"line":509},[151,103067,65133],{"class":503},[151,103069,103070],{"class":481},"'CPU Core count'",[151,103072,106],{"class":503},[151,103074,99065],{"class":15210},[151,103076,1876],{"class":1869},[151,103078,67140],{"class":477},[151,103080,3640],{"class":503},[151,103082,103083,103085,103088,103090,103092,103094,103096],{"class":469,"line":517},[151,103084,65143],{"class":503},[151,103086,103087],{"class":481},"'Clock speed (MHz)'",[151,103089,106],{"class":503},[151,103091,99065],{"class":15210},[151,103093,1876],{"class":1869},[151,103095,67140],{"class":477},[151,103097,3640],{"class":503},[151,103099,103100,103102,103104,103106,103108],{"class":469,"line":534},[151,103101,65163],{"class":503},[151,103103,99065],{"class":15210},[151,103105,1876],{"class":1869},[151,103107,42327],{"class":477},[151,103109,3640],{"class":503},[151,103111,103112,103114,103116,103118,103120],{"class":469,"line":1413},[151,103113,99502],{"class":503},[151,103115,99065],{"class":15210},[151,103117,1876],{"class":1869},[151,103119,42327],{"class":477},[151,103121,3640],{"class":503},[151,103123,103124,103126,103129],{"class":469,"line":1418},[151,103125,93826],{"class":503},[151,103127,103128],{"class":481},"'~/Documents/GitHub/briancaffey.github.io/img/cpu/speed_vs_cores.png'",[151,103130,12451],{"class":503},[11,103132,103133],{},[2718,103134],{"alt":20386,"src":103135},"/static/pcpp/cpu/speed_vs_cores.png",[11,103137,103138],{},"So once there are 22 cooks in the kitchen, everyone has to go slower, but the team of cooks can easily handle things that would totally swamp an individual cook, like rendering 4K video while live-streaming a AAA title at 1080p60.",[11,103140,103141],{},"You have to pay for each cook, so the core count has a big impact on CPU price:",[459,103143,103145],{"className":13136,"code":103144,"language":12886,"meta":464,"style":464},"df[(df.cores>0)&(df.avg>0)].boxplot(column='avg', by='cores', figsize=(12,8))\nplt.suptitle('')\nplt.title('CPU Prices by Core count', fontsize=14)\nplt.xlabel('CPU Core count', fontsize=14)\nplt.ylabel('Prices', fontsize=14)\nplt.xticks(fontsize=13)\nplt.yticks(fontsize=13)\nplt.savefig(os.path.expanduser('~/Documents/GitHub/briancaffey.github.io/img/cpu/price_by_core.png'))\n",[30,103146,103147,103197,103205,103222,103238,103254,103266,103278],{"__ignoreMap":464},[151,103148,103149,103151,103153,103155,103157,103159,103161,103163,103165,103167,103169,103171,103173,103175,103177,103179,103181,103183,103185,103187,103189,103191,103193,103195],{"class":469,"line":470},[151,103150,102990],{"class":503},[151,103152,3663],{"class":1869},[151,103154,9181],{"class":477},[151,103156,748],{"class":503},[151,103158,54214],{"class":1869},[151,103160,100313],{"class":503},[151,103162,3663],{"class":1869},[151,103164,9181],{"class":477},[151,103166,100694],{"class":503},[151,103168,100697],{"class":15210},[151,103170,1876],{"class":1869},[151,103172,99593],{"class":481},[151,103174,106],{"class":503},[151,103176,65808],{"class":15210},[151,103178,1876],{"class":1869},[151,103180,103022],{"class":481},[151,103182,106],{"class":503},[151,103184,44358],{"class":15210},[151,103186,1876],{"class":1869},[151,103188,12386],{"class":503},[151,103190,42360],{"class":477},[151,103192,3634],{"class":503},[151,103194,24369],{"class":477},[151,103196,12451],{"class":503},[151,103198,103199,103201,103203],{"class":469,"line":488},[151,103200,100745],{"class":503},[151,103202,2301],{"class":481},[151,103204,3640],{"class":503},[151,103206,103207,103209,103212,103214,103216,103218,103220],{"class":469,"line":500},[151,103208,65123],{"class":503},[151,103210,103211],{"class":481},"'CPU Prices by Core count'",[151,103213,106],{"class":503},[151,103215,99065],{"class":15210},[151,103217,1876],{"class":1869},[151,103219,67140],{"class":477},[151,103221,3640],{"class":503},[151,103223,103224,103226,103228,103230,103232,103234,103236],{"class":469,"line":509},[151,103225,65133],{"class":503},[151,103227,103070],{"class":481},[151,103229,106],{"class":503},[151,103231,99065],{"class":15210},[151,103233,1876],{"class":1869},[151,103235,67140],{"class":477},[151,103237,3640],{"class":503},[151,103239,103240,103242,103244,103246,103248,103250,103252],{"class":469,"line":517},[151,103241,65143],{"class":503},[151,103243,98126],{"class":481},[151,103245,106],{"class":503},[151,103247,99065],{"class":15210},[151,103249,1876],{"class":1869},[151,103251,67140],{"class":477},[151,103253,3640],{"class":503},[151,103255,103256,103258,103260,103262,103264],{"class":469,"line":534},[151,103257,65163],{"class":503},[151,103259,99065],{"class":15210},[151,103261,1876],{"class":1869},[151,103263,42327],{"class":477},[151,103265,3640],{"class":503},[151,103267,103268,103270,103272,103274,103276],{"class":469,"line":1413},[151,103269,99502],{"class":503},[151,103271,99065],{"class":15210},[151,103273,1876],{"class":1869},[151,103275,42327],{"class":477},[151,103277,3640],{"class":503},[151,103279,103280,103282,103285],{"class":469,"line":1418},[151,103281,93826],{"class":503},[151,103283,103284],{"class":481},"'~/Documents/GitHub/briancaffey.github.io/img/cpu/price_by_core.png'",[151,103286,12451],{"class":503},[11,103288,103289],{},[2718,103290],{"alt":20386,"src":103291},"/static/pcpp/cpu/price_by_core.png",[11,103293,103294],{},"A kitchen with 22 cooks gets very hot in the same way that a CPU with 22 cores generates a lot of heat. The next graph shows how much heat (measured in something called TDP, or thermal design power) CPUs generate by core count:",[459,103296,103298],{"className":13136,"code":103297,"language":12886,"meta":464,"style":464},"df.boxplot(column='TDP', by='Cores', figsize=(12,8))\nplt.suptitle('')\nplt.ylim(0,200)\nplt.title('Thermal Design Power by Core count', fontsize=14)\nplt.xlabel('Cores', fontsize=14)\nplt.ylabel('Thermal Design Power (Watts)', fontsize=14)\nplt.xticks(fontsize=13)\nplt.yticks(fontsize=13)\nplt.savefig(os.path.expanduser('~/Documents/GitHub/briancaffey.github.io/img/cpu/tdp_by_cores.png'))\n",[30,103299,103300,103337,103345,103358,103375,103391,103408,103420,103432],{"__ignoreMap":464},[151,103301,103302,103305,103307,103309,103312,103314,103316,103318,103321,103323,103325,103327,103329,103331,103333,103335],{"class":469,"line":470},[151,103303,103304],{"class":503},"df.boxplot(",[151,103306,100697],{"class":15210},[151,103308,1876],{"class":1869},[151,103310,103311],{"class":481},"'TDP'",[151,103313,106],{"class":503},[151,103315,65808],{"class":15210},[151,103317,1876],{"class":1869},[151,103319,103320],{"class":481},"'Cores'",[151,103322,106],{"class":503},[151,103324,44358],{"class":15210},[151,103326,1876],{"class":1869},[151,103328,12386],{"class":503},[151,103330,42360],{"class":477},[151,103332,3634],{"class":503},[151,103334,24369],{"class":477},[151,103336,12451],{"class":503},[151,103338,103339,103341,103343],{"class":469,"line":488},[151,103340,100745],{"class":503},[151,103342,2301],{"class":481},[151,103344,3640],{"class":503},[151,103346,103347,103350,103352,103354,103356],{"class":469,"line":500},[151,103348,103349],{"class":503},"plt.ylim(",[151,103351,9181],{"class":477},[151,103353,3634],{"class":503},[151,103355,41624],{"class":477},[151,103357,3640],{"class":503},[151,103359,103360,103362,103365,103367,103369,103371,103373],{"class":469,"line":509},[151,103361,65123],{"class":503},[151,103363,103364],{"class":481},"'Thermal Design Power by Core count'",[151,103366,106],{"class":503},[151,103368,99065],{"class":15210},[151,103370,1876],{"class":1869},[151,103372,67140],{"class":477},[151,103374,3640],{"class":503},[151,103376,103377,103379,103381,103383,103385,103387,103389],{"class":469,"line":517},[151,103378,65133],{"class":503},[151,103380,103320],{"class":481},[151,103382,106],{"class":503},[151,103384,99065],{"class":15210},[151,103386,1876],{"class":1869},[151,103388,67140],{"class":477},[151,103390,3640],{"class":503},[151,103392,103393,103395,103398,103400,103402,103404,103406],{"class":469,"line":534},[151,103394,65143],{"class":503},[151,103396,103397],{"class":481},"'Thermal Design Power (Watts)'",[151,103399,106],{"class":503},[151,103401,99065],{"class":15210},[151,103403,1876],{"class":1869},[151,103405,67140],{"class":477},[151,103407,3640],{"class":503},[151,103409,103410,103412,103414,103416,103418],{"class":469,"line":1413},[151,103411,65163],{"class":503},[151,103413,99065],{"class":15210},[151,103415,1876],{"class":1869},[151,103417,42327],{"class":477},[151,103419,3640],{"class":503},[151,103421,103422,103424,103426,103428,103430],{"class":469,"line":1418},[151,103423,99502],{"class":503},[151,103425,99065],{"class":15210},[151,103427,1876],{"class":1869},[151,103429,42327],{"class":477},[151,103431,3640],{"class":503},[151,103433,103434,103436,103439],{"class":469,"line":2462},[151,103435,93826],{"class":503},[151,103437,103438],{"class":481},"'~/Documents/GitHub/briancaffey.github.io/img/cpu/tdp_by_cores.png'",[151,103440,12451],{"class":503},[11,103442,103443],{},[2718,103444],{"alt":20386,"src":103445},"/static/pcpp/cpu/tdp_by_cores.png",[14063,103447,103449],{"id":103448},"thermal-design-power-tdp","Thermal Design Power (TDP)",[11,103451,103452,103453,208],{},"Here is a desrciption of TDP from the ",[20,103454,103457],{"href":103455,"rel":103456},"https://en.wikipedia.org/wiki/Thermal_design_power",[24],"Wikipedia article on Thermal Design Power",[210,103459,103460],{},[11,103461,103462],{},"The thermal design power (TDP), sometimes called thermal design point, is the maximum amount of heat generated by a computer chip or component (often the CPU or GPU) that the cooling system in a computer is designed to dissipate in typical operation.",[11,103464,103465],{},"This is a specification for any type of processor and it is measured in joules per second (or watts) produced by the CPU while it performs a computationally intensive task. Each point on the following scatter plot shows core clock and boost clock speeds with color representing TDP:",[459,103467,103469],{"className":13136,"code":103468,"language":12886,"meta":464,"style":464},"plt.figure(figsize=(12,8))\nplt.subplot(111)\ndf1 = df[(df.opfreq>0)&(df.turbo>0)]\nplt.scatter(df1.opfreq, df1.turbo ,c=df1.TDP, s=60, cmap='Blues')\nplt.colorbar(label='Thermal Design Power')\nplt.title('Core clock vs. Boost clock and TDP (color)', fontsize=14)\nplt.ylabel('Boost clock (MHz)', fontsize=14)\nplt.xlabel('Core clock (MHz)', fontsize=14)\nplt.axis([1500,5000,1500,5000])\n",[30,103470,103471,103489,103498,103524,103559,103574,103591,103608,103625],{"__ignoreMap":464},[151,103472,103473,103475,103477,103479,103481,103483,103485,103487],{"class":469,"line":470},[151,103474,44355],{"class":503},[151,103476,44358],{"class":15210},[151,103478,1876],{"class":1869},[151,103480,12386],{"class":503},[151,103482,42360],{"class":477},[151,103484,3634],{"class":503},[151,103486,24369],{"class":477},[151,103488,12451],{"class":503},[151,103490,103491,103494,103496],{"class":469,"line":488},[151,103492,103493],{"class":503},"plt.subplot(",[151,103495,45392],{"class":477},[151,103497,3640],{"class":503},[151,103499,103500,103502,103504,103507,103509,103511,103513,103515,103518,103520,103522],{"class":469,"line":500},[151,103501,86777],{"class":503},[151,103503,1876],{"class":1869},[151,103505,103506],{"class":503}," df[(df.opfreq",[151,103508,3663],{"class":1869},[151,103510,9181],{"class":477},[151,103512,748],{"class":503},[151,103514,54214],{"class":1869},[151,103516,103517],{"class":503},"(df.turbo",[151,103519,3663],{"class":1869},[151,103521,9181],{"class":477},[151,103523,44576],{"class":503},[151,103525,103526,103529,103531,103533,103536,103539,103541,103543,103545,103547,103549,103552,103554,103557],{"class":469,"line":509},[151,103527,103528],{"class":503},"plt.scatter(df1.opfreq, df1.turbo ,",[151,103530,65290],{"class":15210},[151,103532,1876],{"class":1869},[151,103534,103535],{"class":503},"df1.",[151,103537,103538],{"class":477},"TDP",[151,103540,106],{"class":503},[151,103542,55630],{"class":15210},[151,103544,1876],{"class":1869},[151,103546,39825],{"class":477},[151,103548,106],{"class":503},[151,103550,103551],{"class":15210},"cmap",[151,103553,1876],{"class":1869},[151,103555,103556],{"class":481},"'Blues'",[151,103558,3640],{"class":503},[151,103560,103561,103564,103567,103569,103572],{"class":469,"line":517},[151,103562,103563],{"class":503},"plt.colorbar(",[151,103565,103566],{"class":15210},"label",[151,103568,1876],{"class":1869},[151,103570,103571],{"class":481},"'Thermal Design Power'",[151,103573,3640],{"class":503},[151,103575,103576,103578,103581,103583,103585,103587,103589],{"class":469,"line":534},[151,103577,65123],{"class":503},[151,103579,103580],{"class":481},"'Core clock vs. Boost clock and TDP (color)'",[151,103582,106],{"class":503},[151,103584,99065],{"class":15210},[151,103586,1876],{"class":1869},[151,103588,67140],{"class":477},[151,103590,3640],{"class":503},[151,103592,103593,103595,103598,103600,103602,103604,103606],{"class":469,"line":1413},[151,103594,65143],{"class":503},[151,103596,103597],{"class":481},"'Boost clock (MHz)'",[151,103599,106],{"class":503},[151,103601,99065],{"class":15210},[151,103603,1876],{"class":1869},[151,103605,67140],{"class":477},[151,103607,3640],{"class":503},[151,103609,103610,103612,103615,103617,103619,103621,103623],{"class":469,"line":1418},[151,103611,65133],{"class":503},[151,103613,103614],{"class":481},"'Core clock (MHz)'",[151,103616,106],{"class":503},[151,103618,99065],{"class":15210},[151,103620,1876],{"class":1869},[151,103622,67140],{"class":477},[151,103624,3640],{"class":503},[151,103626,103627,103629,103632,103634,103636,103638,103640,103642,103644],{"class":469,"line":2462},[151,103628,99036],{"class":503},[151,103630,103631],{"class":477},"1500",[151,103633,3634],{"class":503},[151,103635,23145],{"class":477},[151,103637,3634],{"class":503},[151,103639,103631],{"class":477},[151,103641,3634],{"class":503},[151,103643,23145],{"class":477},[151,103645,38820],{"class":503},[11,103647,103648],{},[2718,103649],{"alt":20386,"src":103650},"/static/pcpp/cpu/core_v_boost.png",[11,103652,103653],{},"Here are two more graphs that show the relationship between clock speed, core count, TDP and price:",[459,103655,103657],{"className":13136,"code":103656,"language":12886,"meta":464,"style":464},"plt.figure(figsize=(12,8))\ndf2 = df[(df.avg>0)&(df['Hyper-Threading']==\"Yes\")]\nplt.scatter(df2.opfreq, df2.Cores, c=df2.avg, s=df2.TDP*2, cmap='Blues')\nplt.colorbar(label='Price')\nplt.title('Frequency vs. Core count for \\n Color: Price; Size:TPD', fontsize=14)\nplt.ylabel('Cores', fontsize=14)\nplt.xlabel('Operating Frequency', fontsize=14)\nplt.xticks(fontsize=13)\nplt.yticks(fontsize=13)\nplt.savefig(os.path.expanduser('~/Documents/GitHub/briancaffey.github.io/img/cpu/freq_v_cores.png'))\n",[30,103658,103659,103677,103707,103742,103754,103776,103792,103809,103821,103833],{"__ignoreMap":464},[151,103660,103661,103663,103665,103667,103669,103671,103673,103675],{"class":469,"line":470},[151,103662,44355],{"class":503},[151,103664,44358],{"class":15210},[151,103666,1876],{"class":1869},[151,103668,12386],{"class":503},[151,103670,42360],{"class":477},[151,103672,3634],{"class":503},[151,103674,24369],{"class":477},[151,103676,12451],{"class":503},[151,103678,103679,103681,103683,103685,103687,103689,103691,103693,103695,103698,103700,103702,103705],{"class":469,"line":488},[151,103680,87049],{"class":503},[151,103682,1876],{"class":1869},[151,103684,100420],{"class":503},[151,103686,3663],{"class":1869},[151,103688,9181],{"class":477},[151,103690,748],{"class":503},[151,103692,54214],{"class":1869},[151,103694,100431],{"class":503},[151,103696,103697],{"class":481},"'Hyper-Threading'",[151,103699,8582],{"class":503},[151,103701,17223],{"class":1869},[151,103703,103704],{"class":481},"\"Yes\"",[151,103706,44576],{"class":503},[151,103708,103709,103712,103714,103716,103719,103721,103723,103726,103728,103730,103732,103734,103736,103738,103740],{"class":469,"line":500},[151,103710,103711],{"class":503},"plt.scatter(df2.opfreq, df2.Cores, ",[151,103713,65290],{"class":15210},[151,103715,1876],{"class":1869},[151,103717,103718],{"class":503},"df2.avg, ",[151,103720,55630],{"class":15210},[151,103722,1876],{"class":1869},[151,103724,103725],{"class":503},"df2.",[151,103727,103538],{"class":477},[151,103729,23268],{"class":1869},[151,103731,6619],{"class":477},[151,103733,106],{"class":503},[151,103735,103551],{"class":15210},[151,103737,1876],{"class":1869},[151,103739,103556],{"class":481},[151,103741,3640],{"class":503},[151,103743,103744,103746,103748,103750,103752],{"class":469,"line":509},[151,103745,103563],{"class":503},[151,103747,103566],{"class":15210},[151,103749,1876],{"class":1869},[151,103751,99095],{"class":481},[151,103753,3640],{"class":503},[151,103755,103756,103758,103761,103763,103766,103768,103770,103772,103774],{"class":469,"line":517},[151,103757,65123],{"class":503},[151,103759,103760],{"class":481},"'Frequency vs. Core count for ",[151,103762,8043],{"class":477},[151,103764,103765],{"class":481}," Color: Price; Size:TPD'",[151,103767,106],{"class":503},[151,103769,99065],{"class":15210},[151,103771,1876],{"class":1869},[151,103773,67140],{"class":477},[151,103775,3640],{"class":503},[151,103777,103778,103780,103782,103784,103786,103788,103790],{"class":469,"line":534},[151,103779,65143],{"class":503},[151,103781,103320],{"class":481},[151,103783,106],{"class":503},[151,103785,99065],{"class":15210},[151,103787,1876],{"class":1869},[151,103789,67140],{"class":477},[151,103791,3640],{"class":503},[151,103793,103794,103796,103799,103801,103803,103805,103807],{"class":469,"line":1413},[151,103795,65133],{"class":503},[151,103797,103798],{"class":481},"'Operating Frequency'",[151,103800,106],{"class":503},[151,103802,99065],{"class":15210},[151,103804,1876],{"class":1869},[151,103806,67140],{"class":477},[151,103808,3640],{"class":503},[151,103810,103811,103813,103815,103817,103819],{"class":469,"line":1418},[151,103812,65163],{"class":503},[151,103814,99065],{"class":15210},[151,103816,1876],{"class":1869},[151,103818,42327],{"class":477},[151,103820,3640],{"class":503},[151,103822,103823,103825,103827,103829,103831],{"class":469,"line":2462},[151,103824,99502],{"class":503},[151,103826,99065],{"class":15210},[151,103828,1876],{"class":1869},[151,103830,42327],{"class":477},[151,103832,3640],{"class":503},[151,103834,103835,103837,103840],{"class":469,"line":2471},[151,103836,93826],{"class":503},[151,103838,103839],{"class":481},"'~/Documents/GitHub/briancaffey.github.io/img/cpu/freq_v_cores.png'",[151,103841,12451],{"class":503},[11,103843,103844],{},[2718,103845],{"alt":20386,"src":103846},"/static/pcpp/cpu/freq_v_cores.png",[11,103848,103849],{},"The CPUs shown in the graph above are all Intel CPUs that utilize a proprietary technology called hyper-threading. Hyper-threading is a new feature on CPUs that allows them to better schedule the tasks that they do so that they can minimize the time the need to wait for information to process. Another good analogy I have heard to explain hyper-threading is eating M&Ms as fast as possible with one hand vs. two hands. Hyper-threading is like using two hands to eat M&Ms, while you are eating an M&M with your left hand, your right hand is retrieving the next M&M from the bowl. Here is the same data with price on the y-axis and core count by color:",[459,103851,103853],{"className":13136,"code":103852,"language":12886,"meta":464,"style":464},"plt.figure(figsize=(12,8))\ndf2 = df[(df.avg>0)&(df['Hyper-Threading']==\"Yes\")]\nplt.scatter(df2.opfreq, df2.avg, c=df2.Cores, s=df2.TDP*2, cmap='Blues')\nplt.colorbar(label='Cores')\nplt.title('Frequency vs. Price and TPD (diameter)', fontsize=14)\nplt.ylabel('Price', fontsize=14)\nplt.xlabel('Operating Frequency', fontsize=14)\nplt.xticks(fontsize=13)\nplt.yticks(fontsize=13)\nplt.savefig(os.path.expanduser('~/Documents/GitHub/briancaffey.github.io/img/cpu/freq_v_price.png'))\n",[30,103854,103855,103873,103901,103935,103947,103964,103980,103996,104008,104020],{"__ignoreMap":464},[151,103856,103857,103859,103861,103863,103865,103867,103869,103871],{"class":469,"line":470},[151,103858,44355],{"class":503},[151,103860,44358],{"class":15210},[151,103862,1876],{"class":1869},[151,103864,12386],{"class":503},[151,103866,42360],{"class":477},[151,103868,3634],{"class":503},[151,103870,24369],{"class":477},[151,103872,12451],{"class":503},[151,103874,103875,103877,103879,103881,103883,103885,103887,103889,103891,103893,103895,103897,103899],{"class":469,"line":488},[151,103876,87049],{"class":503},[151,103878,1876],{"class":1869},[151,103880,100420],{"class":503},[151,103882,3663],{"class":1869},[151,103884,9181],{"class":477},[151,103886,748],{"class":503},[151,103888,54214],{"class":1869},[151,103890,100431],{"class":503},[151,103892,103697],{"class":481},[151,103894,8582],{"class":503},[151,103896,17223],{"class":1869},[151,103898,103704],{"class":481},[151,103900,44576],{"class":503},[151,103902,103903,103906,103908,103910,103913,103915,103917,103919,103921,103923,103925,103927,103929,103931,103933],{"class":469,"line":500},[151,103904,103905],{"class":503},"plt.scatter(df2.opfreq, df2.avg, ",[151,103907,65290],{"class":15210},[151,103909,1876],{"class":1869},[151,103911,103912],{"class":503},"df2.Cores, ",[151,103914,55630],{"class":15210},[151,103916,1876],{"class":1869},[151,103918,103725],{"class":503},[151,103920,103538],{"class":477},[151,103922,23268],{"class":1869},[151,103924,6619],{"class":477},[151,103926,106],{"class":503},[151,103928,103551],{"class":15210},[151,103930,1876],{"class":1869},[151,103932,103556],{"class":481},[151,103934,3640],{"class":503},[151,103936,103937,103939,103941,103943,103945],{"class":469,"line":509},[151,103938,103563],{"class":503},[151,103940,103566],{"class":15210},[151,103942,1876],{"class":1869},[151,103944,103320],{"class":481},[151,103946,3640],{"class":503},[151,103948,103949,103951,103954,103956,103958,103960,103962],{"class":469,"line":517},[151,103950,65123],{"class":503},[151,103952,103953],{"class":481},"'Frequency vs. Price and TPD (diameter)'",[151,103955,106],{"class":503},[151,103957,99065],{"class":15210},[151,103959,1876],{"class":1869},[151,103961,67140],{"class":477},[151,103963,3640],{"class":503},[151,103965,103966,103968,103970,103972,103974,103976,103978],{"class":469,"line":534},[151,103967,65143],{"class":503},[151,103969,99095],{"class":481},[151,103971,106],{"class":503},[151,103973,99065],{"class":15210},[151,103975,1876],{"class":1869},[151,103977,67140],{"class":477},[151,103979,3640],{"class":503},[151,103981,103982,103984,103986,103988,103990,103992,103994],{"class":469,"line":1413},[151,103983,65133],{"class":503},[151,103985,103798],{"class":481},[151,103987,106],{"class":503},[151,103989,99065],{"class":15210},[151,103991,1876],{"class":1869},[151,103993,67140],{"class":477},[151,103995,3640],{"class":503},[151,103997,103998,104000,104002,104004,104006],{"class":469,"line":1418},[151,103999,65163],{"class":503},[151,104001,99065],{"class":15210},[151,104003,1876],{"class":1869},[151,104005,42327],{"class":477},[151,104007,3640],{"class":503},[151,104009,104010,104012,104014,104016,104018],{"class":469,"line":2462},[151,104011,99502],{"class":503},[151,104013,99065],{"class":15210},[151,104015,1876],{"class":1869},[151,104017,42327],{"class":477},[151,104019,3640],{"class":503},[151,104021,104022,104024,104027],{"class":469,"line":2471},[151,104023,93826],{"class":503},[151,104025,104026],{"class":481},"'~/Documents/GitHub/briancaffey.github.io/img/cpu/freq_v_price.png'",[151,104028,12451],{"class":503},[11,104030,104031],{},[2718,104032],{"alt":20386,"src":104033},"/static/pcpp/cpu/freq_v_price.png",[11,104035,104036],{},"And finally, here is one more graph showing the difference in price between CPUs with and without hyperthreading technology:",[459,104038,104040],{"className":13136,"code":104039,"language":12886,"meta":464,"style":464},"df['Hyper_Threading'] = df['Hyper-Threading']\ndf[(df.Manufacturer=='Intel')&(df.avg>0)&((df.Cores==4)|(df.Cores==2)|(df.Cores==6)|(df.Cores==8))].groupby(['Hyper_Threading','Cores']).avg.agg(['mean', 'count']).plot(kind='bar', figsize=(12,8))\nplt.xlabel('Hyper Threading and Core combinations', fontsize=14)\nplt.ylabel('Price', fontsize=14)\nplt.title('Mean Price and Count for Combinations of 2, 4, 6 and 8 Core CPUs with and without Hyperthreading', fontsize=14)\nplt.legend(loc='upper left', fontsize=14)\nplt.xticks(fontsize=13)\nplt.yticks(fontsize=13)\nplt.legend(fontsize=13, loc='upper left')\nplt.savefig(os.path.expanduser('~/Documents/GitHub/briancaffey.github.io/img/cpu/hyper_threading_prices.png'))\n",[30,104041,104042,104059,104162,104179,104195,104212,104234,104246,104258,104278],{"__ignoreMap":464},[151,104043,104044,104046,104049,104051,104053,104055,104057],{"class":469,"line":470},[151,104045,70736],{"class":503},[151,104047,104048],{"class":481},"'Hyper_Threading'",[151,104050,16654],{"class":503},[151,104052,1876],{"class":1869},[151,104054,70760],{"class":503},[151,104056,103697],{"class":481},[151,104058,3691],{"class":503},[151,104060,104061,104064,104066,104068,104070,104072,104074,104076,104078,104080,104082,104085,104087,104089,104091,104093,104096,104098,104100,104102,104104,104106,104108,104110,104112,104114,104116,104118,104120,104123,104125,104127,104129,104131,104133,104135,104137,104140,104142,104144,104146,104148,104150,104152,104154,104156,104158,104160],{"class":469,"line":488},[151,104062,104063],{"class":503},"df[(df.Manufacturer",[151,104065,17223],{"class":1869},[151,104067,102947],{"class":481},[151,104069,748],{"class":503},[151,104071,54214],{"class":1869},[151,104073,100313],{"class":503},[151,104075,3663],{"class":1869},[151,104077,9181],{"class":477},[151,104079,748],{"class":503},[151,104081,54214],{"class":1869},[151,104083,104084],{"class":503},"((df.Cores",[151,104086,17223],{"class":1869},[151,104088,9187],{"class":477},[151,104090,748],{"class":503},[151,104092,3947],{"class":1869},[151,104094,104095],{"class":503},"(df.Cores",[151,104097,17223],{"class":1869},[151,104099,6619],{"class":477},[151,104101,748],{"class":503},[151,104103,3947],{"class":1869},[151,104105,104095],{"class":503},[151,104107,17223],{"class":1869},[151,104109,25038],{"class":477},[151,104111,748],{"class":503},[151,104113,3947],{"class":1869},[151,104115,104095],{"class":503},[151,104117,17223],{"class":1869},[151,104119,24369],{"class":477},[151,104121,104122],{"class":503},"))].groupby([",[151,104124,104048],{"class":481},[151,104126,3634],{"class":503},[151,104128,103320],{"class":481},[151,104130,100873],{"class":503},[151,104132,100876],{"class":481},[151,104134,106],{"class":503},[151,104136,100881],{"class":481},[151,104138,104139],{"class":503},"]).plot(",[151,104141,100637],{"class":15210},[151,104143,1876],{"class":1869},[151,104145,100642],{"class":481},[151,104147,106],{"class":503},[151,104149,44358],{"class":15210},[151,104151,1876],{"class":1869},[151,104153,12386],{"class":503},[151,104155,42360],{"class":477},[151,104157,3634],{"class":503},[151,104159,24369],{"class":477},[151,104161,12451],{"class":503},[151,104163,104164,104166,104169,104171,104173,104175,104177],{"class":469,"line":500},[151,104165,65133],{"class":503},[151,104167,104168],{"class":481},"'Hyper Threading and Core combinations'",[151,104170,106],{"class":503},[151,104172,99065],{"class":15210},[151,104174,1876],{"class":1869},[151,104176,67140],{"class":477},[151,104178,3640],{"class":503},[151,104180,104181,104183,104185,104187,104189,104191,104193],{"class":469,"line":509},[151,104182,65143],{"class":503},[151,104184,99095],{"class":481},[151,104186,106],{"class":503},[151,104188,99065],{"class":15210},[151,104190,1876],{"class":1869},[151,104192,67140],{"class":477},[151,104194,3640],{"class":503},[151,104196,104197,104199,104202,104204,104206,104208,104210],{"class":469,"line":517},[151,104198,65123],{"class":503},[151,104200,104201],{"class":481},"'Mean Price and Count for Combinations of 2, 4, 6 and 8 Core CPUs with and without Hyperthreading'",[151,104203,106],{"class":503},[151,104205,99065],{"class":15210},[151,104207,1876],{"class":1869},[151,104209,67140],{"class":477},[151,104211,3640],{"class":503},[151,104213,104214,104217,104220,104222,104224,104226,104228,104230,104232],{"class":469,"line":534},[151,104215,104216],{"class":503},"plt.legend(",[151,104218,104219],{"class":15210},"loc",[151,104221,1876],{"class":1869},[151,104223,99461],{"class":481},[151,104225,106],{"class":503},[151,104227,99065],{"class":15210},[151,104229,1876],{"class":1869},[151,104231,67140],{"class":477},[151,104233,3640],{"class":503},[151,104235,104236,104238,104240,104242,104244],{"class":469,"line":1413},[151,104237,65163],{"class":503},[151,104239,99065],{"class":15210},[151,104241,1876],{"class":1869},[151,104243,42327],{"class":477},[151,104245,3640],{"class":503},[151,104247,104248,104250,104252,104254,104256],{"class":469,"line":1418},[151,104249,99502],{"class":503},[151,104251,99065],{"class":15210},[151,104253,1876],{"class":1869},[151,104255,42327],{"class":477},[151,104257,3640],{"class":503},[151,104259,104260,104262,104264,104266,104268,104270,104272,104274,104276],{"class":469,"line":2462},[151,104261,104216],{"class":503},[151,104263,99065],{"class":15210},[151,104265,1876],{"class":1869},[151,104267,42327],{"class":477},[151,104269,106],{"class":503},[151,104271,104219],{"class":15210},[151,104273,1876],{"class":1869},[151,104275,99461],{"class":481},[151,104277,3640],{"class":503},[151,104279,104280,104282,104285],{"class":469,"line":2471},[151,104281,93826],{"class":503},[151,104283,104284],{"class":481},"'~/Documents/GitHub/briancaffey.github.io/img/cpu/hyper_threading_prices.png'",[151,104286,12451],{"class":503},[11,104288,104289],{},[2718,104290],{"alt":20386,"src":104291},"/static/pcpp/cpu/hyper_threading_prices.png",[14063,104293,104295],{"id":104294},"cpu-cooler","CPU Cooler",[11,104297,104298],{},"To manage the small amount of heat that is generated at each clock cycle it is necessary to direct heat away from the CPU. This brings us to CPU coolers, the next major class of PC components. Cooling a CPU is achieved with large blocks of aluminum that are attached to the CPU with thermal paste. Aluminum conducts heat well, so the heat is dispersed into fins and the fins are then cooled with fans, or a liquid passes over the cooling block and is sent through a radiator which is cooled with fans.",[11,104300,104301],{},"Here's a comparison of the number of liquid vs. non-liquid CPU coolers broken down by price:",[459,104303,104305],{"className":13136,"code":104304,"language":12886,"meta":464,"style":464},"plt.figure(figsize=(12,8))\nplt.hist(df[(df.avg>0)&(df.liquid==\"No\")].avg, bins = 30, alpha=.75, label='Non-liquid Cooler')\ndf[(df.avg>0)&(df.liquid==\"Yes\")].avg.hist(bins=30, alpha=.75, label='Liquid Cooler')\nplt.legend(loc='upper right', fontsize=14)\nplt.title('Count of Liquid and Non-Liquid CPU Coolers', fontsize=14)\nplt.xlabel('Price', fontsize=14)\nplt.ylabel('Count', fontsize=14)\nplt.xticks(fontsize=13)\nplt.yticks(fontsize=13)\nplt.savefig(os.path.expanduser('~/Documents/GitHub/briancaffey.github.io/img/cpu_cooler/prices_hist.png'))\n",[30,104306,104307,104325,104376,104421,104442,104459,104475,104491,104503,104515],{"__ignoreMap":464},[151,104308,104309,104311,104313,104315,104317,104319,104321,104323],{"class":469,"line":470},[151,104310,44355],{"class":503},[151,104312,44358],{"class":15210},[151,104314,1876],{"class":1869},[151,104316,12386],{"class":503},[151,104318,42360],{"class":477},[151,104320,3634],{"class":503},[151,104322,24369],{"class":477},[151,104324,12451],{"class":503},[151,104326,104327,104330,104332,104334,104336,104338,104341,104343,104346,104349,104351,104353,104356,104358,104360,104362,104365,104367,104369,104371,104374],{"class":469,"line":488},[151,104328,104329],{"class":503},"plt.hist(df[(df.avg",[151,104331,3663],{"class":1869},[151,104333,9181],{"class":477},[151,104335,748],{"class":503},[151,104337,54214],{"class":1869},[151,104339,104340],{"class":503},"(df.liquid",[151,104342,17223],{"class":1869},[151,104344,104345],{"class":481},"\"No\"",[151,104347,104348],{"class":503},")].avg, ",[151,104350,87626],{"class":15210},[151,104352,19865],{"class":1869},[151,104354,104355],{"class":477}," 30",[151,104357,106],{"class":503},[151,104359,26256],{"class":15210},[151,104361,1876],{"class":1869},[151,104363,104364],{"class":477},".75",[151,104366,106],{"class":503},[151,104368,103566],{"class":15210},[151,104370,1876],{"class":1869},[151,104372,104373],{"class":481},"'Non-liquid Cooler'",[151,104375,3640],{"class":503},[151,104377,104378,104380,104382,104384,104386,104388,104390,104392,104394,104396,104398,104400,104402,104404,104406,104408,104410,104412,104414,104416,104419],{"class":469,"line":500},[151,104379,100850],{"class":503},[151,104381,3663],{"class":1869},[151,104383,9181],{"class":477},[151,104385,748],{"class":503},[151,104387,54214],{"class":1869},[151,104389,104340],{"class":503},[151,104391,17223],{"class":1869},[151,104393,103704],{"class":481},[151,104395,101002],{"class":503},[151,104397,87626],{"class":15210},[151,104399,1876],{"class":1869},[151,104401,42017],{"class":477},[151,104403,106],{"class":503},[151,104405,26256],{"class":15210},[151,104407,1876],{"class":1869},[151,104409,104364],{"class":477},[151,104411,106],{"class":503},[151,104413,103566],{"class":15210},[151,104415,1876],{"class":1869},[151,104417,104418],{"class":481},"'Liquid Cooler'",[151,104420,3640],{"class":503},[151,104422,104423,104425,104427,104429,104432,104434,104436,104438,104440],{"class":469,"line":509},[151,104424,104216],{"class":503},[151,104426,104219],{"class":15210},[151,104428,1876],{"class":1869},[151,104430,104431],{"class":481},"'upper right'",[151,104433,106],{"class":503},[151,104435,99065],{"class":15210},[151,104437,1876],{"class":1869},[151,104439,67140],{"class":477},[151,104441,3640],{"class":503},[151,104443,104444,104446,104449,104451,104453,104455,104457],{"class":469,"line":517},[151,104445,65123],{"class":503},[151,104447,104448],{"class":481},"'Count of Liquid and Non-Liquid CPU Coolers'",[151,104450,106],{"class":503},[151,104452,99065],{"class":15210},[151,104454,1876],{"class":1869},[151,104456,67140],{"class":477},[151,104458,3640],{"class":503},[151,104460,104461,104463,104465,104467,104469,104471,104473],{"class":469,"line":534},[151,104462,65133],{"class":503},[151,104464,99095],{"class":481},[151,104466,106],{"class":503},[151,104468,99065],{"class":15210},[151,104470,1876],{"class":1869},[151,104472,67140],{"class":477},[151,104474,3640],{"class":503},[151,104476,104477,104479,104481,104483,104485,104487,104489],{"class":469,"line":1413},[151,104478,65143],{"class":503},[151,104480,87648],{"class":481},[151,104482,106],{"class":503},[151,104484,99065],{"class":15210},[151,104486,1876],{"class":1869},[151,104488,67140],{"class":477},[151,104490,3640],{"class":503},[151,104492,104493,104495,104497,104499,104501],{"class":469,"line":1418},[151,104494,65163],{"class":503},[151,104496,99065],{"class":15210},[151,104498,1876],{"class":1869},[151,104500,42327],{"class":477},[151,104502,3640],{"class":503},[151,104504,104505,104507,104509,104511,104513],{"class":469,"line":2462},[151,104506,99502],{"class":503},[151,104508,99065],{"class":15210},[151,104510,1876],{"class":1869},[151,104512,42327],{"class":477},[151,104514,3640],{"class":503},[151,104516,104517,104519,104522],{"class":469,"line":2471},[151,104518,93826],{"class":503},[151,104520,104521],{"class":481},"'~/Documents/GitHub/briancaffey.github.io/img/cpu_cooler/prices_hist.png'",[151,104523,12451],{"class":503},[11,104525,104526],{},[2718,104527],{"alt":20386,"src":104528},"/static/pcpp/cpu_cooler/prices_hist.png",[11,104530,104531],{},"Liquid coolers come with radiators in five different sizes. This chart shows average prices for liquid coolrs by radiator length:",[459,104533,104536],{"className":104534,"code":104535,"language":997},[995],"df[df.liquid==\"Yes\"].groupby('rad_size').avg.agg(['mean', 'count']).sort_values('count', ascending=False).plot(kind='bar', figsize=(12,8), rot=0)\nplt.title('CPU Cooler Price and Count by radiator size', fontsize=14)\nplt.legend(loc='upper left', fontsize=14)\nplt.xlabel('Radiator Size (mm)', fontsize=14)\nplt.xticks(fontsize=13)\nplt.yticks(fontsize=13)\nplt.savefig(os.path.expanduser('~/Documents/GitHub/briancaffey.github.io/img/cpu_cooler/rad_vs_price.png'))\n",[30,104537,104535],{"__ignoreMap":464},[11,104539,104540],{},[2718,104541],{"alt":20386,"src":104542},"/static/pcpp/cpu_cooler/rad_vs_price.png",[11,104544,104545],{},"Non-liquid coolers can be quite large and bulky to allow for more heat dispersion. Here's a scatterplot showing CPU cooler height and prices:",[459,104547,104549],{"className":13136,"code":104548,"language":12886,"meta":464,"style":464},"df1 = df[(df.avg>0)&(df.cooler_height>0)]\nplt.figure(figsize=(12,8))\nplt.scatter(df1.cooler_height, df1.avg)\nplt.title('CPU Height vs. Price (Non-liquid CPU Coolers only)', fontsize=14)\nplt.xlabel('CPU Cooler Height (mm)', fontsize=14)\nplt.ylabel('Price', fontsize=14)\nplt.xticks(fontsize=13)\nplt.yticks(fontsize=13)\nplt.savefig(os.path.expanduser('~/Documents/GitHub/briancaffey.github.io/img/cpu_cooler/cooler_height.png'))\n",[30,104550,104551,104576,104594,104599,104616,104633,104649,104661,104673],{"__ignoreMap":464},[151,104552,104553,104555,104557,104559,104561,104563,104565,104567,104570,104572,104574],{"class":469,"line":470},[151,104554,86777],{"class":503},[151,104556,1876],{"class":1869},[151,104558,100420],{"class":503},[151,104560,3663],{"class":1869},[151,104562,9181],{"class":477},[151,104564,748],{"class":503},[151,104566,54214],{"class":1869},[151,104568,104569],{"class":503},"(df.cooler_height",[151,104571,3663],{"class":1869},[151,104573,9181],{"class":477},[151,104575,44576],{"class":503},[151,104577,104578,104580,104582,104584,104586,104588,104590,104592],{"class":469,"line":488},[151,104579,44355],{"class":503},[151,104581,44358],{"class":15210},[151,104583,1876],{"class":1869},[151,104585,12386],{"class":503},[151,104587,42360],{"class":477},[151,104589,3634],{"class":503},[151,104591,24369],{"class":477},[151,104593,12451],{"class":503},[151,104595,104596],{"class":469,"line":500},[151,104597,104598],{"class":503},"plt.scatter(df1.cooler_height, df1.avg)\n",[151,104600,104601,104603,104606,104608,104610,104612,104614],{"class":469,"line":509},[151,104602,65123],{"class":503},[151,104604,104605],{"class":481},"'CPU Height vs. Price (Non-liquid CPU Coolers only)'",[151,104607,106],{"class":503},[151,104609,99065],{"class":15210},[151,104611,1876],{"class":1869},[151,104613,67140],{"class":477},[151,104615,3640],{"class":503},[151,104617,104618,104620,104623,104625,104627,104629,104631],{"class":469,"line":517},[151,104619,65133],{"class":503},[151,104621,104622],{"class":481},"'CPU Cooler Height (mm)'",[151,104624,106],{"class":503},[151,104626,99065],{"class":15210},[151,104628,1876],{"class":1869},[151,104630,67140],{"class":477},[151,104632,3640],{"class":503},[151,104634,104635,104637,104639,104641,104643,104645,104647],{"class":469,"line":534},[151,104636,65143],{"class":503},[151,104638,99095],{"class":481},[151,104640,106],{"class":503},[151,104642,99065],{"class":15210},[151,104644,1876],{"class":1869},[151,104646,67140],{"class":477},[151,104648,3640],{"class":503},[151,104650,104651,104653,104655,104657,104659],{"class":469,"line":1413},[151,104652,65163],{"class":503},[151,104654,99065],{"class":15210},[151,104656,1876],{"class":1869},[151,104658,42327],{"class":477},[151,104660,3640],{"class":503},[151,104662,104663,104665,104667,104669,104671],{"class":469,"line":1418},[151,104664,99502],{"class":503},[151,104666,99065],{"class":15210},[151,104668,1876],{"class":1869},[151,104670,42327],{"class":477},[151,104672,3640],{"class":503},[151,104674,104675,104677,104680],{"class":469,"line":2462},[151,104676,93826],{"class":503},[151,104678,104679],{"class":481},"'~/Documents/GitHub/briancaffey.github.io/img/cpu_cooler/cooler_height.png'",[151,104681,12451],{"class":503},[11,104683,104684],{},[2718,104685],{"alt":20386,"src":104686},"/static/pcpp/cpu_cooler/cooler_height.png",[11,104688,104689],{},"Here's one more graph on non-liquid coolers showing the relationship between maximum fan RPM and the maximum level of noise generated by the cooler:",[459,104691,104693],{"className":13136,"code":104692,"language":12886,"meta":464,"style":464},"plt.figure(figsize=(12,8))\ndf1 = df[(df.liquid=='No')&(df.avg>0)]\nplt.scatter(df1.rpm_max, df1.max_noise, s=100, c = df1.avg, cmap='Blues')\nplt.colorbar(label='Price')\nplt.axis([0,7500,10,60])\nplt.xlabel('Maximum RPM', fontsize=14)\nplt.ylabel('Maximum Noise', fontsize=14)\nplt.title('Maximum RPM vs. Maximum Noise and Price (color)', fontsize=14)\nplt.xticks(fontsize=13)\nplt.yticks(fontsize=13)\nplt.savefig(os.path.expanduser('~/Documents/GitHub/briancaffey.github.io/img/cpu_cooler/rpm_vs_noise.png'))\n",[30,104694,104695,104713,104739,104767,104779,104800,104817,104834,104851,104863,104875],{"__ignoreMap":464},[151,104696,104697,104699,104701,104703,104705,104707,104709,104711],{"class":469,"line":470},[151,104698,44355],{"class":503},[151,104700,44358],{"class":15210},[151,104702,1876],{"class":1869},[151,104704,12386],{"class":503},[151,104706,42360],{"class":477},[151,104708,3634],{"class":503},[151,104710,24369],{"class":477},[151,104712,12451],{"class":503},[151,104714,104715,104717,104719,104722,104724,104727,104729,104731,104733,104735,104737],{"class":469,"line":488},[151,104716,86777],{"class":503},[151,104718,1876],{"class":1869},[151,104720,104721],{"class":503}," df[(df.liquid",[151,104723,17223],{"class":1869},[151,104725,104726],{"class":481},"'No'",[151,104728,748],{"class":503},[151,104730,54214],{"class":1869},[151,104732,100313],{"class":503},[151,104734,3663],{"class":1869},[151,104736,9181],{"class":477},[151,104738,44576],{"class":503},[151,104740,104741,104744,104746,104748,104750,104752,104754,104756,104759,104761,104763,104765],{"class":469,"line":500},[151,104742,104743],{"class":503},"plt.scatter(df1.rpm_max, df1.max_noise, ",[151,104745,55630],{"class":15210},[151,104747,1876],{"class":1869},[151,104749,71821],{"class":477},[151,104751,106],{"class":503},[151,104753,65290],{"class":15210},[151,104755,19865],{"class":1869},[151,104757,104758],{"class":503}," df1.avg, ",[151,104760,103551],{"class":15210},[151,104762,1876],{"class":1869},[151,104764,103556],{"class":481},[151,104766,3640],{"class":503},[151,104768,104769,104771,104773,104775,104777],{"class":469,"line":509},[151,104770,103563],{"class":503},[151,104772,103566],{"class":15210},[151,104774,1876],{"class":1869},[151,104776,99095],{"class":481},[151,104778,3640],{"class":503},[151,104780,104781,104783,104785,104787,104790,104792,104794,104796,104798],{"class":469,"line":517},[151,104782,99036],{"class":503},[151,104784,9181],{"class":477},[151,104786,3634],{"class":503},[151,104788,104789],{"class":477},"7500",[151,104791,3634],{"class":503},[151,104793,12423],{"class":477},[151,104795,3634],{"class":503},[151,104797,39825],{"class":477},[151,104799,38820],{"class":503},[151,104801,104802,104804,104807,104809,104811,104813,104815],{"class":469,"line":534},[151,104803,65133],{"class":503},[151,104805,104806],{"class":481},"'Maximum RPM'",[151,104808,106],{"class":503},[151,104810,99065],{"class":15210},[151,104812,1876],{"class":1869},[151,104814,67140],{"class":477},[151,104816,3640],{"class":503},[151,104818,104819,104821,104824,104826,104828,104830,104832],{"class":469,"line":1413},[151,104820,65143],{"class":503},[151,104822,104823],{"class":481},"'Maximum Noise'",[151,104825,106],{"class":503},[151,104827,99065],{"class":15210},[151,104829,1876],{"class":1869},[151,104831,67140],{"class":477},[151,104833,3640],{"class":503},[151,104835,104836,104838,104841,104843,104845,104847,104849],{"class":469,"line":1418},[151,104837,65123],{"class":503},[151,104839,104840],{"class":481},"'Maximum RPM vs. Maximum Noise and Price (color)'",[151,104842,106],{"class":503},[151,104844,99065],{"class":15210},[151,104846,1876],{"class":1869},[151,104848,67140],{"class":477},[151,104850,3640],{"class":503},[151,104852,104853,104855,104857,104859,104861],{"class":469,"line":2462},[151,104854,65163],{"class":503},[151,104856,99065],{"class":15210},[151,104858,1876],{"class":1869},[151,104860,42327],{"class":477},[151,104862,3640],{"class":503},[151,104864,104865,104867,104869,104871,104873],{"class":469,"line":2471},[151,104866,99502],{"class":503},[151,104868,99065],{"class":15210},[151,104870,1876],{"class":1869},[151,104872,42327],{"class":477},[151,104874,3640],{"class":503},[151,104876,104877,104879,104882],{"class":469,"line":2480},[151,104878,93826],{"class":503},[151,104880,104881],{"class":481},"'~/Documents/GitHub/briancaffey.github.io/img/cpu_cooler/rpm_vs_noise.png'",[151,104883,12451],{"class":503},[11,104885,104886],{},[2718,104887],{"alt":20386,"src":104888},"/static/pcpp/cpu_cooler/rpm_vs_noise.png",[14063,104890,97465],{"id":14841},[11,104892,104893],{},"Memory is another important PC component that is particularlly important for content creators working with large video files. Memory speed and type are also important for benchmarking performed by hard-core PC gaming. PC memory is analogous to the space on your desk whereas hard drive disks are like file cabinets behind your desk. Things stored in memory can be accessed very quickly, but if you don't have enough memory then papers will start falling off of your desk and your application will crash becuase it won't be able to find what it is looking for, or won't have any space to put new information that it may need.",[11,104895,104896],{},"Here's a linear regression that captures the relationship between memory size and price:",[459,104898,104900],{"className":13136,"code":104899,"language":12886,"meta":464,"style":464},"from sklearn.linear_model import LinearRegression\nlreg = LinearRegression()\ndf1 = df[df.avg>0]\nfeat_cols = [u'size_gb']\nX = df1[feat_cols]\ny = df1.avg.reshape(df1.size_gb.shape[0],1)\nlreg.fit(X, y, sample_weight=None)\n\n#plot memory size vs. price\nplt.figure(figsize=(12,8))\nplt.scatter(df[df.avg>0].size_gb, df[df.avg>0].avg, s = 100, alpha=.05)\nplt.axis([0,130,0,1100])\nplt.title('Memory Kit prices by Size (GB)', fontsize=14)\nplt.xlabel('Memory Kit Size (GB)', fontsize=14)\nplt.ylabel('Price', fontsize=14)\n\n#plot regression line\nsize = df1.size_gb.reshape(df1.size_gb.shape[0],1)\npred = lreg.predict(df1.size_gb.reshape(df1.size_gb.shape[0],1))\nplt.plot(size ,pred, color='red')\nplt.savefig(os.path.expanduser('~/Documents/GitHub/briancaffey.github.io/img/memory/size_vs_price.png'))\n",[30,104901,104902,104912,104921,104936,104952,104961,104979,104993,104997,105002,105020,105055,105076,105093,105110,105126,105130,105135,105153,105171,105184],{"__ignoreMap":464},[151,104903,104904,104906,104908,104910],{"class":469,"line":470},[151,104905,16853],{"class":1869},[151,104907,100477],{"class":503},[151,104909,16859],{"class":1869},[151,104911,100482],{"class":503},[151,104913,104914,104917,104919],{"class":469,"line":488},[151,104915,104916],{"class":503},"lreg ",[151,104918,1876],{"class":1869},[151,104920,100492],{"class":503},[151,104922,104923,104925,104927,104930,104932,104934],{"class":469,"line":500},[151,104924,86777],{"class":503},[151,104926,1876],{"class":1869},[151,104928,104929],{"class":503}," df[df.avg",[151,104931,3663],{"class":1869},[151,104933,9181],{"class":477},[151,104935,3691],{"class":503},[151,104937,104938,104941,104943,104945,104947,104950],{"class":469,"line":509},[151,104939,104940],{"class":503},"feat_cols ",[151,104942,1876],{"class":1869},[151,104944,6604],{"class":503},[151,104946,68688],{"class":12347},[151,104948,104949],{"class":481},"'size_gb'",[151,104951,3691],{"class":503},[151,104953,104954,104956,104958],{"class":469,"line":517},[151,104955,87698],{"class":503},[151,104957,1876],{"class":1869},[151,104959,104960],{"class":503}," df1[feat_cols]\n",[151,104962,104963,104965,104967,104970,104972,104975,104977],{"class":469,"line":534},[151,104964,98878],{"class":503},[151,104966,1876],{"class":1869},[151,104968,104969],{"class":503}," df1.avg.reshape(df1.size_gb.shape[",[151,104971,9181],{"class":477},[151,104973,104974],{"class":503},"],",[151,104976,6760],{"class":477},[151,104978,3640],{"class":503},[151,104980,104981,104984,104987,104989,104991],{"class":469,"line":1413},[151,104982,104983],{"class":503},"lreg.fit(X, y, ",[151,104985,104986],{"class":15210},"sample_weight",[151,104988,1876],{"class":1869},[151,104990,15437],{"class":477},[151,104992,3640],{"class":503},[151,104994,104995],{"class":469,"line":1418},[151,104996,1090],{"emptyLinePlaceholder":609},[151,104998,104999],{"class":469,"line":2462},[151,105000,105001],{"class":1527},"#plot memory size vs. price\n",[151,105003,105004,105006,105008,105010,105012,105014,105016,105018],{"class":469,"line":2471},[151,105005,44355],{"class":503},[151,105007,44358],{"class":15210},[151,105009,1876],{"class":1869},[151,105011,12386],{"class":503},[151,105013,42360],{"class":477},[151,105015,3634],{"class":503},[151,105017,24369],{"class":477},[151,105019,12451],{"class":503},[151,105021,105022,105025,105027,105029,105032,105034,105036,105038,105040,105042,105044,105046,105048,105050,105053],{"class":469,"line":2480},[151,105023,105024],{"class":503},"plt.scatter(df[df.avg",[151,105026,3663],{"class":1869},[151,105028,9181],{"class":477},[151,105030,105031],{"class":503},"].size_gb, df[df.avg",[151,105033,3663],{"class":1869},[151,105035,9181],{"class":477},[151,105037,99177],{"class":503},[151,105039,55630],{"class":15210},[151,105041,19865],{"class":1869},[151,105043,57927],{"class":477},[151,105045,106],{"class":503},[151,105047,26256],{"class":15210},[151,105049,1876],{"class":1869},[151,105051,105052],{"class":477},".05",[151,105054,3640],{"class":503},[151,105056,105057,105059,105061,105063,105065,105067,105069,105071,105074],{"class":469,"line":2489},[151,105058,99036],{"class":503},[151,105060,9181],{"class":477},[151,105062,3634],{"class":503},[151,105064,73296],{"class":477},[151,105066,3634],{"class":503},[151,105068,9181],{"class":477},[151,105070,3634],{"class":503},[151,105072,105073],{"class":477},"1100",[151,105075,38820],{"class":503},[151,105077,105078,105080,105083,105085,105087,105089,105091],{"class":469,"line":2497},[151,105079,65123],{"class":503},[151,105081,105082],{"class":481},"'Memory Kit prices by Size (GB)'",[151,105084,106],{"class":503},[151,105086,99065],{"class":15210},[151,105088,1876],{"class":1869},[151,105090,67140],{"class":477},[151,105092,3640],{"class":503},[151,105094,105095,105097,105100,105102,105104,105106,105108],{"class":469,"line":3140},[151,105096,65133],{"class":503},[151,105098,105099],{"class":481},"'Memory Kit Size (GB)'",[151,105101,106],{"class":503},[151,105103,99065],{"class":15210},[151,105105,1876],{"class":1869},[151,105107,67140],{"class":477},[151,105109,3640],{"class":503},[151,105111,105112,105114,105116,105118,105120,105122,105124],{"class":469,"line":3149},[151,105113,65143],{"class":503},[151,105115,99095],{"class":481},[151,105117,106],{"class":503},[151,105119,99065],{"class":15210},[151,105121,1876],{"class":1869},[151,105123,67140],{"class":477},[151,105125,3640],{"class":503},[151,105127,105128],{"class":469,"line":3158},[151,105129,1090],{"emptyLinePlaceholder":609},[151,105131,105132],{"class":469,"line":3167},[151,105133,105134],{"class":1527},"#plot regression line\n",[151,105136,105137,105140,105142,105145,105147,105149,105151],{"class":469,"line":3175},[151,105138,105139],{"class":503},"size ",[151,105141,1876],{"class":1869},[151,105143,105144],{"class":503}," df1.size_gb.reshape(df1.size_gb.shape[",[151,105146,9181],{"class":477},[151,105148,104974],{"class":503},[151,105150,6760],{"class":477},[151,105152,3640],{"class":503},[151,105154,105155,105158,105160,105163,105165,105167,105169],{"class":469,"line":3184},[151,105156,105157],{"class":503},"pred ",[151,105159,1876],{"class":1869},[151,105161,105162],{"class":503}," lreg.predict(df1.size_gb.reshape(df1.size_gb.shape[",[151,105164,9181],{"class":477},[151,105166,104974],{"class":503},[151,105168,6760],{"class":477},[151,105170,12451],{"class":503},[151,105172,105173,105176,105178,105180,105182],{"class":469,"line":3193},[151,105174,105175],{"class":503},"plt.plot(size ,pred, ",[151,105177,79362],{"class":15210},[151,105179,1876],{"class":1869},[151,105181,80832],{"class":481},[151,105183,3640],{"class":503},[151,105185,105186,105188,105191],{"class":469,"line":3720},[151,105187,93826],{"class":503},[151,105189,105190],{"class":481},"'~/Documents/GitHub/briancaffey.github.io/img/memory/size_vs_price.png'",[151,105192,12451],{"class":503},[11,105194,105195],{},[2718,105196],{"alt":20386,"src":105197},"/static/pcpp/memory/size_vs_price.png",[11,105199,105200],{},"The R^2 value from the regression is 0.767, which means that 76 percent of the variation in the data is explained by the model. The variation might come from the fact that there are two important features that differentiate regular memory from enthusiast-grade memory: memory speed and CAS.",[11,105202,105203],{},"Memory module speed is measured in megatransfers per second (MT/s). Here is a graph of memory speeds vs. prices:",[459,105205,105207],{"className":13136,"code":105206,"language":12886,"meta":464,"style":464},"plt.figure(figsize=(12,8))\na=.5\ns=75\n\nddr2_speed = df.ddr_speed[(df.ddr_type == 'DDR2')&(df.size_gb==8)&(df.ddr_speed>0)&(df.avg>0)]\nddr2_ppgb = df.ppgb[(df.ddr_type == 'DDR2')&(df.size_gb==8)&(df.ddr_speed>0)&(df.avg>0)]\nplt.scatter(ddr2_speed,ddr2_ppgb,alpha = a, c='red', s=s)\n\nddr3_speed = df.ddr_speed[(df.ddr_type == 'DDR3')&(df.size_gb==8)&(df.ddr_speed>0)&(df.avg>0)]\nddr3_ppgb = df.ppgb[(df.ddr_type == 'DDR3')&(df.size_gb==8)&(df.ddr_speed>0)&(df.avg>0)]\nplt.scatter(ddr3_speed,ddr3_ppgb,alpha = a, c='yellow', s=s)\n\nddr4_speed = df.ddr_speed[(df.ddr_type == 'DDR4')&(df.size_gb==8)&(df.ddr_speed>0)&(df.avg>0)]\nddr4_ppgb = df.ppgb[(df.ddr_type == 'DDR4')&(df.size_gb==8)&(df.ddr_speed>0)&(df.avg>0)]\nplt.scatter(ddr4_speed,ddr4_ppgb,alpha = a, c='green', s=s)\n\nplt.axis([500,4000,0,25])\nplt.xlabel('Memory Speed (MT/s)', fontsize=14)\nplt.ylabel('Price per GB', fontsize=14)\nplt.title(\"Memory Speed vs. Price\", fontsize=14)\nplt.legend(['DDR2', 'DDR3', 'DDR4'], loc='lower right', fontsize=14)\nplt.xticks(fontsize=13)\nplt.yticks(fontsize=13)\nplt.savefig(os.path.expanduser('~/Documents/GitHub/briancaffey.github.io/img/memory/speed_vs_price.png'))\n",[30,105208,105209,105227,105236,105245,105249,105298,105344,105371,105375,105421,105466,105492,105496,105542,105587,105613,105617,105638,105655,105672,105689,105725,105737,105749],{"__ignoreMap":464},[151,105210,105211,105213,105215,105217,105219,105221,105223,105225],{"class":469,"line":470},[151,105212,44355],{"class":503},[151,105214,44358],{"class":15210},[151,105216,1876],{"class":1869},[151,105218,12386],{"class":503},[151,105220,42360],{"class":477},[151,105222,3634],{"class":503},[151,105224,24369],{"class":477},[151,105226,12451],{"class":503},[151,105228,105229,105231,105233],{"class":469,"line":488},[151,105230,20],{"class":503},[151,105232,1876],{"class":1869},[151,105234,105235],{"class":477},".5\n",[151,105237,105238,105240,105242],{"class":469,"line":500},[151,105239,55630],{"class":503},[151,105241,1876],{"class":1869},[151,105243,105244],{"class":477},"75\n",[151,105246,105247],{"class":469,"line":509},[151,105248,1090],{"emptyLinePlaceholder":609},[151,105250,105251,105254,105256,105259,105261,105264,105266,105268,105271,105273,105275,105277,105279,105282,105284,105286,105288,105290,105292,105294,105296],{"class":469,"line":517},[151,105252,105253],{"class":503},"ddr2_speed ",[151,105255,1876],{"class":1869},[151,105257,105258],{"class":503}," df.ddr_speed[(df.ddr_type ",[151,105260,17223],{"class":1869},[151,105262,105263],{"class":481}," 'DDR2'",[151,105265,748],{"class":503},[151,105267,54214],{"class":1869},[151,105269,105270],{"class":503},"(df.size_gb",[151,105272,17223],{"class":1869},[151,105274,24369],{"class":477},[151,105276,748],{"class":503},[151,105278,54214],{"class":1869},[151,105280,105281],{"class":503},"(df.ddr_speed",[151,105283,3663],{"class":1869},[151,105285,9181],{"class":477},[151,105287,748],{"class":503},[151,105289,54214],{"class":1869},[151,105291,100313],{"class":503},[151,105293,3663],{"class":1869},[151,105295,9181],{"class":477},[151,105297,44576],{"class":503},[151,105299,105300,105303,105305,105308,105310,105312,105314,105316,105318,105320,105322,105324,105326,105328,105330,105332,105334,105336,105338,105340,105342],{"class":469,"line":534},[151,105301,105302],{"class":503},"ddr2_ppgb ",[151,105304,1876],{"class":1869},[151,105306,105307],{"class":503}," df.ppgb[(df.ddr_type ",[151,105309,17223],{"class":1869},[151,105311,105263],{"class":481},[151,105313,748],{"class":503},[151,105315,54214],{"class":1869},[151,105317,105270],{"class":503},[151,105319,17223],{"class":1869},[151,105321,24369],{"class":477},[151,105323,748],{"class":503},[151,105325,54214],{"class":1869},[151,105327,105281],{"class":503},[151,105329,3663],{"class":1869},[151,105331,9181],{"class":477},[151,105333,748],{"class":503},[151,105335,54214],{"class":1869},[151,105337,100313],{"class":503},[151,105339,3663],{"class":1869},[151,105341,9181],{"class":477},[151,105343,44576],{"class":503},[151,105345,105346,105349,105351,105353,105356,105358,105360,105362,105364,105366,105368],{"class":469,"line":1413},[151,105347,105348],{"class":503},"plt.scatter(ddr2_speed,ddr2_ppgb,",[151,105350,26256],{"class":15210},[151,105352,19865],{"class":1869},[151,105354,105355],{"class":503}," a, ",[151,105357,65290],{"class":15210},[151,105359,1876],{"class":1869},[151,105361,80832],{"class":481},[151,105363,106],{"class":503},[151,105365,55630],{"class":15210},[151,105367,1876],{"class":1869},[151,105369,105370],{"class":503},"s)\n",[151,105372,105373],{"class":469,"line":1418},[151,105374,1090],{"emptyLinePlaceholder":609},[151,105376,105377,105380,105382,105384,105386,105389,105391,105393,105395,105397,105399,105401,105403,105405,105407,105409,105411,105413,105415,105417,105419],{"class":469,"line":2462},[151,105378,105379],{"class":503},"ddr3_speed ",[151,105381,1876],{"class":1869},[151,105383,105258],{"class":503},[151,105385,17223],{"class":1869},[151,105387,105388],{"class":481}," 'DDR3'",[151,105390,748],{"class":503},[151,105392,54214],{"class":1869},[151,105394,105270],{"class":503},[151,105396,17223],{"class":1869},[151,105398,24369],{"class":477},[151,105400,748],{"class":503},[151,105402,54214],{"class":1869},[151,105404,105281],{"class":503},[151,105406,3663],{"class":1869},[151,105408,9181],{"class":477},[151,105410,748],{"class":503},[151,105412,54214],{"class":1869},[151,105414,100313],{"class":503},[151,105416,3663],{"class":1869},[151,105418,9181],{"class":477},[151,105420,44576],{"class":503},[151,105422,105423,105426,105428,105430,105432,105434,105436,105438,105440,105442,105444,105446,105448,105450,105452,105454,105456,105458,105460,105462,105464],{"class":469,"line":2471},[151,105424,105425],{"class":503},"ddr3_ppgb ",[151,105427,1876],{"class":1869},[151,105429,105307],{"class":503},[151,105431,17223],{"class":1869},[151,105433,105388],{"class":481},[151,105435,748],{"class":503},[151,105437,54214],{"class":1869},[151,105439,105270],{"class":503},[151,105441,17223],{"class":1869},[151,105443,24369],{"class":477},[151,105445,748],{"class":503},[151,105447,54214],{"class":1869},[151,105449,105281],{"class":503},[151,105451,3663],{"class":1869},[151,105453,9181],{"class":477},[151,105455,748],{"class":503},[151,105457,54214],{"class":1869},[151,105459,100313],{"class":503},[151,105461,3663],{"class":1869},[151,105463,9181],{"class":477},[151,105465,44576],{"class":503},[151,105467,105468,105471,105473,105475,105477,105479,105481,105484,105486,105488,105490],{"class":469,"line":2480},[151,105469,105470],{"class":503},"plt.scatter(ddr3_speed,ddr3_ppgb,",[151,105472,26256],{"class":15210},[151,105474,19865],{"class":1869},[151,105476,105355],{"class":503},[151,105478,65290],{"class":15210},[151,105480,1876],{"class":1869},[151,105482,105483],{"class":481},"'yellow'",[151,105485,106],{"class":503},[151,105487,55630],{"class":15210},[151,105489,1876],{"class":1869},[151,105491,105370],{"class":503},[151,105493,105494],{"class":469,"line":2489},[151,105495,1090],{"emptyLinePlaceholder":609},[151,105497,105498,105501,105503,105505,105507,105510,105512,105514,105516,105518,105520,105522,105524,105526,105528,105530,105532,105534,105536,105538,105540],{"class":469,"line":2497},[151,105499,105500],{"class":503},"ddr4_speed ",[151,105502,1876],{"class":1869},[151,105504,105258],{"class":503},[151,105506,17223],{"class":1869},[151,105508,105509],{"class":481}," 'DDR4'",[151,105511,748],{"class":503},[151,105513,54214],{"class":1869},[151,105515,105270],{"class":503},[151,105517,17223],{"class":1869},[151,105519,24369],{"class":477},[151,105521,748],{"class":503},[151,105523,54214],{"class":1869},[151,105525,105281],{"class":503},[151,105527,3663],{"class":1869},[151,105529,9181],{"class":477},[151,105531,748],{"class":503},[151,105533,54214],{"class":1869},[151,105535,100313],{"class":503},[151,105537,3663],{"class":1869},[151,105539,9181],{"class":477},[151,105541,44576],{"class":503},[151,105543,105544,105547,105549,105551,105553,105555,105557,105559,105561,105563,105565,105567,105569,105571,105573,105575,105577,105579,105581,105583,105585],{"class":469,"line":3140},[151,105545,105546],{"class":503},"ddr4_ppgb ",[151,105548,1876],{"class":1869},[151,105550,105307],{"class":503},[151,105552,17223],{"class":1869},[151,105554,105509],{"class":481},[151,105556,748],{"class":503},[151,105558,54214],{"class":1869},[151,105560,105270],{"class":503},[151,105562,17223],{"class":1869},[151,105564,24369],{"class":477},[151,105566,748],{"class":503},[151,105568,54214],{"class":1869},[151,105570,105281],{"class":503},[151,105572,3663],{"class":1869},[151,105574,9181],{"class":477},[151,105576,748],{"class":503},[151,105578,54214],{"class":1869},[151,105580,100313],{"class":503},[151,105582,3663],{"class":1869},[151,105584,9181],{"class":477},[151,105586,44576],{"class":503},[151,105588,105589,105592,105594,105596,105598,105600,105602,105605,105607,105609,105611],{"class":469,"line":3149},[151,105590,105591],{"class":503},"plt.scatter(ddr4_speed,ddr4_ppgb,",[151,105593,26256],{"class":15210},[151,105595,19865],{"class":1869},[151,105597,105355],{"class":503},[151,105599,65290],{"class":15210},[151,105601,1876],{"class":1869},[151,105603,105604],{"class":481},"'green'",[151,105606,106],{"class":503},[151,105608,55630],{"class":15210},[151,105610,1876],{"class":1869},[151,105612,105370],{"class":503},[151,105614,105615],{"class":469,"line":3158},[151,105616,1090],{"emptyLinePlaceholder":609},[151,105618,105619,105621,105623,105625,105628,105630,105632,105634,105636],{"class":469,"line":3167},[151,105620,99036],{"class":503},[151,105622,12208],{"class":477},[151,105624,3634],{"class":503},[151,105626,105627],{"class":477},"4000",[151,105629,3634],{"class":503},[151,105631,9181],{"class":477},[151,105633,3634],{"class":503},[151,105635,80933],{"class":477},[151,105637,38820],{"class":503},[151,105639,105640,105642,105645,105647,105649,105651,105653],{"class":469,"line":3175},[151,105641,65133],{"class":503},[151,105643,105644],{"class":481},"'Memory Speed (MT/s)'",[151,105646,106],{"class":503},[151,105648,99065],{"class":15210},[151,105650,1876],{"class":1869},[151,105652,67140],{"class":477},[151,105654,3640],{"class":503},[151,105656,105657,105659,105662,105664,105666,105668,105670],{"class":469,"line":3184},[151,105658,65143],{"class":503},[151,105660,105661],{"class":481},"'Price per GB'",[151,105663,106],{"class":503},[151,105665,99065],{"class":15210},[151,105667,1876],{"class":1869},[151,105669,67140],{"class":477},[151,105671,3640],{"class":503},[151,105673,105674,105676,105679,105681,105683,105685,105687],{"class":469,"line":3193},[151,105675,65123],{"class":503},[151,105677,105678],{"class":481},"\"Memory Speed vs. Price\"",[151,105680,106],{"class":503},[151,105682,99065],{"class":15210},[151,105684,1876],{"class":1869},[151,105686,67140],{"class":477},[151,105688,3640],{"class":503},[151,105690,105691,105693,105696,105698,105701,105703,105706,105708,105710,105712,105715,105717,105719,105721,105723],{"class":469,"line":3720},[151,105692,102944],{"class":503},[151,105694,105695],{"class":481},"'DDR2'",[151,105697,106],{"class":503},[151,105699,105700],{"class":481},"'DDR3'",[151,105702,106],{"class":503},[151,105704,105705],{"class":481},"'DDR4'",[151,105707,60308],{"class":503},[151,105709,104219],{"class":15210},[151,105711,1876],{"class":1869},[151,105713,105714],{"class":481},"'lower right'",[151,105716,106],{"class":503},[151,105718,99065],{"class":15210},[151,105720,1876],{"class":1869},[151,105722,67140],{"class":477},[151,105724,3640],{"class":503},[151,105726,105727,105729,105731,105733,105735],{"class":469,"line":3729},[151,105728,65163],{"class":503},[151,105730,99065],{"class":15210},[151,105732,1876],{"class":1869},[151,105734,42327],{"class":477},[151,105736,3640],{"class":503},[151,105738,105739,105741,105743,105745,105747],{"class":469,"line":3735},[151,105740,99502],{"class":503},[151,105742,99065],{"class":15210},[151,105744,1876],{"class":1869},[151,105746,42327],{"class":477},[151,105748,3640],{"class":503},[151,105750,105751,105753,105756],{"class":469,"line":3745},[151,105752,93826],{"class":503},[151,105754,105755],{"class":481},"'~/Documents/GitHub/briancaffey.github.io/img/memory/speed_vs_price.png'",[151,105757,12451],{"class":503},[11,105759,105760],{},[2718,105761],{"alt":20386,"src":105762},"/static/pcpp/memory/speed_vs_price.png",[11,105764,105765,105766,13576],{},"CAS stands for column access strobe (CAS) latency, and it is the delay time between the moment a memory controller tells the memory module to access a particular memory column on a RAM module, and the moment the data from the given array location is available on the module's output pins (from the ",[20,105767,105770],{"href":105768,"rel":105769},"https://en.wikipedia.org/wiki/CAS_latency",[24],"Wikipedia article on CAS latency",[11,105772,105773],{},"On synchronous dynamic random-access memory modules like the ones used in modern PCs, CAS is measured in clock cycles and ranges from 4 to 20. Here's a look at memory speeds and their CAS latency:",[459,105775,105777],{"className":13136,"code":105776,"language":12886,"meta":464,"style":464},"plt.figure(figsize=(12,8))\ns = 100\na = 0.1\n\n#plot ddr3\nddr3_cas = df.CAS[(df.is_ddr4 == False)&(df.avg>0)]\nddr3_speed = df.ddr_speed[(df.is_ddr4 == False)&(df.avg>0)]\nplt.scatter(ddr3_cas,ddr3_speed, c = 'blue', s=s, alpha=a)\n\n#plot ddr4\nddr4_cas = df.CAS[(df.is_ddr4 == True)&(df.avg>0)]\nddr4_speed = df.ddr_speed[(df.is_ddr4 == True)&(df.avg>0)]\nplt.scatter(ddr4_cas,ddr4_speed, c = 'red', s = s, alpha=a)\n\nplt.xlabel('CAS', fontsize=14)\nplt.ylabel('Speed (MHz)', fontsize=14)\nplt.title(\"CAS vs. Clock Speed\", fontsize=14)\nplt.xticks(fontsize=13)\nplt.yticks(fontsize=13)\nplt.legend(['DDR3', 'DDR4'], loc='upper left', fontsize=13)\nplt.savefig(os.path.expanduser('~/Documents/GitHub/briancaffey.github.io/img/memory/cas_vs_speed.png'))\n",[30,105778,105779,105797,105805,105813,105817,105822,105854,105879,105907,105911,105916,105945,105969,105996,106000,106017,106034,106051,106063,106075,106103],{"__ignoreMap":464},[151,105780,105781,105783,105785,105787,105789,105791,105793,105795],{"class":469,"line":470},[151,105782,44355],{"class":503},[151,105784,44358],{"class":15210},[151,105786,1876],{"class":1869},[151,105788,12386],{"class":503},[151,105790,42360],{"class":477},[151,105792,3634],{"class":503},[151,105794,24369],{"class":477},[151,105796,12451],{"class":503},[151,105798,105799,105801,105803],{"class":469,"line":488},[151,105800,74751],{"class":503},[151,105802,1876],{"class":1869},[151,105804,59175],{"class":477},[151,105806,105807,105809,105811],{"class":469,"line":500},[151,105808,61268],{"class":503},[151,105810,1876],{"class":1869},[151,105812,24493],{"class":477},[151,105814,105815],{"class":469,"line":509},[151,105816,1090],{"emptyLinePlaceholder":609},[151,105818,105819],{"class":469,"line":517},[151,105820,105821],{"class":1527},"#plot ddr3\n",[151,105823,105824,105827,105829,105831,105834,105837,105839,105842,105844,105846,105848,105850,105852],{"class":469,"line":534},[151,105825,105826],{"class":503},"ddr3_cas ",[151,105828,1876],{"class":1869},[151,105830,102041],{"class":503},[151,105832,105833],{"class":477},"CAS",[151,105835,105836],{"class":503},"[(df.is_ddr4 ",[151,105838,17223],{"class":1869},[151,105840,105841],{"class":477}," False",[151,105843,748],{"class":503},[151,105845,54214],{"class":1869},[151,105847,100313],{"class":503},[151,105849,3663],{"class":1869},[151,105851,9181],{"class":477},[151,105853,44576],{"class":503},[151,105855,105856,105858,105860,105863,105865,105867,105869,105871,105873,105875,105877],{"class":469,"line":1413},[151,105857,105379],{"class":503},[151,105859,1876],{"class":1869},[151,105861,105862],{"class":503}," df.ddr_speed[(df.is_ddr4 ",[151,105864,17223],{"class":1869},[151,105866,105841],{"class":477},[151,105868,748],{"class":503},[151,105870,54214],{"class":1869},[151,105872,100313],{"class":503},[151,105874,3663],{"class":1869},[151,105876,9181],{"class":477},[151,105878,44576],{"class":503},[151,105880,105881,105884,105886,105888,105891,105893,105895,105897,105900,105902,105904],{"class":469,"line":1418},[151,105882,105883],{"class":503},"plt.scatter(ddr3_cas,ddr3_speed, ",[151,105885,65290],{"class":15210},[151,105887,19865],{"class":1869},[151,105889,105890],{"class":481}," 'blue'",[151,105892,106],{"class":503},[151,105894,55630],{"class":15210},[151,105896,1876],{"class":1869},[151,105898,105899],{"class":503},"s, ",[151,105901,26256],{"class":15210},[151,105903,1876],{"class":1869},[151,105905,105906],{"class":503},"a)\n",[151,105908,105909],{"class":469,"line":2462},[151,105910,1090],{"emptyLinePlaceholder":609},[151,105912,105913],{"class":469,"line":2471},[151,105914,105915],{"class":1527},"#plot ddr4\n",[151,105917,105918,105921,105923,105925,105927,105929,105931,105933,105935,105937,105939,105941,105943],{"class":469,"line":2480},[151,105919,105920],{"class":503},"ddr4_cas ",[151,105922,1876],{"class":1869},[151,105924,102041],{"class":503},[151,105926,105833],{"class":477},[151,105928,105836],{"class":503},[151,105930,17223],{"class":1869},[151,105932,68564],{"class":477},[151,105934,748],{"class":503},[151,105936,54214],{"class":1869},[151,105938,100313],{"class":503},[151,105940,3663],{"class":1869},[151,105942,9181],{"class":477},[151,105944,44576],{"class":503},[151,105946,105947,105949,105951,105953,105955,105957,105959,105961,105963,105965,105967],{"class":469,"line":2489},[151,105948,105500],{"class":503},[151,105950,1876],{"class":1869},[151,105952,105862],{"class":503},[151,105954,17223],{"class":1869},[151,105956,68564],{"class":477},[151,105958,748],{"class":503},[151,105960,54214],{"class":1869},[151,105962,100313],{"class":503},[151,105964,3663],{"class":1869},[151,105966,9181],{"class":477},[151,105968,44576],{"class":503},[151,105970,105971,105974,105976,105978,105981,105983,105985,105987,105990,105992,105994],{"class":469,"line":2497},[151,105972,105973],{"class":503},"plt.scatter(ddr4_cas,ddr4_speed, ",[151,105975,65290],{"class":15210},[151,105977,19865],{"class":1869},[151,105979,105980],{"class":481}," 'red'",[151,105982,106],{"class":503},[151,105984,55630],{"class":15210},[151,105986,19865],{"class":1869},[151,105988,105989],{"class":503}," s, ",[151,105991,26256],{"class":15210},[151,105993,1876],{"class":1869},[151,105995,105906],{"class":503},[151,105997,105998],{"class":469,"line":3140},[151,105999,1090],{"emptyLinePlaceholder":609},[151,106001,106002,106004,106007,106009,106011,106013,106015],{"class":469,"line":3149},[151,106003,65133],{"class":503},[151,106005,106006],{"class":481},"'CAS'",[151,106008,106],{"class":503},[151,106010,99065],{"class":15210},[151,106012,1876],{"class":1869},[151,106014,67140],{"class":477},[151,106016,3640],{"class":503},[151,106018,106019,106021,106024,106026,106028,106030,106032],{"class":469,"line":3158},[151,106020,65143],{"class":503},[151,106022,106023],{"class":481},"'Speed (MHz)'",[151,106025,106],{"class":503},[151,106027,99065],{"class":15210},[151,106029,1876],{"class":1869},[151,106031,67140],{"class":477},[151,106033,3640],{"class":503},[151,106035,106036,106038,106041,106043,106045,106047,106049],{"class":469,"line":3167},[151,106037,65123],{"class":503},[151,106039,106040],{"class":481},"\"CAS vs. Clock Speed\"",[151,106042,106],{"class":503},[151,106044,99065],{"class":15210},[151,106046,1876],{"class":1869},[151,106048,67140],{"class":477},[151,106050,3640],{"class":503},[151,106052,106053,106055,106057,106059,106061],{"class":469,"line":3175},[151,106054,65163],{"class":503},[151,106056,99065],{"class":15210},[151,106058,1876],{"class":1869},[151,106060,42327],{"class":477},[151,106062,3640],{"class":503},[151,106064,106065,106067,106069,106071,106073],{"class":469,"line":3184},[151,106066,99502],{"class":503},[151,106068,99065],{"class":15210},[151,106070,1876],{"class":1869},[151,106072,42327],{"class":477},[151,106074,3640],{"class":503},[151,106076,106077,106079,106081,106083,106085,106087,106089,106091,106093,106095,106097,106099,106101],{"class":469,"line":3193},[151,106078,102944],{"class":503},[151,106080,105700],{"class":481},[151,106082,106],{"class":503},[151,106084,105705],{"class":481},[151,106086,60308],{"class":503},[151,106088,104219],{"class":15210},[151,106090,1876],{"class":1869},[151,106092,99461],{"class":481},[151,106094,106],{"class":503},[151,106096,99065],{"class":15210},[151,106098,1876],{"class":1869},[151,106100,42327],{"class":477},[151,106102,3640],{"class":503},[151,106104,106105,106107,106110],{"class":469,"line":3720},[151,106106,93826],{"class":503},[151,106108,106109],{"class":481},"'~/Documents/GitHub/briancaffey.github.io/img/memory/cas_vs_speed.png'",[151,106111,12451],{"class":503},[11,106113,106114],{},[2718,106115],{"alt":20386,"src":106116},"/static/pcpp/memory/cas_vs_speed.png",[11,106118,106119],{},"It's a misconception that faster memory has higher latency, because CAS is not actually a good representation of memory's latency. This is because it is measured in clock cycles, which become smaller as the memory clock speed increases. Here's how the math for memory and true latency works out:",[459,106121,106124],{"className":106122,"code":106123,"language":997,"meta":464},[995],"true latency (nanoseconds) = clock cycle time (nanoseconds) x CAS (clock cycles)\n",[30,106125,106123],{"__ignoreMap":464},[11,106127,106128],{},"So here's how we can calculate true latency using the data in the dataset:",[459,106130,106133],{"className":106131,"code":106132,"language":997,"meta":464},[995],"#filter for DDR4 memory with valid CAS\ndf_lat = df[(df.CAS>0)&(df.ddr_speed>0)&(df.ddr_type=='DDR4')]\ndf_lat['true_latency'] = [((x/2.)/1000.)*y for x, y in zip(df_lat.ddr_speed, df_lat.CAS)]\n",[30,106134,106132],{"__ignoreMap":464},[11,106136,106137],{},"Let's calculate the true latency of two different memory modules: DDR4-2666/CAS18 and DDR3-1333/CAS9.",[11,106139,106140,106141,106144],{},"The 'DDR' in DDR memory stands for ",[51,106142,106143],{},"double data rate",", which means that information is transfared twice per clock cycle. Because of this, we must first take the MT/s rate and divide by 2 to calculate the clock speed. Next we need to find how long each clock cycle takes. To find this amount of time we can divide 1 by the frequency. Finally we multiply the time of one clock cycle by the CAS (the latency in number cycles) to get the latency in seconds (nanoseconds):",[459,106146,106149],{"className":106147,"code":106148,"language":997,"meta":464},[995],"DDR4-2666\n2666 MT/s\n2666000000 T/s\n1333000000 Hz\n1/(1333000000 Hz)\n0.00000000075 s\n0.75 nanoseconds = 1 clock cycle\n\ntrue latency = clock cycle x CAS\ntrue latency = 0.75 ns x 18\ntrue latency = 13.5 nanoseconds\n\nDDR3-1333\n1333 MT/s\n1333000000 T/s\n666500000 Hz\n1/(666500000 Hz)\n0.0000000015 seconds\n1.5 nanoseconds = 1 clock cycle\n\ntrue latency = clock cycle x CAS\ntrue latency = 1.5 ns x 9\ntrue latency = 13.5 nanoseconds\n",[30,106150,106148],{"__ignoreMap":464},[11,106152,106153,106154,106158],{},"Here's an ",[20,106155,20099],{"href":106156,"rel":106157},"http://www.crucial.com/usa/en/memory-performance-speed-latency",[24]," from Crucial, a major memory manufacturer, that explains this idea fully. The main conclusion is that speed is more important than memory, but the two measurements must be considered together.",[11,106160,106161,106162,106165],{},"The equation for ",[30,106163,106164],{},"true_latency"," above divides half the DDR MT/s rate by 1000 to both multiply by 10^6 (for converting MT/s to T/s) and divide by 10^9 (for converting seconds to nano seconds).",[11,106167,106168],{},"Here's a histogram of true latency for DDR4 memory modules:",[459,106170,106172],{"className":13136,"code":106171,"language":12886,"meta":464,"style":464},"df_lat = df[(df.CAS>0)&(df.ddr_speed>0)&((df.ddr_type=='DDR4')|(df.ddr_type=='DDR3'))]\ndf_lat['true_latency'] = [((x/2.)/1000.)*y for x, y in zip(df_lat.ddr_speed, df_lat.CAS)]\ndf_lat.true_latency.hist(bins=25, figsize=(12,8))\nplt.xlabel('True Latency (nanoseconds)', fontsize=14)\nplt.ylabel('Count', fontsize=14)\nplt.title(\"Histogram for True Latency of DDR3 and DDR4 Memory modules\", fontsize=14)\nplt.xticks(fontsize=13)\nplt.yticks(fontsize=13)\nplt.savefig(os.path.expanduser('~/Documents/GitHub/briancaffey.github.io/img/memory/true_latency.png'))\n",[30,106173,106174,106225,106272,106299,106316,106332,106349,106361,106373],{"__ignoreMap":464},[151,106175,106176,106179,106181,106184,106186,106188,106190,106192,106194,106196,106198,106200,106202,106204,106207,106209,106211,106213,106215,106218,106220,106222],{"class":469,"line":470},[151,106177,106178],{"class":503},"df_lat ",[151,106180,1876],{"class":1869},[151,106182,106183],{"class":503}," df[(df.",[151,106185,105833],{"class":477},[151,106187,3663],{"class":1869},[151,106189,9181],{"class":477},[151,106191,748],{"class":503},[151,106193,54214],{"class":1869},[151,106195,105281],{"class":503},[151,106197,3663],{"class":1869},[151,106199,9181],{"class":477},[151,106201,748],{"class":503},[151,106203,54214],{"class":1869},[151,106205,106206],{"class":503},"((df.ddr_type",[151,106208,17223],{"class":1869},[151,106210,105705],{"class":481},[151,106212,748],{"class":503},[151,106214,3947],{"class":1869},[151,106216,106217],{"class":503},"(df.ddr_type",[151,106219,17223],{"class":1869},[151,106221,105700],{"class":481},[151,106223,106224],{"class":503},"))]\n",[151,106226,106227,106230,106233,106235,106237,106240,106242,106244,106247,106249,106251,106253,106255,106257,106259,106261,106263,106265,106268,106270],{"class":469,"line":488},[151,106228,106229],{"class":503},"df_lat[",[151,106231,106232],{"class":481},"'true_latency'",[151,106234,16654],{"class":503},[151,106236,1876],{"class":1869},[151,106238,106239],{"class":503}," [((x",[151,106241,19883],{"class":1869},[151,106243,6619],{"class":477},[151,106245,106246],{"class":503},".)",[151,106248,19883],{"class":1869},[151,106250,45779],{"class":477},[151,106252,106246],{"class":503},[151,106254,23268],{"class":1869},[151,106256,98878],{"class":503},[151,106258,16732],{"class":1869},[151,106260,88158],{"class":503},[151,106262,16417],{"class":1869},[151,106264,44908],{"class":2226},[151,106266,106267],{"class":503},"(df_lat.ddr_speed, df_lat.",[151,106269,105833],{"class":477},[151,106271,44576],{"class":503},[151,106273,106274,106277,106279,106281,106283,106285,106287,106289,106291,106293,106295,106297],{"class":469,"line":500},[151,106275,106276],{"class":503},"df_lat.true_latency.hist(",[151,106278,87626],{"class":15210},[151,106280,1876],{"class":1869},[151,106282,80933],{"class":477},[151,106284,106],{"class":503},[151,106286,44358],{"class":15210},[151,106288,1876],{"class":1869},[151,106290,12386],{"class":503},[151,106292,42360],{"class":477},[151,106294,3634],{"class":503},[151,106296,24369],{"class":477},[151,106298,12451],{"class":503},[151,106300,106301,106303,106306,106308,106310,106312,106314],{"class":469,"line":509},[151,106302,65133],{"class":503},[151,106304,106305],{"class":481},"'True Latency (nanoseconds)'",[151,106307,106],{"class":503},[151,106309,99065],{"class":15210},[151,106311,1876],{"class":1869},[151,106313,67140],{"class":477},[151,106315,3640],{"class":503},[151,106317,106318,106320,106322,106324,106326,106328,106330],{"class":469,"line":517},[151,106319,65143],{"class":503},[151,106321,87648],{"class":481},[151,106323,106],{"class":503},[151,106325,99065],{"class":15210},[151,106327,1876],{"class":1869},[151,106329,67140],{"class":477},[151,106331,3640],{"class":503},[151,106333,106334,106336,106339,106341,106343,106345,106347],{"class":469,"line":534},[151,106335,65123],{"class":503},[151,106337,106338],{"class":481},"\"Histogram for True Latency of DDR3 and DDR4 Memory modules\"",[151,106340,106],{"class":503},[151,106342,99065],{"class":15210},[151,106344,1876],{"class":1869},[151,106346,67140],{"class":477},[151,106348,3640],{"class":503},[151,106350,106351,106353,106355,106357,106359],{"class":469,"line":1413},[151,106352,65163],{"class":503},[151,106354,99065],{"class":15210},[151,106356,1876],{"class":1869},[151,106358,42327],{"class":477},[151,106360,3640],{"class":503},[151,106362,106363,106365,106367,106369,106371],{"class":469,"line":1418},[151,106364,99502],{"class":503},[151,106366,99065],{"class":15210},[151,106368,1876],{"class":1869},[151,106370,42327],{"class":477},[151,106372,3640],{"class":503},[151,106374,106375,106377,106380],{"class":469,"line":2462},[151,106376,93826],{"class":503},[151,106378,106379],{"class":481},"'~/Documents/GitHub/briancaffey.github.io/img/memory/true_latency.png'",[151,106381,12451],{"class":503},[11,106383,106384],{},[2718,106385],{"alt":20386,"src":106386},"/static/pcpp/memory/true_latency.png",[11,106388,106389],{},"Here's an graph from the Crucial article showing speed vs true latency:",[11,106391,106392],{},[2718,106393],{"alt":20386,"src":106394},"/static/pcpp/memory/crucial_latency.png",[11,106396,106397],{},"And here is a similar graph from our dataset:",[459,106399,106401],{"className":13136,"code":106400,"language":12886,"meta":464,"style":464},"df_lat = df[(df.CAS>0)&(df.ddr_speed>0)&((df.ddr_type=='DDR4')|(df.ddr_type=='DDR3'))]\ndf_lat['true_latency'] = [((x/2.)/1000.)*y for x, y in zip(df_lat.ddr_speed, df_lat.CAS)]\nplt.figure(figsize=(12,8))\nplt.scatter(df_lat.ddr_speed, df_lat.true_latency)\nplt.xlabel('Memory Speed (MT/s)', fontsize=14)\nplt.ylabel('True Latency (nanoseconds)', fontsize=14)\nplt.title('Memory Speed vs. True Latency (nanoseconds) for DDR3 and DDR4 memory', fontsize=14)\nplt.xticks(fontsize=13)\nplt.yticks(fontsize=13)\nplt.savefig(os.path.expanduser('~/Documents/GitHub/briancaffey.github.io/img/memory/speed_vs_true_latency.png'))\n",[30,106402,106403,106449,106491,106509,106514,106530,106546,106563,106575,106587],{"__ignoreMap":464},[151,106404,106405,106407,106409,106411,106413,106415,106417,106419,106421,106423,106425,106427,106429,106431,106433,106435,106437,106439,106441,106443,106445,106447],{"class":469,"line":470},[151,106406,106178],{"class":503},[151,106408,1876],{"class":1869},[151,106410,106183],{"class":503},[151,106412,105833],{"class":477},[151,106414,3663],{"class":1869},[151,106416,9181],{"class":477},[151,106418,748],{"class":503},[151,106420,54214],{"class":1869},[151,106422,105281],{"class":503},[151,106424,3663],{"class":1869},[151,106426,9181],{"class":477},[151,106428,748],{"class":503},[151,106430,54214],{"class":1869},[151,106432,106206],{"class":503},[151,106434,17223],{"class":1869},[151,106436,105705],{"class":481},[151,106438,748],{"class":503},[151,106440,3947],{"class":1869},[151,106442,106217],{"class":503},[151,106444,17223],{"class":1869},[151,106446,105700],{"class":481},[151,106448,106224],{"class":503},[151,106450,106451,106453,106455,106457,106459,106461,106463,106465,106467,106469,106471,106473,106475,106477,106479,106481,106483,106485,106487,106489],{"class":469,"line":488},[151,106452,106229],{"class":503},[151,106454,106232],{"class":481},[151,106456,16654],{"class":503},[151,106458,1876],{"class":1869},[151,106460,106239],{"class":503},[151,106462,19883],{"class":1869},[151,106464,6619],{"class":477},[151,106466,106246],{"class":503},[151,106468,19883],{"class":1869},[151,106470,45779],{"class":477},[151,106472,106246],{"class":503},[151,106474,23268],{"class":1869},[151,106476,98878],{"class":503},[151,106478,16732],{"class":1869},[151,106480,88158],{"class":503},[151,106482,16417],{"class":1869},[151,106484,44908],{"class":2226},[151,106486,106267],{"class":503},[151,106488,105833],{"class":477},[151,106490,44576],{"class":503},[151,106492,106493,106495,106497,106499,106501,106503,106505,106507],{"class":469,"line":500},[151,106494,44355],{"class":503},[151,106496,44358],{"class":15210},[151,106498,1876],{"class":1869},[151,106500,12386],{"class":503},[151,106502,42360],{"class":477},[151,106504,3634],{"class":503},[151,106506,24369],{"class":477},[151,106508,12451],{"class":503},[151,106510,106511],{"class":469,"line":509},[151,106512,106513],{"class":503},"plt.scatter(df_lat.ddr_speed, df_lat.true_latency)\n",[151,106515,106516,106518,106520,106522,106524,106526,106528],{"class":469,"line":517},[151,106517,65133],{"class":503},[151,106519,105644],{"class":481},[151,106521,106],{"class":503},[151,106523,99065],{"class":15210},[151,106525,1876],{"class":1869},[151,106527,67140],{"class":477},[151,106529,3640],{"class":503},[151,106531,106532,106534,106536,106538,106540,106542,106544],{"class":469,"line":534},[151,106533,65143],{"class":503},[151,106535,106305],{"class":481},[151,106537,106],{"class":503},[151,106539,99065],{"class":15210},[151,106541,1876],{"class":1869},[151,106543,67140],{"class":477},[151,106545,3640],{"class":503},[151,106547,106548,106550,106553,106555,106557,106559,106561],{"class":469,"line":1413},[151,106549,65123],{"class":503},[151,106551,106552],{"class":481},"'Memory Speed vs. True Latency (nanoseconds) for DDR3 and DDR4 memory'",[151,106554,106],{"class":503},[151,106556,99065],{"class":15210},[151,106558,1876],{"class":1869},[151,106560,67140],{"class":477},[151,106562,3640],{"class":503},[151,106564,106565,106567,106569,106571,106573],{"class":469,"line":1418},[151,106566,65163],{"class":503},[151,106568,99065],{"class":15210},[151,106570,1876],{"class":1869},[151,106572,42327],{"class":477},[151,106574,3640],{"class":503},[151,106576,106577,106579,106581,106583,106585],{"class":469,"line":2462},[151,106578,99502],{"class":503},[151,106580,99065],{"class":15210},[151,106582,1876],{"class":1869},[151,106584,42327],{"class":477},[151,106586,3640],{"class":503},[151,106588,106589,106591,106594],{"class":469,"line":2471},[151,106590,93826],{"class":503},[151,106592,106593],{"class":481},"'~/Documents/GitHub/briancaffey.github.io/img/memory/speed_vs_true_latency.png'",[151,106595,12451],{"class":503},[11,106597,106598],{},[2718,106599],{"alt":20386,"src":106600},"/static/pcpp/memory/speed_vs_true_latency.png",[11,106602,106603],{},"Here's another look at memory prices, showing memory module prices and price per GB of memory:",[459,106605,106607],{"className":13136,"code":106606,"language":12886,"meta":464,"style":464},"plt.figure(figsize=(12,8))\ns_8 = df1[df1.size_gb==8]\ns_16 = df1[df1.size_gb==16]\ns_32 = df1[df1.size_gb==32]\ns_4 = df1[df1.size_gb==4]\ns_64 = df1[df1.size_gb==64]\ns_2 = df1[df1.size_gb==2]\n\nplt.scatter(s_2.avg, s_2.ppgb, c='orange', s=50)\nplt.scatter(s_4.avg, s_4.ppgb, c='red', s=50)\nplt.scatter(s_8.avg, s_8.ppgb, c='black', s=50)\nplt.scatter(s_16.avg, s_16.ppgb, c='green', s=50)\nplt.scatter(s_32.avg, s_32.ppgb, c='blue', s=50)\nplt.scatter(s_64.avg, s_64.ppgb, c='yellow', s=50)\n\n\nplt.axis([0,600,0,30])\nplt.title('Memory Kit Price vs. Price per GB', fontsize=14)\nplt.xlabel('Memory Kit Price', fontsize=14)\nplt.ylabel('Memory Kit Price per GB', fontsize=14)\nplt.xticks(fontsize=13)\nplt.yticks(fontsize=13)\nplt.legend(['2GB','4GB', '8GB', '16GB', '32GB', '64GB'], fontsize=14)\nplt.savefig(os.path.expanduser('~/Documents/GitHub/briancaffey.github.io/img/memory/price_vs_ppgb.png'))\n",[30,106608,106609,106627,106643,106658,106673,106688,106703,106718,106722,106744,106765,106786,106807,106828,106849,106853,106857,106877,106894,106911,106928,106940,106952,106994],{"__ignoreMap":464},[151,106610,106611,106613,106615,106617,106619,106621,106623,106625],{"class":469,"line":470},[151,106612,44355],{"class":503},[151,106614,44358],{"class":15210},[151,106616,1876],{"class":1869},[151,106618,12386],{"class":503},[151,106620,42360],{"class":477},[151,106622,3634],{"class":503},[151,106624,24369],{"class":477},[151,106626,12451],{"class":503},[151,106628,106629,106632,106634,106637,106639,106641],{"class":469,"line":488},[151,106630,106631],{"class":503},"s_8 ",[151,106633,1876],{"class":1869},[151,106635,106636],{"class":503}," df1[df1.size_gb",[151,106638,17223],{"class":1869},[151,106640,24369],{"class":477},[151,106642,3691],{"class":503},[151,106644,106645,106648,106650,106652,106654,106656],{"class":469,"line":500},[151,106646,106647],{"class":503},"s_16 ",[151,106649,1876],{"class":1869},[151,106651,106636],{"class":503},[151,106653,17223],{"class":1869},[151,106655,87061],{"class":477},[151,106657,3691],{"class":503},[151,106659,106660,106663,106665,106667,106669,106671],{"class":469,"line":509},[151,106661,106662],{"class":503},"s_32 ",[151,106664,1876],{"class":1869},[151,106666,106636],{"class":503},[151,106668,17223],{"class":1869},[151,106670,9302],{"class":477},[151,106672,3691],{"class":503},[151,106674,106675,106678,106680,106682,106684,106686],{"class":469,"line":517},[151,106676,106677],{"class":503},"s_4 ",[151,106679,1876],{"class":1869},[151,106681,106636],{"class":503},[151,106683,17223],{"class":1869},[151,106685,9187],{"class":477},[151,106687,3691],{"class":503},[151,106689,106690,106693,106695,106697,106699,106701],{"class":469,"line":534},[151,106691,106692],{"class":503},"s_64 ",[151,106694,1876],{"class":1869},[151,106696,106636],{"class":503},[151,106698,17223],{"class":1869},[151,106700,57832],{"class":477},[151,106702,3691],{"class":503},[151,106704,106705,106708,106710,106712,106714,106716],{"class":469,"line":1413},[151,106706,106707],{"class":503},"s_2 ",[151,106709,1876],{"class":1869},[151,106711,106636],{"class":503},[151,106713,17223],{"class":1869},[151,106715,6619],{"class":477},[151,106717,3691],{"class":503},[151,106719,106720],{"class":469,"line":1418},[151,106721,1090],{"emptyLinePlaceholder":609},[151,106723,106724,106727,106729,106731,106734,106736,106738,106740,106742],{"class":469,"line":2462},[151,106725,106726],{"class":503},"plt.scatter(s_2.avg, s_2.ppgb, ",[151,106728,65290],{"class":15210},[151,106730,1876],{"class":1869},[151,106732,106733],{"class":481},"'orange'",[151,106735,106],{"class":503},[151,106737,55630],{"class":15210},[151,106739,1876],{"class":1869},[151,106741,73146],{"class":477},[151,106743,3640],{"class":503},[151,106745,106746,106749,106751,106753,106755,106757,106759,106761,106763],{"class":469,"line":2471},[151,106747,106748],{"class":503},"plt.scatter(s_4.avg, s_4.ppgb, ",[151,106750,65290],{"class":15210},[151,106752,1876],{"class":1869},[151,106754,80832],{"class":481},[151,106756,106],{"class":503},[151,106758,55630],{"class":15210},[151,106760,1876],{"class":1869},[151,106762,73146],{"class":477},[151,106764,3640],{"class":503},[151,106766,106767,106770,106772,106774,106776,106778,106780,106782,106784],{"class":469,"line":2480},[151,106768,106769],{"class":503},"plt.scatter(s_8.avg, s_8.ppgb, ",[151,106771,65290],{"class":15210},[151,106773,1876],{"class":1869},[151,106775,45401],{"class":481},[151,106777,106],{"class":503},[151,106779,55630],{"class":15210},[151,106781,1876],{"class":1869},[151,106783,73146],{"class":477},[151,106785,3640],{"class":503},[151,106787,106788,106791,106793,106795,106797,106799,106801,106803,106805],{"class":469,"line":2489},[151,106789,106790],{"class":503},"plt.scatter(s_16.avg, s_16.ppgb, ",[151,106792,65290],{"class":15210},[151,106794,1876],{"class":1869},[151,106796,105604],{"class":481},[151,106798,106],{"class":503},[151,106800,55630],{"class":15210},[151,106802,1876],{"class":1869},[151,106804,73146],{"class":477},[151,106806,3640],{"class":503},[151,106808,106809,106812,106814,106816,106818,106820,106822,106824,106826],{"class":469,"line":2497},[151,106810,106811],{"class":503},"plt.scatter(s_32.avg, s_32.ppgb, ",[151,106813,65290],{"class":15210},[151,106815,1876],{"class":1869},[151,106817,102887],{"class":481},[151,106819,106],{"class":503},[151,106821,55630],{"class":15210},[151,106823,1876],{"class":1869},[151,106825,73146],{"class":477},[151,106827,3640],{"class":503},[151,106829,106830,106833,106835,106837,106839,106841,106843,106845,106847],{"class":469,"line":3140},[151,106831,106832],{"class":503},"plt.scatter(s_64.avg, s_64.ppgb, ",[151,106834,65290],{"class":15210},[151,106836,1876],{"class":1869},[151,106838,105483],{"class":481},[151,106840,106],{"class":503},[151,106842,55630],{"class":15210},[151,106844,1876],{"class":1869},[151,106846,73146],{"class":477},[151,106848,3640],{"class":503},[151,106850,106851],{"class":469,"line":3149},[151,106852,1090],{"emptyLinePlaceholder":609},[151,106854,106855],{"class":469,"line":3158},[151,106856,1090],{"emptyLinePlaceholder":609},[151,106858,106859,106861,106863,106865,106867,106869,106871,106873,106875],{"class":469,"line":3167},[151,106860,99036],{"class":503},[151,106862,9181],{"class":477},[151,106864,3634],{"class":503},[151,106866,44836],{"class":477},[151,106868,3634],{"class":503},[151,106870,9181],{"class":477},[151,106872,3634],{"class":503},[151,106874,42017],{"class":477},[151,106876,38820],{"class":503},[151,106878,106879,106881,106884,106886,106888,106890,106892],{"class":469,"line":3175},[151,106880,65123],{"class":503},[151,106882,106883],{"class":481},"'Memory Kit Price vs. Price per GB'",[151,106885,106],{"class":503},[151,106887,99065],{"class":15210},[151,106889,1876],{"class":1869},[151,106891,67140],{"class":477},[151,106893,3640],{"class":503},[151,106895,106896,106898,106901,106903,106905,106907,106909],{"class":469,"line":3184},[151,106897,65133],{"class":503},[151,106899,106900],{"class":481},"'Memory Kit Price'",[151,106902,106],{"class":503},[151,106904,99065],{"class":15210},[151,106906,1876],{"class":1869},[151,106908,67140],{"class":477},[151,106910,3640],{"class":503},[151,106912,106913,106915,106918,106920,106922,106924,106926],{"class":469,"line":3193},[151,106914,65143],{"class":503},[151,106916,106917],{"class":481},"'Memory Kit Price per GB'",[151,106919,106],{"class":503},[151,106921,99065],{"class":15210},[151,106923,1876],{"class":1869},[151,106925,67140],{"class":477},[151,106927,3640],{"class":503},[151,106929,106930,106932,106934,106936,106938],{"class":469,"line":3720},[151,106931,65163],{"class":503},[151,106933,99065],{"class":15210},[151,106935,1876],{"class":1869},[151,106937,42327],{"class":477},[151,106939,3640],{"class":503},[151,106941,106942,106944,106946,106948,106950],{"class":469,"line":3729},[151,106943,99502],{"class":503},[151,106945,99065],{"class":15210},[151,106947,1876],{"class":1869},[151,106949,42327],{"class":477},[151,106951,3640],{"class":503},[151,106953,106954,106956,106959,106961,106964,106966,106969,106971,106974,106976,106979,106981,106984,106986,106988,106990,106992],{"class":469,"line":3735},[151,106955,102944],{"class":503},[151,106957,106958],{"class":481},"'2GB'",[151,106960,3634],{"class":503},[151,106962,106963],{"class":481},"'4GB'",[151,106965,106],{"class":503},[151,106967,106968],{"class":481},"'8GB'",[151,106970,106],{"class":503},[151,106972,106973],{"class":481},"'16GB'",[151,106975,106],{"class":503},[151,106977,106978],{"class":481},"'32GB'",[151,106980,106],{"class":503},[151,106982,106983],{"class":481},"'64GB'",[151,106985,60308],{"class":503},[151,106987,99065],{"class":15210},[151,106989,1876],{"class":1869},[151,106991,67140],{"class":477},[151,106993,3640],{"class":503},[151,106995,106996,106998,107001],{"class":469,"line":3745},[151,106997,93826],{"class":503},[151,106999,107000],{"class":481},"'~/Documents/GitHub/briancaffey.github.io/img/memory/price_vs_ppgb.png'",[151,107002,12451],{"class":503},[11,107004,107005],{},[2718,107006],{"alt":20386,"src":107007},"/static/pcpp/memory/price_vs_ppgb.png",[11,107009,107010],{},"This graph shows the variation in price for the most popular memrory kit sizes, and it also shows the variation in pricing data. The price per GB corresponds to one price (which is usually the lowest listed price) and the price is the average of all vendors' prices.",[11,107012,107013],{},"One more thing to note about DDR3 and DDR4 is that DDR4 requires lower voltage:",[459,107015,107017],{"className":13136,"code":107016,"language":12886,"meta":464,"style":464},"df3 = df[(df.voltage>0)]\ndf.boxplot(column='voltage', by='ddr_type', figsize=(12, 8))\nplt.suptitle('')\nplt.title('Voltage for DDR2, DDR3 and DDR4', fontsize=14)\nplt.ylim(1,2)\nplt.xlabel('Memory Type', fontsize=14)\nplt.ylabel('Voltage', fontsize=14)\nplt.xticks(fontsize=13)\nplt.yticks(fontsize=13)\nplt.savefig(os.path.expanduser('~/Documents/GitHub/briancaffey.github.io/img/memory/voltage.png'))\n",[30,107018,107019,107035,107071,107079,107096,107108,107124,107141,107153,107165],{"__ignoreMap":464},[151,107020,107021,107024,107026,107029,107031,107033],{"class":469,"line":470},[151,107022,107023],{"class":503},"df3 ",[151,107025,1876],{"class":1869},[151,107027,107028],{"class":503}," df[(df.voltage",[151,107030,3663],{"class":1869},[151,107032,9181],{"class":477},[151,107034,44576],{"class":503},[151,107036,107037,107039,107041,107043,107046,107048,107050,107052,107055,107057,107059,107061,107063,107065,107067,107069],{"class":469,"line":488},[151,107038,103304],{"class":503},[151,107040,100697],{"class":15210},[151,107042,1876],{"class":1869},[151,107044,107045],{"class":481},"'voltage'",[151,107047,106],{"class":503},[151,107049,65808],{"class":15210},[151,107051,1876],{"class":1869},[151,107053,107054],{"class":481},"'ddr_type'",[151,107056,106],{"class":503},[151,107058,44358],{"class":15210},[151,107060,1876],{"class":1869},[151,107062,12386],{"class":503},[151,107064,42360],{"class":477},[151,107066,106],{"class":503},[151,107068,24369],{"class":477},[151,107070,12451],{"class":503},[151,107072,107073,107075,107077],{"class":469,"line":500},[151,107074,100745],{"class":503},[151,107076,2301],{"class":481},[151,107078,3640],{"class":503},[151,107080,107081,107083,107086,107088,107090,107092,107094],{"class":469,"line":509},[151,107082,65123],{"class":503},[151,107084,107085],{"class":481},"'Voltage for DDR2, DDR3 and DDR4'",[151,107087,106],{"class":503},[151,107089,99065],{"class":15210},[151,107091,1876],{"class":1869},[151,107093,67140],{"class":477},[151,107095,3640],{"class":503},[151,107097,107098,107100,107102,107104,107106],{"class":469,"line":517},[151,107099,103349],{"class":503},[151,107101,6760],{"class":477},[151,107103,3634],{"class":503},[151,107105,6619],{"class":477},[151,107107,3640],{"class":503},[151,107109,107110,107112,107114,107116,107118,107120,107122],{"class":469,"line":534},[151,107111,65133],{"class":503},[151,107113,101815],{"class":481},[151,107115,106],{"class":503},[151,107117,99065],{"class":15210},[151,107119,1876],{"class":1869},[151,107121,67140],{"class":477},[151,107123,3640],{"class":503},[151,107125,107126,107128,107131,107133,107135,107137,107139],{"class":469,"line":1413},[151,107127,65143],{"class":503},[151,107129,107130],{"class":481},"'Voltage'",[151,107132,106],{"class":503},[151,107134,99065],{"class":15210},[151,107136,1876],{"class":1869},[151,107138,67140],{"class":477},[151,107140,3640],{"class":503},[151,107142,107143,107145,107147,107149,107151],{"class":469,"line":1418},[151,107144,65163],{"class":503},[151,107146,99065],{"class":15210},[151,107148,1876],{"class":1869},[151,107150,42327],{"class":477},[151,107152,3640],{"class":503},[151,107154,107155,107157,107159,107161,107163],{"class":469,"line":2462},[151,107156,99502],{"class":503},[151,107158,99065],{"class":15210},[151,107160,1876],{"class":1869},[151,107162,42327],{"class":477},[151,107164,3640],{"class":503},[151,107166,107167,107169,107172],{"class":469,"line":2471},[151,107168,93826],{"class":503},[151,107170,107171],{"class":481},"'~/Documents/GitHub/briancaffey.github.io/img/memory/voltage.png'",[151,107173,12451],{"class":503},[11,107175,107176],{},[2718,107177],{"alt":20386,"src":107178},"/static/pcpp/memory/voltage.png",[14063,107180,107182],{"id":107181},"video-card-graphics-card-gpu","Video Card / Graphics Card / GPU",[11,107184,107185],{},"Video cards are often the single biggest expense for high-end PCs. They deliver parallel computing performance that is necessary for modern applications like 4K gaming, virtual reality and deep learning. Like the CPU market, the GPU market is dominated by two major players: NVIDIA and AMD. These companies produce graphical processing units, but there are a number of vendors who sell graphics cards using the core chipsets provided by NVIDIA and AMD (and these two companies also sell their own consumer products).",[11,107187,107188],{},"GPUs are attached to the motherboard by PC expansion slots as well as the rear of the case where their display connections are exposed. One GPU metric is the number of DisplayPort type connections. GPUs that support multi-screen setups have higher performance and therefor higher cost, and in general the more screens it can support the more costly the card will be. Here's a breakdown of prices by the number of DisplayPort connections:",[459,107190,107192],{"className":13136,"code":107191,"language":12886,"meta":464,"style":464},"df['DisplayPort_count'] = df['DisplayPort'].fillna(0)\ndf[(df.avg>0)].boxplot(column='avg', by='DisplayPort_count', figsize=(12,8))\nplt.ylim(0,2000)\nplt.suptitle('')\nplt.title('GPU Prices by Display Port Connections', fontsize=14)\nplt.xlabel('DisplayPort Connections', fontsize=14)\nplt.ylabel('Prices', fontsize=14)\nplt.xticks(fontsize=13)\nplt.yticks(fontsize=13)\nplt.savefig(os.path.expanduser('~/Documents/GitHub/briancaffey.github.io/img/gpu/prices_by_display.png'))\n",[30,107193,107194,107217,107257,107270,107278,107295,107312,107328,107340,107352],{"__ignoreMap":464},[151,107195,107196,107198,107201,107203,107205,107207,107210,107213,107215],{"class":469,"line":470},[151,107197,70736],{"class":503},[151,107199,107200],{"class":481},"'DisplayPort_count'",[151,107202,16654],{"class":503},[151,107204,1876],{"class":1869},[151,107206,70760],{"class":503},[151,107208,107209],{"class":481},"'DisplayPort'",[151,107211,107212],{"class":503},"].fillna(",[151,107214,9181],{"class":477},[151,107216,3640],{"class":503},[151,107218,107219,107221,107223,107225,107227,107229,107231,107233,107235,107237,107239,107241,107243,107245,107247,107249,107251,107253,107255],{"class":469,"line":488},[151,107220,100850],{"class":503},[151,107222,3663],{"class":1869},[151,107224,9181],{"class":477},[151,107226,100694],{"class":503},[151,107228,100697],{"class":15210},[151,107230,1876],{"class":1869},[151,107232,99593],{"class":481},[151,107234,106],{"class":503},[151,107236,65808],{"class":15210},[151,107238,1876],{"class":1869},[151,107240,107200],{"class":481},[151,107242,106],{"class":503},[151,107244,44358],{"class":15210},[151,107246,1876],{"class":1869},[151,107248,12386],{"class":503},[151,107250,42360],{"class":477},[151,107252,3634],{"class":503},[151,107254,24369],{"class":477},[151,107256,12451],{"class":503},[151,107258,107259,107261,107263,107265,107268],{"class":469,"line":500},[151,107260,103349],{"class":503},[151,107262,9181],{"class":477},[151,107264,3634],{"class":503},[151,107266,107267],{"class":477},"2000",[151,107269,3640],{"class":503},[151,107271,107272,107274,107276],{"class":469,"line":509},[151,107273,100745],{"class":503},[151,107275,2301],{"class":481},[151,107277,3640],{"class":503},[151,107279,107280,107282,107285,107287,107289,107291,107293],{"class":469,"line":517},[151,107281,65123],{"class":503},[151,107283,107284],{"class":481},"'GPU Prices by Display Port Connections'",[151,107286,106],{"class":503},[151,107288,99065],{"class":15210},[151,107290,1876],{"class":1869},[151,107292,67140],{"class":477},[151,107294,3640],{"class":503},[151,107296,107297,107299,107302,107304,107306,107308,107310],{"class":469,"line":534},[151,107298,65133],{"class":503},[151,107300,107301],{"class":481},"'DisplayPort Connections'",[151,107303,106],{"class":503},[151,107305,99065],{"class":15210},[151,107307,1876],{"class":1869},[151,107309,67140],{"class":477},[151,107311,3640],{"class":503},[151,107313,107314,107316,107318,107320,107322,107324,107326],{"class":469,"line":1413},[151,107315,65143],{"class":503},[151,107317,98126],{"class":481},[151,107319,106],{"class":503},[151,107321,99065],{"class":15210},[151,107323,1876],{"class":1869},[151,107325,67140],{"class":477},[151,107327,3640],{"class":503},[151,107329,107330,107332,107334,107336,107338],{"class":469,"line":1418},[151,107331,65163],{"class":503},[151,107333,99065],{"class":15210},[151,107335,1876],{"class":1869},[151,107337,42327],{"class":477},[151,107339,3640],{"class":503},[151,107341,107342,107344,107346,107348,107350],{"class":469,"line":2462},[151,107343,99502],{"class":503},[151,107345,99065],{"class":15210},[151,107347,1876],{"class":1869},[151,107349,42327],{"class":477},[151,107351,3640],{"class":503},[151,107353,107354,107356,107359],{"class":469,"line":2471},[151,107355,93826],{"class":503},[151,107357,107358],{"class":481},"'~/Documents/GitHub/briancaffey.github.io/img/gpu/prices_by_display.png'",[151,107360,12451],{"class":503},[11,107362,107363],{},[2718,107364],{"alt":20386,"src":107365},"/static/pcpp/gpu/prices_by_display.png",[11,107367,107368],{},"GPUs also sometimes the largest components, here's a histogram for the height of all GPUs:",[459,107370,107372],{"className":13136,"code":107371,"language":12886,"meta":464,"style":464},"plt.figure(figsize=(12,8))\ndf[df.gpu_length>0].gpu_length.hist(bins=25)\nplt.title('GPU Length', fontsize=14)\nplt.xlabel('Length (inches)', fontsize=14)\nplt.ylabel('Count', fontsize=14)\nplt.xticks(fontsize=13)\nplt.yticks(fontsize=13)\nplt.savefig(os.path.expanduser('~/Documents/GitHub/briancaffey.github.io/img/gpu/length_hist.png'))\n",[30,107373,107374,107392,107412,107429,107446,107462,107474,107486],{"__ignoreMap":464},[151,107375,107376,107378,107380,107382,107384,107386,107388,107390],{"class":469,"line":470},[151,107377,44355],{"class":503},[151,107379,44358],{"class":15210},[151,107381,1876],{"class":1869},[151,107383,12386],{"class":503},[151,107385,42360],{"class":477},[151,107387,3634],{"class":503},[151,107389,24369],{"class":477},[151,107391,12451],{"class":503},[151,107393,107394,107397,107399,107401,107404,107406,107408,107410],{"class":469,"line":488},[151,107395,107396],{"class":503},"df[df.gpu_length",[151,107398,3663],{"class":1869},[151,107400,9181],{"class":477},[151,107402,107403],{"class":503},"].gpu_length.hist(",[151,107405,87626],{"class":15210},[151,107407,1876],{"class":1869},[151,107409,80933],{"class":477},[151,107411,3640],{"class":503},[151,107413,107414,107416,107419,107421,107423,107425,107427],{"class":469,"line":500},[151,107415,65123],{"class":503},[151,107417,107418],{"class":481},"'GPU Length'",[151,107420,106],{"class":503},[151,107422,99065],{"class":15210},[151,107424,1876],{"class":1869},[151,107426,67140],{"class":477},[151,107428,3640],{"class":503},[151,107430,107431,107433,107436,107438,107440,107442,107444],{"class":469,"line":509},[151,107432,65133],{"class":503},[151,107434,107435],{"class":481},"'Length (inches)'",[151,107437,106],{"class":503},[151,107439,99065],{"class":15210},[151,107441,1876],{"class":1869},[151,107443,67140],{"class":477},[151,107445,3640],{"class":503},[151,107447,107448,107450,107452,107454,107456,107458,107460],{"class":469,"line":517},[151,107449,65143],{"class":503},[151,107451,87648],{"class":481},[151,107453,106],{"class":503},[151,107455,99065],{"class":15210},[151,107457,1876],{"class":1869},[151,107459,67140],{"class":477},[151,107461,3640],{"class":503},[151,107463,107464,107466,107468,107470,107472],{"class":469,"line":534},[151,107465,65163],{"class":503},[151,107467,99065],{"class":15210},[151,107469,1876],{"class":1869},[151,107471,42327],{"class":477},[151,107473,3640],{"class":503},[151,107475,107476,107478,107480,107482,107484],{"class":469,"line":1413},[151,107477,99502],{"class":503},[151,107479,99065],{"class":15210},[151,107481,1876],{"class":1869},[151,107483,42327],{"class":477},[151,107485,3640],{"class":503},[151,107487,107488,107490,107493],{"class":469,"line":1418},[151,107489,93826],{"class":503},[151,107491,107492],{"class":481},"'~/Documents/GitHub/briancaffey.github.io/img/gpu/length_hist.png'",[151,107494,12451],{"class":503},[11,107496,107497],{},[2718,107498],{"alt":20386,"src":107499},"/static/pcpp/gpu/length_hist.png",[11,107501,107502],{},"Length is an interesting feature, because you can pack more GPUs into a graphics card that is larger, and you can also have more fans and cooling equipment on larger cards. Here's a look at the relationship between video card length, price, memory and clock speed:",[459,107504,107506],{"className":13136,"code":107505,"language":12886,"meta":464,"style":464},"plt.figure(figsize=(12,8))\ndf_len = df[(df.avg>0)&(df.gpu_length>0)]\nplt.scatter(df_len.gpu_length, df_len.avg, c=df_len.memory_mb, cmap='CMRmap', s=df_len.clock_speed_in_mhz/4)\nplt.colorbar(label='Memory (MB)')\nplt.axis([5,14,0,3300])\nplt.title('GPU Length vs. Price, Memory (color) and Clockspeed (diameter)', fontsize=14)\nplt.xlabel('Graphics Card Length (inches)', fontsize=14)\nplt.ylabel('Price', fontsize=14)\nplt.xticks(fontsize=13)\nplt.yticks(fontsize=13)\nplt.savefig(os.path.expanduser('~/Documents/GitHub/briancaffey.github.io/img/gpu/length_vs_price.png'))\n",[30,107507,107508,107526,107552,107586,107599,107620,107637,107654,107670,107682,107694],{"__ignoreMap":464},[151,107509,107510,107512,107514,107516,107518,107520,107522,107524],{"class":469,"line":470},[151,107511,44355],{"class":503},[151,107513,44358],{"class":15210},[151,107515,1876],{"class":1869},[151,107517,12386],{"class":503},[151,107519,42360],{"class":477},[151,107521,3634],{"class":503},[151,107523,24369],{"class":477},[151,107525,12451],{"class":503},[151,107527,107528,107531,107533,107535,107537,107539,107541,107543,107546,107548,107550],{"class":469,"line":488},[151,107529,107530],{"class":503},"df_len ",[151,107532,1876],{"class":1869},[151,107534,100420],{"class":503},[151,107536,3663],{"class":1869},[151,107538,9181],{"class":477},[151,107540,748],{"class":503},[151,107542,54214],{"class":1869},[151,107544,107545],{"class":503},"(df.gpu_length",[151,107547,3663],{"class":1869},[151,107549,9181],{"class":477},[151,107551,44576],{"class":503},[151,107553,107554,107557,107559,107561,107564,107566,107568,107571,107573,107575,107577,107580,107582,107584],{"class":469,"line":500},[151,107555,107556],{"class":503},"plt.scatter(df_len.gpu_length, df_len.avg, ",[151,107558,65290],{"class":15210},[151,107560,1876],{"class":1869},[151,107562,107563],{"class":503},"df_len.memory_mb, ",[151,107565,103551],{"class":15210},[151,107567,1876],{"class":1869},[151,107569,107570],{"class":481},"'CMRmap'",[151,107572,106],{"class":503},[151,107574,55630],{"class":15210},[151,107576,1876],{"class":1869},[151,107578,107579],{"class":503},"df_len.clock_speed_in_mhz",[151,107581,19883],{"class":1869},[151,107583,9187],{"class":477},[151,107585,3640],{"class":503},[151,107587,107588,107590,107592,107594,107597],{"class":469,"line":509},[151,107589,103563],{"class":503},[151,107591,103566],{"class":15210},[151,107593,1876],{"class":1869},[151,107595,107596],{"class":481},"'Memory (MB)'",[151,107598,3640],{"class":503},[151,107600,107601,107603,107605,107607,107609,107611,107613,107615,107618],{"class":469,"line":517},[151,107602,99036],{"class":503},[151,107604,24380],{"class":477},[151,107606,3634],{"class":503},[151,107608,67140],{"class":477},[151,107610,3634],{"class":503},[151,107612,9181],{"class":477},[151,107614,3634],{"class":503},[151,107616,107617],{"class":477},"3300",[151,107619,38820],{"class":503},[151,107621,107622,107624,107627,107629,107631,107633,107635],{"class":469,"line":534},[151,107623,65123],{"class":503},[151,107625,107626],{"class":481},"'GPU Length vs. Price, Memory (color) and Clockspeed (diameter)'",[151,107628,106],{"class":503},[151,107630,99065],{"class":15210},[151,107632,1876],{"class":1869},[151,107634,67140],{"class":477},[151,107636,3640],{"class":503},[151,107638,107639,107641,107644,107646,107648,107650,107652],{"class":469,"line":1413},[151,107640,65133],{"class":503},[151,107642,107643],{"class":481},"'Graphics Card Length (inches)'",[151,107645,106],{"class":503},[151,107647,99065],{"class":15210},[151,107649,1876],{"class":1869},[151,107651,67140],{"class":477},[151,107653,3640],{"class":503},[151,107655,107656,107658,107660,107662,107664,107666,107668],{"class":469,"line":1418},[151,107657,65143],{"class":503},[151,107659,99095],{"class":481},[151,107661,106],{"class":503},[151,107663,99065],{"class":15210},[151,107665,1876],{"class":1869},[151,107667,67140],{"class":477},[151,107669,3640],{"class":503},[151,107671,107672,107674,107676,107678,107680],{"class":469,"line":2462},[151,107673,65163],{"class":503},[151,107675,99065],{"class":15210},[151,107677,1876],{"class":1869},[151,107679,42327],{"class":477},[151,107681,3640],{"class":503},[151,107683,107684,107686,107688,107690,107692],{"class":469,"line":2471},[151,107685,99502],{"class":503},[151,107687,99065],{"class":15210},[151,107689,1876],{"class":1869},[151,107691,42327],{"class":477},[151,107693,3640],{"class":503},[151,107695,107696,107698,107701],{"class":469,"line":2480},[151,107697,93826],{"class":503},[151,107699,107700],{"class":481},"'~/Documents/GitHub/briancaffey.github.io/img/gpu/length_vs_price.png'",[151,107702,12451],{"class":503},[11,107704,107705],{},[2718,107706],{"alt":20386,"src":107707},"/static/pcpp/gpu/length_vs_price.png",[11,107709,107710],{},"And here is another look at the same data, filtered for graphics cards under $1,100:",[459,107712,107714],{"className":13136,"code":107713,"language":12886,"meta":464,"style":464},"plt.figure(figsize=(12,8))\ndf_len = df[(df.avg>0)&(df.gpu_length>0)&(df.memory_mb\u003C12000)]\nplt.scatter(df_len.gpu_length, df_len.avg, c=df_len.memory_mb, cmap='CMRmap', s=df_len.clock_speed_in_mhz/4)\nplt.colorbar(label='Memory (MB)')\nplt.axis([5,14,0,1100])\nplt.title('GPU Length vs. Price, Memory and Clockspeed (diameter)', fontsize=14)\nplt.xlabel('Graphics Card Length (inches)', fontsize=14)\nplt.ylabel('Price', fontsize=14)\nplt.xticks(fontsize=13)\nplt.yticks(fontsize=13)\nplt.savefig(os.path.expanduser('~/Documents/GitHub/briancaffey.github.io/img/gpu/length_vs_price_2.png'))\n",[30,107715,107716,107734,107770,107800,107812,107832,107849,107865,107881,107893,107905],{"__ignoreMap":464},[151,107717,107718,107720,107722,107724,107726,107728,107730,107732],{"class":469,"line":470},[151,107719,44355],{"class":503},[151,107721,44358],{"class":15210},[151,107723,1876],{"class":1869},[151,107725,12386],{"class":503},[151,107727,42360],{"class":477},[151,107729,3634],{"class":503},[151,107731,24369],{"class":477},[151,107733,12451],{"class":503},[151,107735,107736,107738,107740,107742,107744,107746,107748,107750,107752,107754,107756,107758,107760,107763,107765,107768],{"class":469,"line":488},[151,107737,107530],{"class":503},[151,107739,1876],{"class":1869},[151,107741,100420],{"class":503},[151,107743,3663],{"class":1869},[151,107745,9181],{"class":477},[151,107747,748],{"class":503},[151,107749,54214],{"class":1869},[151,107751,107545],{"class":503},[151,107753,3663],{"class":1869},[151,107755,9181],{"class":477},[151,107757,748],{"class":503},[151,107759,54214],{"class":1869},[151,107761,107762],{"class":503},"(df.memory_mb",[151,107764,3613],{"class":1869},[151,107766,107767],{"class":477},"12000",[151,107769,44576],{"class":503},[151,107771,107772,107774,107776,107778,107780,107782,107784,107786,107788,107790,107792,107794,107796,107798],{"class":469,"line":500},[151,107773,107556],{"class":503},[151,107775,65290],{"class":15210},[151,107777,1876],{"class":1869},[151,107779,107563],{"class":503},[151,107781,103551],{"class":15210},[151,107783,1876],{"class":1869},[151,107785,107570],{"class":481},[151,107787,106],{"class":503},[151,107789,55630],{"class":15210},[151,107791,1876],{"class":1869},[151,107793,107579],{"class":503},[151,107795,19883],{"class":1869},[151,107797,9187],{"class":477},[151,107799,3640],{"class":503},[151,107801,107802,107804,107806,107808,107810],{"class":469,"line":509},[151,107803,103563],{"class":503},[151,107805,103566],{"class":15210},[151,107807,1876],{"class":1869},[151,107809,107596],{"class":481},[151,107811,3640],{"class":503},[151,107813,107814,107816,107818,107820,107822,107824,107826,107828,107830],{"class":469,"line":517},[151,107815,99036],{"class":503},[151,107817,24380],{"class":477},[151,107819,3634],{"class":503},[151,107821,67140],{"class":477},[151,107823,3634],{"class":503},[151,107825,9181],{"class":477},[151,107827,3634],{"class":503},[151,107829,105073],{"class":477},[151,107831,38820],{"class":503},[151,107833,107834,107836,107839,107841,107843,107845,107847],{"class":469,"line":534},[151,107835,65123],{"class":503},[151,107837,107838],{"class":481},"'GPU Length vs. Price, Memory and Clockspeed (diameter)'",[151,107840,106],{"class":503},[151,107842,99065],{"class":15210},[151,107844,1876],{"class":1869},[151,107846,67140],{"class":477},[151,107848,3640],{"class":503},[151,107850,107851,107853,107855,107857,107859,107861,107863],{"class":469,"line":1413},[151,107852,65133],{"class":503},[151,107854,107643],{"class":481},[151,107856,106],{"class":503},[151,107858,99065],{"class":15210},[151,107860,1876],{"class":1869},[151,107862,67140],{"class":477},[151,107864,3640],{"class":503},[151,107866,107867,107869,107871,107873,107875,107877,107879],{"class":469,"line":1418},[151,107868,65143],{"class":503},[151,107870,99095],{"class":481},[151,107872,106],{"class":503},[151,107874,99065],{"class":15210},[151,107876,1876],{"class":1869},[151,107878,67140],{"class":477},[151,107880,3640],{"class":503},[151,107882,107883,107885,107887,107889,107891],{"class":469,"line":2462},[151,107884,65163],{"class":503},[151,107886,99065],{"class":15210},[151,107888,1876],{"class":1869},[151,107890,42327],{"class":477},[151,107892,3640],{"class":503},[151,107894,107895,107897,107899,107901,107903],{"class":469,"line":2471},[151,107896,99502],{"class":503},[151,107898,99065],{"class":15210},[151,107900,1876],{"class":1869},[151,107902,42327],{"class":477},[151,107904,3640],{"class":503},[151,107906,107907,107909,107912],{"class":469,"line":2480},[151,107908,93826],{"class":503},[151,107910,107911],{"class":481},"'~/Documents/GitHub/briancaffey.github.io/img/gpu/length_vs_price_2.png'",[151,107913,12451],{"class":503},[11,107915,107916],{},[2718,107917],{"alt":20386,"src":107918},"/static/pcpp/gpu/length_vs_price_2.png",[11,107920,107921],{},"GPUs are also measured in thermal design power (TDP) that we previously examined in CPUs. This scatterplot shows the relationship between TDP, Price, clockspeed and memory:",[459,107923,107925],{"className":13136,"code":107924,"language":12886,"meta":464,"style":464},"df1 = df[(df.avg>0)&(df.clock_speed_in_mhz>0)]\nplt.figure(figsize=(12,8))\nplt.scatter(df1.tdp, df1.avg, c=df1.clock_speed_in_mhz, s=df1.memory_mb/10, cmap='Blues')\nplt.colorbar(label='Clock Speed (MHz)')\nplt.axis([0,350,0,1000])\nplt.title('GPU TDP vs. Price, Clock Speed (color) and Memory (diameter)', fontsize=14)\nplt.xlabel('Thermal Design Power (TDP)', fontsize=14)\nplt.ylabel('Price', fontsize=14)\nplt.xticks(fontsize=13)\nplt.yticks(fontsize=13)\nplt.savefig(os.path.expanduser('~/Documents/GitHub/briancaffey.github.io/img/gpu/tdp_vs_price.png'))\n",[30,107926,107927,107952,107970,108003,108016,108036,108053,108070,108086,108098,108110],{"__ignoreMap":464},[151,107928,107929,107931,107933,107935,107937,107939,107941,107943,107946,107948,107950],{"class":469,"line":470},[151,107930,86777],{"class":503},[151,107932,1876],{"class":1869},[151,107934,100420],{"class":503},[151,107936,3663],{"class":1869},[151,107938,9181],{"class":477},[151,107940,748],{"class":503},[151,107942,54214],{"class":1869},[151,107944,107945],{"class":503},"(df.clock_speed_in_mhz",[151,107947,3663],{"class":1869},[151,107949,9181],{"class":477},[151,107951,44576],{"class":503},[151,107953,107954,107956,107958,107960,107962,107964,107966,107968],{"class":469,"line":488},[151,107955,44355],{"class":503},[151,107957,44358],{"class":15210},[151,107959,1876],{"class":1869},[151,107961,12386],{"class":503},[151,107963,42360],{"class":477},[151,107965,3634],{"class":503},[151,107967,24369],{"class":477},[151,107969,12451],{"class":503},[151,107971,107972,107975,107977,107979,107982,107984,107986,107989,107991,107993,107995,107997,107999,108001],{"class":469,"line":500},[151,107973,107974],{"class":503},"plt.scatter(df1.tdp, df1.avg, ",[151,107976,65290],{"class":15210},[151,107978,1876],{"class":1869},[151,107980,107981],{"class":503},"df1.clock_speed_in_mhz, ",[151,107983,55630],{"class":15210},[151,107985,1876],{"class":1869},[151,107987,107988],{"class":503},"df1.memory_mb",[151,107990,19883],{"class":1869},[151,107992,12423],{"class":477},[151,107994,106],{"class":503},[151,107996,103551],{"class":15210},[151,107998,1876],{"class":1869},[151,108000,103556],{"class":481},[151,108002,3640],{"class":503},[151,108004,108005,108007,108009,108011,108014],{"class":469,"line":509},[151,108006,103563],{"class":503},[151,108008,103566],{"class":15210},[151,108010,1876],{"class":1869},[151,108012,108013],{"class":481},"'Clock Speed (MHz)'",[151,108015,3640],{"class":503},[151,108017,108018,108020,108022,108024,108026,108028,108030,108032,108034],{"class":469,"line":517},[151,108019,99036],{"class":503},[151,108021,9181],{"class":477},[151,108023,3634],{"class":503},[151,108025,73574],{"class":477},[151,108027,3634],{"class":503},[151,108029,9181],{"class":477},[151,108031,3634],{"class":503},[151,108033,45779],{"class":477},[151,108035,38820],{"class":503},[151,108037,108038,108040,108043,108045,108047,108049,108051],{"class":469,"line":534},[151,108039,65123],{"class":503},[151,108041,108042],{"class":481},"'GPU TDP vs. Price, Clock Speed (color) and Memory (diameter)'",[151,108044,106],{"class":503},[151,108046,99065],{"class":15210},[151,108048,1876],{"class":1869},[151,108050,67140],{"class":477},[151,108052,3640],{"class":503},[151,108054,108055,108057,108060,108062,108064,108066,108068],{"class":469,"line":1413},[151,108056,65133],{"class":503},[151,108058,108059],{"class":481},"'Thermal Design Power (TDP)'",[151,108061,106],{"class":503},[151,108063,99065],{"class":15210},[151,108065,1876],{"class":1869},[151,108067,67140],{"class":477},[151,108069,3640],{"class":503},[151,108071,108072,108074,108076,108078,108080,108082,108084],{"class":469,"line":1418},[151,108073,65143],{"class":503},[151,108075,99095],{"class":481},[151,108077,106],{"class":503},[151,108079,99065],{"class":15210},[151,108081,1876],{"class":1869},[151,108083,67140],{"class":477},[151,108085,3640],{"class":503},[151,108087,108088,108090,108092,108094,108096],{"class":469,"line":2462},[151,108089,65163],{"class":503},[151,108091,99065],{"class":15210},[151,108093,1876],{"class":1869},[151,108095,42327],{"class":477},[151,108097,3640],{"class":503},[151,108099,108100,108102,108104,108106,108108],{"class":469,"line":2471},[151,108101,99502],{"class":503},[151,108103,99065],{"class":15210},[151,108105,1876],{"class":1869},[151,108107,42327],{"class":477},[151,108109,3640],{"class":503},[151,108111,108112,108114,108117],{"class":469,"line":2480},[151,108113,93826],{"class":503},[151,108115,108116],{"class":481},"'~/Documents/GitHub/briancaffey.github.io/img/gpu/tdp_vs_price.png'",[151,108118,12451],{"class":503},[11,108120,108121],{},[2718,108122],{"alt":20386,"src":108123},"/static/pcpp/gpu/tdp_vs_price.png",[11,108125,108126],{},"Here is a look at the top 20 most common GPU chipsets for graphics cards:",[459,108128,108130],{"className":13136,"code":108129,"language":12886,"meta":464,"style":464},"df1 = df[(df.avg>0)]\nplt.figure(figsize=(12,8))\ndf1.groupby('Chipset').avg.count().sort_values( ascending=False)[:30].plot(kind='bar')\nplt.title('GPUs by Chipset', fontsize=14)\nplt.xlabel('Chipset', fontsize=14)\nplt.ylabel('Count', fontsize=14)\nplt.xticks(fontsize=13)\nplt.yticks(fontsize=13)\nplt.savefig(os.path.expanduser('~/Documents/GitHub/briancaffey.github.io/img/gpu/gpu_chipsets.png'))\n",[30,108131,108132,108146,108164,108195,108212,108228,108244,108256,108268],{"__ignoreMap":464},[151,108133,108134,108136,108138,108140,108142,108144],{"class":469,"line":470},[151,108135,86777],{"class":503},[151,108137,1876],{"class":1869},[151,108139,100420],{"class":503},[151,108141,3663],{"class":1869},[151,108143,9181],{"class":477},[151,108145,44576],{"class":503},[151,108147,108148,108150,108152,108154,108156,108158,108160,108162],{"class":469,"line":488},[151,108149,44355],{"class":503},[151,108151,44358],{"class":15210},[151,108153,1876],{"class":1869},[151,108155,12386],{"class":503},[151,108157,42360],{"class":477},[151,108159,3634],{"class":503},[151,108161,24369],{"class":477},[151,108163,12451],{"class":503},[151,108165,108166,108169,108172,108175,108177,108179,108181,108183,108185,108187,108189,108191,108193],{"class":469,"line":500},[151,108167,108168],{"class":503},"df1.groupby(",[151,108170,108171],{"class":481},"'Chipset'",[151,108173,108174],{"class":503},").avg.count().sort_values( ",[151,108176,65817],{"class":15210},[151,108178,1876],{"class":1869},[151,108180,39461],{"class":477},[151,108182,94822],{"class":503},[151,108184,42017],{"class":477},[151,108186,100905],{"class":503},[151,108188,100637],{"class":15210},[151,108190,1876],{"class":1869},[151,108192,100642],{"class":481},[151,108194,3640],{"class":503},[151,108196,108197,108199,108202,108204,108206,108208,108210],{"class":469,"line":509},[151,108198,65123],{"class":503},[151,108200,108201],{"class":481},"'GPUs by Chipset'",[151,108203,106],{"class":503},[151,108205,99065],{"class":15210},[151,108207,1876],{"class":1869},[151,108209,67140],{"class":477},[151,108211,3640],{"class":503},[151,108213,108214,108216,108218,108220,108222,108224,108226],{"class":469,"line":517},[151,108215,65133],{"class":503},[151,108217,108171],{"class":481},[151,108219,106],{"class":503},[151,108221,99065],{"class":15210},[151,108223,1876],{"class":1869},[151,108225,67140],{"class":477},[151,108227,3640],{"class":503},[151,108229,108230,108232,108234,108236,108238,108240,108242],{"class":469,"line":534},[151,108231,65143],{"class":503},[151,108233,87648],{"class":481},[151,108235,106],{"class":503},[151,108237,99065],{"class":15210},[151,108239,1876],{"class":1869},[151,108241,67140],{"class":477},[151,108243,3640],{"class":503},[151,108245,108246,108248,108250,108252,108254],{"class":469,"line":1413},[151,108247,65163],{"class":503},[151,108249,99065],{"class":15210},[151,108251,1876],{"class":1869},[151,108253,42327],{"class":477},[151,108255,3640],{"class":503},[151,108257,108258,108260,108262,108264,108266],{"class":469,"line":1418},[151,108259,99502],{"class":503},[151,108261,99065],{"class":15210},[151,108263,1876],{"class":1869},[151,108265,42327],{"class":477},[151,108267,3640],{"class":503},[151,108269,108270,108272,108275],{"class":469,"line":2462},[151,108271,93826],{"class":503},[151,108273,108274],{"class":481},"'~/Documents/GitHub/briancaffey.github.io/img/gpu/gpu_chipsets.png'",[151,108276,12451],{"class":503},[11,108278,108279],{},[2718,108280],{"alt":20386,"src":108281},"/static/pcpp/gpu/gpu_chipsets.png",[11,108283,108284],{},"And here is the same graph with average prices for the top 20 most common GPU chipsets:",[459,108286,108288],{"className":13136,"code":108287,"language":12886,"meta":464,"style":464},"df[df.avg>0].groupby('Chipset').avg.agg(['mean', 'count']).sort_values('count', ascending=False)[:20].plot(kind='bar', figsize=(12,8))\nplt.title('Average Price of GPU Chipsets (sorted by count)', fontsize=14)\nplt.legend(loc='upper right', fontsize=14)\nplt.xlabel('GPU Chipset', fontsize=14)\nplt.xticks(fontsize=13)\nplt.yticks(fontsize=13)\nplt.savefig(os.path.expanduser('~/Documents/GitHub/briancaffey.github.io/img/gpu/gpu_chipsets_by_price.png'))\n",[30,108289,108290,108352,108369,108389,108406,108418,108430],{"__ignoreMap":464},[151,108291,108292,108294,108296,108298,108301,108303,108306,108308,108310,108312,108314,108316,108318,108320,108322,108324,108326,108328,108330,108332,108334,108336,108338,108340,108342,108344,108346,108348,108350],{"class":469,"line":470},[151,108293,101132],{"class":503},[151,108295,3663],{"class":1869},[151,108297,9181],{"class":477},[151,108299,108300],{"class":503},"].groupby(",[151,108302,108171],{"class":481},[151,108304,108305],{"class":503},").avg.agg([",[151,108307,100876],{"class":481},[151,108309,106],{"class":503},[151,108311,100881],{"class":481},[151,108313,100884],{"class":503},[151,108315,100881],{"class":481},[151,108317,106],{"class":503},[151,108319,65817],{"class":15210},[151,108321,1876],{"class":1869},[151,108323,39461],{"class":477},[151,108325,94822],{"class":503},[151,108327,9097],{"class":477},[151,108329,100905],{"class":503},[151,108331,100637],{"class":15210},[151,108333,1876],{"class":1869},[151,108335,100642],{"class":481},[151,108337,106],{"class":503},[151,108339,44358],{"class":15210},[151,108341,1876],{"class":1869},[151,108343,12386],{"class":503},[151,108345,42360],{"class":477},[151,108347,3634],{"class":503},[151,108349,24369],{"class":477},[151,108351,12451],{"class":503},[151,108353,108354,108356,108359,108361,108363,108365,108367],{"class":469,"line":488},[151,108355,65123],{"class":503},[151,108357,108358],{"class":481},"'Average Price of GPU Chipsets (sorted by count)'",[151,108360,106],{"class":503},[151,108362,99065],{"class":15210},[151,108364,1876],{"class":1869},[151,108366,67140],{"class":477},[151,108368,3640],{"class":503},[151,108370,108371,108373,108375,108377,108379,108381,108383,108385,108387],{"class":469,"line":500},[151,108372,104216],{"class":503},[151,108374,104219],{"class":15210},[151,108376,1876],{"class":1869},[151,108378,104431],{"class":481},[151,108380,106],{"class":503},[151,108382,99065],{"class":15210},[151,108384,1876],{"class":1869},[151,108386,67140],{"class":477},[151,108388,3640],{"class":503},[151,108390,108391,108393,108396,108398,108400,108402,108404],{"class":469,"line":509},[151,108392,65133],{"class":503},[151,108394,108395],{"class":481},"'GPU Chipset'",[151,108397,106],{"class":503},[151,108399,99065],{"class":15210},[151,108401,1876],{"class":1869},[151,108403,67140],{"class":477},[151,108405,3640],{"class":503},[151,108407,108408,108410,108412,108414,108416],{"class":469,"line":517},[151,108409,65163],{"class":503},[151,108411,99065],{"class":15210},[151,108413,1876],{"class":1869},[151,108415,42327],{"class":477},[151,108417,3640],{"class":503},[151,108419,108420,108422,108424,108426,108428],{"class":469,"line":534},[151,108421,99502],{"class":503},[151,108423,99065],{"class":15210},[151,108425,1876],{"class":1869},[151,108427,42327],{"class":477},[151,108429,3640],{"class":503},[151,108431,108432,108434,108437],{"class":469,"line":1413},[151,108433,93826],{"class":503},[151,108435,108436],{"class":481},"'~/Documents/GitHub/briancaffey.github.io/img/gpu/gpu_chipsets_by_price.png'",[151,108438,12451],{"class":503},[11,108440,108441],{},[2718,108442],{"alt":20386,"src":108443},"/static/pcpp/gpu/gpu_chipsets_by_price.png",[11,108445,108446],{},"In the next graph we can see clusters of GPU chipset families by plotting the clock speed and prices of video cards:",[459,108448,108450],{"className":13136,"code":108449,"language":12886,"meta":464,"style":464},"df2 = df[(df.avg>0)&(df.clock_speed_in_mhz>0)&(df.memory_mb\u003C10000)]\nplt.figure(figsize=(12,8))\nplt.scatter(df2.clock_speed_in_mhz, df2.avg, s=75, c=df2.memory_mb, cmap='CMRmap_r')\nplt.colorbar(label='Memory (MB)')\nplt.axis([500,1800,0,1000])\nplt.title('Clock Speed (MHz) vs. Price and Memory (color)', fontsize=14)\nplt.xlabel('Clock Speed', fontsize=14)\nplt.ylabel('Price', fontsize=14)\nplt.xticks(fontsize=13)\nplt.yticks(fontsize=13)\nplt.savefig(os.path.expanduser('~/Documents/GitHub/briancaffey.github.io/img/gpu/clock_speed_vs_price_and_memory.png'))\n",[30,108451,108452,108486,108504,108533,108545,108565,108582,108599,108615,108627,108639],{"__ignoreMap":464},[151,108453,108454,108456,108458,108460,108462,108464,108466,108468,108470,108472,108474,108476,108478,108480,108482,108484],{"class":469,"line":470},[151,108455,87049],{"class":503},[151,108457,1876],{"class":1869},[151,108459,100420],{"class":503},[151,108461,3663],{"class":1869},[151,108463,9181],{"class":477},[151,108465,748],{"class":503},[151,108467,54214],{"class":1869},[151,108469,107945],{"class":503},[151,108471,3663],{"class":1869},[151,108473,9181],{"class":477},[151,108475,748],{"class":503},[151,108477,54214],{"class":1869},[151,108479,107762],{"class":503},[151,108481,3613],{"class":1869},[151,108483,45984],{"class":477},[151,108485,44576],{"class":503},[151,108487,108488,108490,108492,108494,108496,108498,108500,108502],{"class":469,"line":488},[151,108489,44355],{"class":503},[151,108491,44358],{"class":15210},[151,108493,1876],{"class":1869},[151,108495,12386],{"class":503},[151,108497,42360],{"class":477},[151,108499,3634],{"class":503},[151,108501,24369],{"class":477},[151,108503,12451],{"class":503},[151,108505,108506,108509,108511,108513,108515,108517,108519,108521,108524,108526,108528,108531],{"class":469,"line":500},[151,108507,108508],{"class":503},"plt.scatter(df2.clock_speed_in_mhz, df2.avg, ",[151,108510,55630],{"class":15210},[151,108512,1876],{"class":1869},[151,108514,88018],{"class":477},[151,108516,106],{"class":503},[151,108518,65290],{"class":15210},[151,108520,1876],{"class":1869},[151,108522,108523],{"class":503},"df2.memory_mb, ",[151,108525,103551],{"class":15210},[151,108527,1876],{"class":1869},[151,108529,108530],{"class":481},"'CMRmap_r'",[151,108532,3640],{"class":503},[151,108534,108535,108537,108539,108541,108543],{"class":469,"line":509},[151,108536,103563],{"class":503},[151,108538,103566],{"class":15210},[151,108540,1876],{"class":1869},[151,108542,107596],{"class":481},[151,108544,3640],{"class":503},[151,108546,108547,108549,108551,108553,108555,108557,108559,108561,108563],{"class":469,"line":517},[151,108548,99036],{"class":503},[151,108550,12208],{"class":477},[151,108552,3634],{"class":503},[151,108554,99043],{"class":477},[151,108556,3634],{"class":503},[151,108558,9181],{"class":477},[151,108560,3634],{"class":503},[151,108562,45779],{"class":477},[151,108564,38820],{"class":503},[151,108566,108567,108569,108572,108574,108576,108578,108580],{"class":469,"line":534},[151,108568,65123],{"class":503},[151,108570,108571],{"class":481},"'Clock Speed (MHz) vs. Price and Memory (color)'",[151,108573,106],{"class":503},[151,108575,99065],{"class":15210},[151,108577,1876],{"class":1869},[151,108579,67140],{"class":477},[151,108581,3640],{"class":503},[151,108583,108584,108586,108589,108591,108593,108595,108597],{"class":469,"line":1413},[151,108585,65133],{"class":503},[151,108587,108588],{"class":481},"'Clock Speed'",[151,108590,106],{"class":503},[151,108592,99065],{"class":15210},[151,108594,1876],{"class":1869},[151,108596,67140],{"class":477},[151,108598,3640],{"class":503},[151,108600,108601,108603,108605,108607,108609,108611,108613],{"class":469,"line":1418},[151,108602,65143],{"class":503},[151,108604,99095],{"class":481},[151,108606,106],{"class":503},[151,108608,99065],{"class":15210},[151,108610,1876],{"class":1869},[151,108612,67140],{"class":477},[151,108614,3640],{"class":503},[151,108616,108617,108619,108621,108623,108625],{"class":469,"line":2462},[151,108618,65163],{"class":503},[151,108620,99065],{"class":15210},[151,108622,1876],{"class":1869},[151,108624,42327],{"class":477},[151,108626,3640],{"class":503},[151,108628,108629,108631,108633,108635,108637],{"class":469,"line":2471},[151,108630,99502],{"class":503},[151,108632,99065],{"class":15210},[151,108634,1876],{"class":1869},[151,108636,42327],{"class":477},[151,108638,3640],{"class":503},[151,108640,108641,108643,108646],{"class":469,"line":2480},[151,108642,93826],{"class":503},[151,108644,108645],{"class":481},"'~/Documents/GitHub/briancaffey.github.io/img/gpu/clock_speed_vs_price_and_memory.png'",[151,108647,12451],{"class":503},[11,108649,108650],{},[2718,108651],{"alt":20386,"src":108652},"/static/pcpp/gpu/clock_speed_vs_price_and_memory.png",[11,108654,108655],{},"Now let's look at six generations of NVIDIA GPUs by clock speed:",[459,108657,108659],{"className":13136,"code":108658,"language":12886,"meta":464,"style":464},"#6 generations of NVIDIA graphics cards\ndf_N = df[(df.make==\"NVIDIA\")&(df.avg>0)]\ndf_N['NVIDIA_Series'] = [10 if 'GTX 10' in x else \\\n                          9 if 'GTX 9' in x else \\\n                          7 if 'GTX 7' in x else \\\n                          6 if 'GTX 6' in x else \\\n                          5 if 'GTX 5' in x else \\\n                          4 if 'GTX 4' in x else \\\n                          'other' for x in df_N.Chipset]\n\nplt.figure(figsize=(15,10))\nplt.axis([500,1800,0,1000])\nplt.title('6 Generations of NVIDIA Graphics Cards: Price vs. Clock Speed', fontsize=14)\nplt.xlabel('Clock Speed MHz', fontsize=14)\nplt.ylabel('Price', fontsize=14)\n\ncolors = ['#76b900', '#8946ff', '#cea503', '#9c0000', '#5c5c5c', '#0b75bd']\ns = 150\nn10 = plt.scatter(df_N[df_N.NVIDIA_Series==10].clock_speed_in_mhz, df_N[df_N.NVIDIA_Series==10].avg, color = colors[0], s=s)\nn9 = plt.scatter(df_N[df_N.NVIDIA_Series==9].clock_speed_in_mhz, df_N[df_N.NVIDIA_Series==9].avg, color = colors[1], s=s)\nn7 = plt.scatter(df_N[df_N.NVIDIA_Series==7].clock_speed_in_mhz, df_N[df_N.NVIDIA_Series==7].avg, color = colors[2], s=s)\nn6 = plt.scatter(df_N[df_N.NVIDIA_Series==6].clock_speed_in_mhz, df_N[df_N.NVIDIA_Series==6].avg, color = colors[3], s=s)\nn5 = plt.scatter(df_N[df_N.NVIDIA_Series==5].clock_speed_in_mhz, df_N[df_N.NVIDIA_Series==5].avg, color = colors[4], s=s)\nn4 = plt.scatter(df_N[df_N.NVIDIA_Series==4].clock_speed_in_mhz, df_N[df_N.NVIDIA_Series==4].avg, color = colors[5], s=s)\n\nplt.legend((n10, n9, n7, n6, n5, n4),\n           ('10 Series', '9 Series', '7 Series', '6 Series', '5 Series', '4 Series'),\n           title = 'NVIDIA GeForce Generations',\n           scatterpoints=3,\n           loc='upper left',\n           ncol=2,\n           fontsize=14)\nsns.despine()\nplt.show()\n\nplt.savefig(os.path.expanduser('~/Documents/GitHub/briancaffey.github.io/img/gpu/six_NVIDIA.png'))\n",[30,108660,108661,108666,108693,108722,108740,108758,108776,108794,108812,108826,108830,108848,108868,108885,108902,108918,108922,108960,108969,109013,109054,109095,109136,109177,109218,109222,109227,109261,109272,109282,109292,109302,109312,109317,109321,109325],{"__ignoreMap":464},[151,108662,108663],{"class":469,"line":470},[151,108664,108665],{"class":1527},"#6 generations of NVIDIA graphics cards\n",[151,108667,108668,108671,108673,108676,108678,108681,108683,108685,108687,108689,108691],{"class":469,"line":488},[151,108669,108670],{"class":503},"df_N ",[151,108672,1876],{"class":1869},[151,108674,108675],{"class":503}," df[(df.make",[151,108677,17223],{"class":1869},[151,108679,108680],{"class":481},"\"NVIDIA\"",[151,108682,748],{"class":503},[151,108684,54214],{"class":1869},[151,108686,100313],{"class":503},[151,108688,3663],{"class":1869},[151,108690,9181],{"class":477},[151,108692,44576],{"class":503},[151,108694,108695,108698,108701,108703,108705,108707,108709,108711,108714,108716,108718,108720],{"class":469,"line":500},[151,108696,108697],{"class":503},"df_N[",[151,108699,108700],{"class":481},"'NVIDIA_Series'",[151,108702,16654],{"class":503},[151,108704,1876],{"class":1869},[151,108706,6604],{"class":503},[151,108708,12423],{"class":477},[151,108710,3435],{"class":1869},[151,108712,108713],{"class":481}," 'GTX 10'",[151,108715,2820],{"class":1869},[151,108717,44552],{"class":503},[151,108719,77868],{"class":1869},[151,108721,485],{"class":503},[151,108723,108724,108727,108729,108732,108734,108736,108738],{"class":469,"line":509},[151,108725,108726],{"class":477},"                          9",[151,108728,3435],{"class":1869},[151,108730,108731],{"class":481}," 'GTX 9'",[151,108733,2820],{"class":1869},[151,108735,44552],{"class":503},[151,108737,77868],{"class":1869},[151,108739,485],{"class":503},[151,108741,108742,108745,108747,108750,108752,108754,108756],{"class":469,"line":517},[151,108743,108744],{"class":477},"                          7",[151,108746,3435],{"class":1869},[151,108748,108749],{"class":481}," 'GTX 7'",[151,108751,2820],{"class":1869},[151,108753,44552],{"class":503},[151,108755,77868],{"class":1869},[151,108757,485],{"class":503},[151,108759,108760,108763,108765,108768,108770,108772,108774],{"class":469,"line":534},[151,108761,108762],{"class":477},"                          6",[151,108764,3435],{"class":1869},[151,108766,108767],{"class":481}," 'GTX 6'",[151,108769,2820],{"class":1869},[151,108771,44552],{"class":503},[151,108773,77868],{"class":1869},[151,108775,485],{"class":503},[151,108777,108778,108781,108783,108786,108788,108790,108792],{"class":469,"line":1413},[151,108779,108780],{"class":477},"                          5",[151,108782,3435],{"class":1869},[151,108784,108785],{"class":481}," 'GTX 5'",[151,108787,2820],{"class":1869},[151,108789,44552],{"class":503},[151,108791,77868],{"class":1869},[151,108793,485],{"class":503},[151,108795,108796,108799,108801,108804,108806,108808,108810],{"class":469,"line":1418},[151,108797,108798],{"class":477},"                          4",[151,108800,3435],{"class":1869},[151,108802,108803],{"class":481}," 'GTX 4'",[151,108805,2820],{"class":1869},[151,108807,44552],{"class":503},[151,108809,77868],{"class":1869},[151,108811,485],{"class":503},[151,108813,108814,108817,108819,108821,108823],{"class":469,"line":2462},[151,108815,108816],{"class":481},"                          'other'",[151,108818,2235],{"class":1869},[151,108820,44552],{"class":503},[151,108822,16417],{"class":1869},[151,108824,108825],{"class":503}," df_N.Chipset]\n",[151,108827,108828],{"class":469,"line":2471},[151,108829,1090],{"emptyLinePlaceholder":609},[151,108831,108832,108834,108836,108838,108840,108842,108844,108846],{"class":469,"line":2480},[151,108833,44355],{"class":503},[151,108835,44358],{"class":15210},[151,108837,1876],{"class":1869},[151,108839,12386],{"class":503},[151,108841,42310],{"class":477},[151,108843,3634],{"class":503},[151,108845,12423],{"class":477},[151,108847,12451],{"class":503},[151,108849,108850,108852,108854,108856,108858,108860,108862,108864,108866],{"class":469,"line":2489},[151,108851,99036],{"class":503},[151,108853,12208],{"class":477},[151,108855,3634],{"class":503},[151,108857,99043],{"class":477},[151,108859,3634],{"class":503},[151,108861,9181],{"class":477},[151,108863,3634],{"class":503},[151,108865,45779],{"class":477},[151,108867,38820],{"class":503},[151,108869,108870,108872,108875,108877,108879,108881,108883],{"class":469,"line":2497},[151,108871,65123],{"class":503},[151,108873,108874],{"class":481},"'6 Generations of NVIDIA Graphics Cards: Price vs. Clock Speed'",[151,108876,106],{"class":503},[151,108878,99065],{"class":15210},[151,108880,1876],{"class":1869},[151,108882,67140],{"class":477},[151,108884,3640],{"class":503},[151,108886,108887,108889,108892,108894,108896,108898,108900],{"class":469,"line":3140},[151,108888,65133],{"class":503},[151,108890,108891],{"class":481},"'Clock Speed MHz'",[151,108893,106],{"class":503},[151,108895,99065],{"class":15210},[151,108897,1876],{"class":1869},[151,108899,67140],{"class":477},[151,108901,3640],{"class":503},[151,108903,108904,108906,108908,108910,108912,108914,108916],{"class":469,"line":3149},[151,108905,65143],{"class":503},[151,108907,99095],{"class":481},[151,108909,106],{"class":503},[151,108911,99065],{"class":15210},[151,108913,1876],{"class":1869},[151,108915,67140],{"class":477},[151,108917,3640],{"class":503},[151,108919,108920],{"class":469,"line":3158},[151,108921,1090],{"emptyLinePlaceholder":609},[151,108923,108924,108926,108928,108930,108933,108935,108938,108940,108943,108945,108948,108950,108953,108955,108958],{"class":469,"line":3167},[151,108925,99115],{"class":503},[151,108927,1876],{"class":1869},[151,108929,6604],{"class":503},[151,108931,108932],{"class":481},"'#76b900'",[151,108934,106],{"class":503},[151,108936,108937],{"class":481},"'#8946ff'",[151,108939,106],{"class":503},[151,108941,108942],{"class":481},"'#cea503'",[151,108944,106],{"class":503},[151,108946,108947],{"class":481},"'#9c0000'",[151,108949,106],{"class":503},[151,108951,108952],{"class":481},"'#5c5c5c'",[151,108954,106],{"class":503},[151,108956,108957],{"class":481},"'#0b75bd'",[151,108959,3691],{"class":503},[151,108961,108962,108964,108966],{"class":469,"line":3175},[151,108963,74751],{"class":503},[151,108965,1876],{"class":1869},[151,108967,108968],{"class":477}," 150\n",[151,108970,108971,108974,108976,108979,108982,108984,108986,108989,108991,108993,108995,108997,108999,109001,109003,109005,109007,109009,109011],{"class":469,"line":3184},[151,108972,108973],{"class":503},"n10 ",[151,108975,1876],{"class":1869},[151,108977,108978],{"class":503}," plt.scatter(df_N[df_N.",[151,108980,108981],{"class":477},"NVIDIA_Series",[151,108983,17223],{"class":1869},[151,108985,12423],{"class":477},[151,108987,108988],{"class":503},"].clock_speed_in_mhz, df_N[df_N.",[151,108990,108981],{"class":477},[151,108992,17223],{"class":1869},[151,108994,12423],{"class":477},[151,108996,99177],{"class":503},[151,108998,79362],{"class":15210},[151,109000,19865],{"class":1869},[151,109002,99184],{"class":503},[151,109004,9181],{"class":477},[151,109006,60308],{"class":503},[151,109008,55630],{"class":15210},[151,109010,1876],{"class":1869},[151,109012,105370],{"class":503},[151,109014,109015,109018,109020,109022,109024,109026,109028,109030,109032,109034,109036,109038,109040,109042,109044,109046,109048,109050,109052],{"class":469,"line":3193},[151,109016,109017],{"class":503},"n9 ",[151,109019,1876],{"class":1869},[151,109021,108978],{"class":503},[151,109023,108981],{"class":477},[151,109025,17223],{"class":1869},[151,109027,7918],{"class":477},[151,109029,108988],{"class":503},[151,109031,108981],{"class":477},[151,109033,17223],{"class":1869},[151,109035,7918],{"class":477},[151,109037,99177],{"class":503},[151,109039,79362],{"class":15210},[151,109041,19865],{"class":1869},[151,109043,99184],{"class":503},[151,109045,6760],{"class":477},[151,109047,60308],{"class":503},[151,109049,55630],{"class":15210},[151,109051,1876],{"class":1869},[151,109053,105370],{"class":503},[151,109055,109056,109059,109061,109063,109065,109067,109069,109071,109073,109075,109077,109079,109081,109083,109085,109087,109089,109091,109093],{"class":469,"line":3720},[151,109057,109058],{"class":503},"n7 ",[151,109060,1876],{"class":1869},[151,109062,108978],{"class":503},[151,109064,108981],{"class":477},[151,109066,17223],{"class":1869},[151,109068,25043],{"class":477},[151,109070,108988],{"class":503},[151,109072,108981],{"class":477},[151,109074,17223],{"class":1869},[151,109076,25043],{"class":477},[151,109078,99177],{"class":503},[151,109080,79362],{"class":15210},[151,109082,19865],{"class":1869},[151,109084,99184],{"class":503},[151,109086,6619],{"class":477},[151,109088,60308],{"class":503},[151,109090,55630],{"class":15210},[151,109092,1876],{"class":1869},[151,109094,105370],{"class":503},[151,109096,109097,109100,109102,109104,109106,109108,109110,109112,109114,109116,109118,109120,109122,109124,109126,109128,109130,109132,109134],{"class":469,"line":3729},[151,109098,109099],{"class":503},"n6 ",[151,109101,1876],{"class":1869},[151,109103,108978],{"class":503},[151,109105,108981],{"class":477},[151,109107,17223],{"class":1869},[151,109109,25038],{"class":477},[151,109111,108988],{"class":503},[151,109113,108981],{"class":477},[151,109115,17223],{"class":1869},[151,109117,25038],{"class":477},[151,109119,99177],{"class":503},[151,109121,79362],{"class":15210},[151,109123,19865],{"class":1869},[151,109125,99184],{"class":503},[151,109127,6557],{"class":477},[151,109129,60308],{"class":503},[151,109131,55630],{"class":15210},[151,109133,1876],{"class":1869},[151,109135,105370],{"class":503},[151,109137,109138,109141,109143,109145,109147,109149,109151,109153,109155,109157,109159,109161,109163,109165,109167,109169,109171,109173,109175],{"class":469,"line":3735},[151,109139,109140],{"class":503},"n5 ",[151,109142,1876],{"class":1869},[151,109144,108978],{"class":503},[151,109146,108981],{"class":477},[151,109148,17223],{"class":1869},[151,109150,24380],{"class":477},[151,109152,108988],{"class":503},[151,109154,108981],{"class":477},[151,109156,17223],{"class":1869},[151,109158,24380],{"class":477},[151,109160,99177],{"class":503},[151,109162,79362],{"class":15210},[151,109164,19865],{"class":1869},[151,109166,99184],{"class":503},[151,109168,9187],{"class":477},[151,109170,60308],{"class":503},[151,109172,55630],{"class":15210},[151,109174,1876],{"class":1869},[151,109176,105370],{"class":503},[151,109178,109179,109182,109184,109186,109188,109190,109192,109194,109196,109198,109200,109202,109204,109206,109208,109210,109212,109214,109216],{"class":469,"line":3745},[151,109180,109181],{"class":503},"n4 ",[151,109183,1876],{"class":1869},[151,109185,108978],{"class":503},[151,109187,108981],{"class":477},[151,109189,17223],{"class":1869},[151,109191,9187],{"class":477},[151,109193,108988],{"class":503},[151,109195,108981],{"class":477},[151,109197,17223],{"class":1869},[151,109199,9187],{"class":477},[151,109201,99177],{"class":503},[151,109203,79362],{"class":15210},[151,109205,19865],{"class":1869},[151,109207,99184],{"class":503},[151,109209,24380],{"class":477},[151,109211,60308],{"class":503},[151,109213,55630],{"class":15210},[151,109215,1876],{"class":1869},[151,109217,105370],{"class":503},[151,109219,109220],{"class":469,"line":3754},[151,109221,1090],{"emptyLinePlaceholder":609},[151,109223,109224],{"class":469,"line":3760},[151,109225,109226],{"class":503},"plt.legend((n10, n9, n7, n6, n5, n4),\n",[151,109228,109229,109231,109234,109236,109239,109241,109244,109246,109249,109251,109254,109256,109259],{"class":469,"line":3773},[151,109230,99405],{"class":503},[151,109232,109233],{"class":481},"'10 Series'",[151,109235,106],{"class":503},[151,109237,109238],{"class":481},"'9 Series'",[151,109240,106],{"class":503},[151,109242,109243],{"class":481},"'7 Series'",[151,109245,106],{"class":503},[151,109247,109248],{"class":481},"'6 Series'",[151,109250,106],{"class":503},[151,109252,109253],{"class":481},"'5 Series'",[151,109255,106],{"class":503},[151,109257,109258],{"class":481},"'4 Series'",[151,109260,37985],{"class":503},[151,109262,109263,109265,109267,109270],{"class":469,"line":3782},[151,109264,71582],{"class":15210},[151,109266,19865],{"class":1869},[151,109268,109269],{"class":481}," 'NVIDIA GeForce Generations'",[151,109271,9417],{"class":503},[151,109273,109274,109276,109278,109280],{"class":469,"line":3791},[151,109275,99445],{"class":15210},[151,109277,1876],{"class":1869},[151,109279,6557],{"class":477},[151,109281,9417],{"class":503},[151,109283,109284,109286,109288,109290],{"class":469,"line":3803},[151,109285,99456],{"class":15210},[151,109287,1876],{"class":1869},[151,109289,99461],{"class":481},[151,109291,9417],{"class":503},[151,109293,109294,109296,109298,109300],{"class":469,"line":3811},[151,109295,99468],{"class":15210},[151,109297,1876],{"class":1869},[151,109299,6619],{"class":477},[151,109301,9417],{"class":503},[151,109303,109304,109306,109308,109310],{"class":469,"line":3820},[151,109305,99479],{"class":15210},[151,109307,1876],{"class":1869},[151,109309,67140],{"class":477},[151,109311,3640],{"class":503},[151,109313,109314],{"class":469,"line":7084},[151,109315,109316],{"class":503},"sns.despine()\n",[151,109318,109319],{"class":469,"line":7148},[151,109320,44415],{"class":503},[151,109322,109323],{"class":469,"line":7211},[151,109324,1090],{"emptyLinePlaceholder":609},[151,109326,109327,109329,109332],{"class":469,"line":7273},[151,109328,93826],{"class":503},[151,109330,109331],{"class":481},"'~/Documents/GitHub/briancaffey.github.io/img/gpu/six_NVIDIA.png'",[151,109333,12451],{"class":503},[11,109335,109336],{},[2718,109337],{"alt":20386,"src":109338},"/static/pcpp/gpu/six_NVIDIA.png",[11,109340,109341],{},"It's also helpful to compare the most recent two series of NVIDIA GPUs by chipset families:",[459,109343,109345],{"className":13136,"code":109344,"language":12886,"meta":464,"style":464},"#10 Series vs. 9 Series\ndf_N = df[(df.make==\"NVIDIA\")&(df.avg>0)]\ndf_N['NVIDIA_Series_9_10'] = [1080 if 'GTX 108' in x else \\\n                          1070 if 'GTX 1070' in x else \\\n                          1060 if 'GTX 1060' in x else \\\n                          980 if 'GTX 980' in x else \\\n                          970 if 'GTX 970' in x else \\\n                          960 if 'GTX 960' in x else \\\n                          950 if 'GTX 950' in x else \\\n                          'other' for x in df_N.Chipset]\n\nplt.figure(figsize=(15,10))\nplt.axis([800,1800,0,1000])\nplt.title('NVIDIA GeForce GTX: 10 Series vs. 9 Series', fontsize=14)\nplt.xlabel('Clock Speed (MHz)', fontsize=14)\nplt.ylabel('Price', fontsize=14)\n\ncolors = ['#76b900', '#8946ff', '#5c5c5c', '#9c0000', '#cea503', '#0b75bd', 'red']\ns = 150\n\nn1080 = df_N[df_N.NVIDIA_Series_9_10==1080]\nn1070 = df_N[df_N.NVIDIA_Series_9_10==1070]\nn1060 = df_N[df_N.NVIDIA_Series_9_10==1060]\nn980 = df_N[df_N.NVIDIA_Series_9_10==980]\nn970 = df_N[df_N.NVIDIA_Series_9_10==970]\nn960 = df_N[df_N.NVIDIA_Series_9_10==960]\nn950 = df_N[df_N.NVIDIA_Series_9_10==950]\n\nn_1080 = plt.scatter(n1080.clock_speed_in_mhz, n1080.avg, color = colors[0], s=s)\nn_1070 = plt.scatter(n1070.clock_speed_in_mhz, n1070.avg, color = colors[1], s=s)\nn_1060 = plt.scatter(n1060.clock_speed_in_mhz, n1060.avg, color = colors[2], s=s)\nn_980 = plt.scatter(n980.clock_speed_in_mhz, n980.avg, color = colors[3], s=s)\nn_970 = plt.scatter(n970.clock_speed_in_mhz, n970.avg, color = colors[4], s=s)\nn_960 = plt.scatter(n960.clock_speed_in_mhz, n960.avg, color = colors[5], s=s)\nn_950 = plt.scatter(n950.clock_speed_in_mhz, n950.avg, color = colors[6], s=s)\n\nplt.legend((n_1080, n_1070, n_1060, n_980, n_970, n_960, n_950),\n           ('1080', '1070', '1060', '980', '970', '960', '950'),\n           title = 'NVIDIA GeForce',\n           scatterpoints=3,\n           loc='upper left',\n           ncol=1,\n           fontsize=14)\nplt.xticks(fontsize=13)\nplt.yticks(fontsize=13)\nsns.despine()\nplt.show()\nplt.savefig(os.path.expanduser('~/Documents/GitHub/briancaffey.github.io/img/gpu/9_vs_10.png'))\n",[30,109346,109347,109352,109376,109404,109422,109440,109458,109476,109494,109512,109524,109528,109546,109566,109583,109599,109615,109619,109655,109663,109667,109686,109704,109722,109739,109756,109773,109790,109794,109820,109846,109872,109898,109924,109950,109976,109980,109985,110024,110035,110045,110055,110065,110075,110087,110099,110103,110107],{"__ignoreMap":464},[151,109348,109349],{"class":469,"line":470},[151,109350,109351],{"class":1527},"#10 Series vs. 9 Series\n",[151,109353,109354,109356,109358,109360,109362,109364,109366,109368,109370,109372,109374],{"class":469,"line":488},[151,109355,108670],{"class":503},[151,109357,1876],{"class":1869},[151,109359,108675],{"class":503},[151,109361,17223],{"class":1869},[151,109363,108680],{"class":481},[151,109365,748],{"class":503},[151,109367,54214],{"class":1869},[151,109369,100313],{"class":503},[151,109371,3663],{"class":1869},[151,109373,9181],{"class":477},[151,109375,44576],{"class":503},[151,109377,109378,109380,109383,109385,109387,109389,109391,109393,109396,109398,109400,109402],{"class":469,"line":500},[151,109379,108697],{"class":503},[151,109381,109382],{"class":481},"'NVIDIA_Series_9_10'",[151,109384,16654],{"class":503},[151,109386,1876],{"class":1869},[151,109388,6604],{"class":503},[151,109390,44290],{"class":477},[151,109392,3435],{"class":1869},[151,109394,109395],{"class":481}," 'GTX 108'",[151,109397,2820],{"class":1869},[151,109399,44552],{"class":503},[151,109401,77868],{"class":1869},[151,109403,485],{"class":503},[151,109405,109406,109409,109411,109414,109416,109418,109420],{"class":469,"line":509},[151,109407,109408],{"class":477},"                          1070",[151,109410,3435],{"class":1869},[151,109412,109413],{"class":481}," 'GTX 1070'",[151,109415,2820],{"class":1869},[151,109417,44552],{"class":503},[151,109419,77868],{"class":1869},[151,109421,485],{"class":503},[151,109423,109424,109427,109429,109432,109434,109436,109438],{"class":469,"line":517},[151,109425,109426],{"class":477},"                          1060",[151,109428,3435],{"class":1869},[151,109430,109431],{"class":481}," 'GTX 1060'",[151,109433,2820],{"class":1869},[151,109435,44552],{"class":503},[151,109437,77868],{"class":1869},[151,109439,485],{"class":503},[151,109441,109442,109445,109447,109450,109452,109454,109456],{"class":469,"line":534},[151,109443,109444],{"class":477},"                          980",[151,109446,3435],{"class":1869},[151,109448,109449],{"class":481}," 'GTX 980'",[151,109451,2820],{"class":1869},[151,109453,44552],{"class":503},[151,109455,77868],{"class":1869},[151,109457,485],{"class":503},[151,109459,109460,109463,109465,109468,109470,109472,109474],{"class":469,"line":1413},[151,109461,109462],{"class":477},"                          970",[151,109464,3435],{"class":1869},[151,109466,109467],{"class":481}," 'GTX 970'",[151,109469,2820],{"class":1869},[151,109471,44552],{"class":503},[151,109473,77868],{"class":1869},[151,109475,485],{"class":503},[151,109477,109478,109481,109483,109486,109488,109490,109492],{"class":469,"line":1418},[151,109479,109480],{"class":477},"                          960",[151,109482,3435],{"class":1869},[151,109484,109485],{"class":481}," 'GTX 960'",[151,109487,2820],{"class":1869},[151,109489,44552],{"class":503},[151,109491,77868],{"class":1869},[151,109493,485],{"class":503},[151,109495,109496,109499,109501,109504,109506,109508,109510],{"class":469,"line":2462},[151,109497,109498],{"class":477},"                          950",[151,109500,3435],{"class":1869},[151,109502,109503],{"class":481}," 'GTX 950'",[151,109505,2820],{"class":1869},[151,109507,44552],{"class":503},[151,109509,77868],{"class":1869},[151,109511,485],{"class":503},[151,109513,109514,109516,109518,109520,109522],{"class":469,"line":2471},[151,109515,108816],{"class":481},[151,109517,2235],{"class":1869},[151,109519,44552],{"class":503},[151,109521,16417],{"class":1869},[151,109523,108825],{"class":503},[151,109525,109526],{"class":469,"line":2480},[151,109527,1090],{"emptyLinePlaceholder":609},[151,109529,109530,109532,109534,109536,109538,109540,109542,109544],{"class":469,"line":2489},[151,109531,44355],{"class":503},[151,109533,44358],{"class":15210},[151,109535,1876],{"class":1869},[151,109537,12386],{"class":503},[151,109539,42310],{"class":477},[151,109541,3634],{"class":503},[151,109543,12423],{"class":477},[151,109545,12451],{"class":503},[151,109547,109548,109550,109552,109554,109556,109558,109560,109562,109564],{"class":469,"line":2497},[151,109549,99036],{"class":503},[151,109551,74155],{"class":477},[151,109553,3634],{"class":503},[151,109555,99043],{"class":477},[151,109557,3634],{"class":503},[151,109559,9181],{"class":477},[151,109561,3634],{"class":503},[151,109563,45779],{"class":477},[151,109565,38820],{"class":503},[151,109567,109568,109570,109573,109575,109577,109579,109581],{"class":469,"line":3140},[151,109569,65123],{"class":503},[151,109571,109572],{"class":481},"'NVIDIA GeForce GTX: 10 Series vs. 9 Series'",[151,109574,106],{"class":503},[151,109576,99065],{"class":15210},[151,109578,1876],{"class":1869},[151,109580,67140],{"class":477},[151,109582,3640],{"class":503},[151,109584,109585,109587,109589,109591,109593,109595,109597],{"class":469,"line":3149},[151,109586,65133],{"class":503},[151,109588,108013],{"class":481},[151,109590,106],{"class":503},[151,109592,99065],{"class":15210},[151,109594,1876],{"class":1869},[151,109596,67140],{"class":477},[151,109598,3640],{"class":503},[151,109600,109601,109603,109605,109607,109609,109611,109613],{"class":469,"line":3158},[151,109602,65143],{"class":503},[151,109604,99095],{"class":481},[151,109606,106],{"class":503},[151,109608,99065],{"class":15210},[151,109610,1876],{"class":1869},[151,109612,67140],{"class":477},[151,109614,3640],{"class":503},[151,109616,109617],{"class":469,"line":3167},[151,109618,1090],{"emptyLinePlaceholder":609},[151,109620,109621,109623,109625,109627,109629,109631,109633,109635,109637,109639,109641,109643,109645,109647,109649,109651,109653],{"class":469,"line":3175},[151,109622,99115],{"class":503},[151,109624,1876],{"class":1869},[151,109626,6604],{"class":503},[151,109628,108932],{"class":481},[151,109630,106],{"class":503},[151,109632,108937],{"class":481},[151,109634,106],{"class":503},[151,109636,108952],{"class":481},[151,109638,106],{"class":503},[151,109640,108947],{"class":481},[151,109642,106],{"class":503},[151,109644,108942],{"class":481},[151,109646,106],{"class":503},[151,109648,108957],{"class":481},[151,109650,106],{"class":503},[151,109652,80832],{"class":481},[151,109654,3691],{"class":503},[151,109656,109657,109659,109661],{"class":469,"line":3184},[151,109658,74751],{"class":503},[151,109660,1876],{"class":1869},[151,109662,108968],{"class":477},[151,109664,109665],{"class":469,"line":3193},[151,109666,1090],{"emptyLinePlaceholder":609},[151,109668,109669,109672,109674,109677,109680,109682,109684],{"class":469,"line":3720},[151,109670,109671],{"class":503},"n1080 ",[151,109673,1876],{"class":1869},[151,109675,109676],{"class":503}," df_N[df_N.",[151,109678,109679],{"class":477},"NVIDIA_Series_9_10",[151,109681,17223],{"class":1869},[151,109683,44290],{"class":477},[151,109685,3691],{"class":503},[151,109687,109688,109691,109693,109695,109697,109699,109702],{"class":469,"line":3729},[151,109689,109690],{"class":503},"n1070 ",[151,109692,1876],{"class":1869},[151,109694,109676],{"class":503},[151,109696,109679],{"class":477},[151,109698,17223],{"class":1869},[151,109700,109701],{"class":477},"1070",[151,109703,3691],{"class":503},[151,109705,109706,109709,109711,109713,109715,109717,109720],{"class":469,"line":3735},[151,109707,109708],{"class":503},"n1060 ",[151,109710,1876],{"class":1869},[151,109712,109676],{"class":503},[151,109714,109679],{"class":477},[151,109716,17223],{"class":1869},[151,109718,109719],{"class":477},"1060",[151,109721,3691],{"class":503},[151,109723,109724,109727,109729,109731,109733,109735,109737],{"class":469,"line":3745},[151,109725,109726],{"class":503},"n980 ",[151,109728,1876],{"class":1869},[151,109730,109676],{"class":503},[151,109732,109679],{"class":477},[151,109734,17223],{"class":1869},[151,109736,74377],{"class":477},[151,109738,3691],{"class":503},[151,109740,109741,109744,109746,109748,109750,109752,109754],{"class":469,"line":3754},[151,109742,109743],{"class":503},"n970 ",[151,109745,1876],{"class":1869},[151,109747,109676],{"class":503},[151,109749,109679],{"class":477},[151,109751,17223],{"class":1869},[151,109753,74365],{"class":477},[151,109755,3691],{"class":503},[151,109757,109758,109761,109763,109765,109767,109769,109771],{"class":469,"line":3760},[151,109759,109760],{"class":503},"n960 ",[151,109762,1876],{"class":1869},[151,109764,109676],{"class":503},[151,109766,109679],{"class":477},[151,109768,17223],{"class":1869},[151,109770,74353],{"class":477},[151,109772,3691],{"class":503},[151,109774,109775,109778,109780,109782,109784,109786,109788],{"class":469,"line":3773},[151,109776,109777],{"class":503},"n950 ",[151,109779,1876],{"class":1869},[151,109781,109676],{"class":503},[151,109783,109679],{"class":477},[151,109785,17223],{"class":1869},[151,109787,74341],{"class":477},[151,109789,3691],{"class":503},[151,109791,109792],{"class":469,"line":3782},[151,109793,1090],{"emptyLinePlaceholder":609},[151,109795,109796,109799,109801,109804,109806,109808,109810,109812,109814,109816,109818],{"class":469,"line":3791},[151,109797,109798],{"class":503},"n_1080 ",[151,109800,1876],{"class":1869},[151,109802,109803],{"class":503}," plt.scatter(n1080.clock_speed_in_mhz, n1080.avg, ",[151,109805,79362],{"class":15210},[151,109807,19865],{"class":1869},[151,109809,99184],{"class":503},[151,109811,9181],{"class":477},[151,109813,60308],{"class":503},[151,109815,55630],{"class":15210},[151,109817,1876],{"class":1869},[151,109819,105370],{"class":503},[151,109821,109822,109825,109827,109830,109832,109834,109836,109838,109840,109842,109844],{"class":469,"line":3803},[151,109823,109824],{"class":503},"n_1070 ",[151,109826,1876],{"class":1869},[151,109828,109829],{"class":503}," plt.scatter(n1070.clock_speed_in_mhz, n1070.avg, ",[151,109831,79362],{"class":15210},[151,109833,19865],{"class":1869},[151,109835,99184],{"class":503},[151,109837,6760],{"class":477},[151,109839,60308],{"class":503},[151,109841,55630],{"class":15210},[151,109843,1876],{"class":1869},[151,109845,105370],{"class":503},[151,109847,109848,109851,109853,109856,109858,109860,109862,109864,109866,109868,109870],{"class":469,"line":3811},[151,109849,109850],{"class":503},"n_1060 ",[151,109852,1876],{"class":1869},[151,109854,109855],{"class":503}," plt.scatter(n1060.clock_speed_in_mhz, n1060.avg, ",[151,109857,79362],{"class":15210},[151,109859,19865],{"class":1869},[151,109861,99184],{"class":503},[151,109863,6619],{"class":477},[151,109865,60308],{"class":503},[151,109867,55630],{"class":15210},[151,109869,1876],{"class":1869},[151,109871,105370],{"class":503},[151,109873,109874,109877,109879,109882,109884,109886,109888,109890,109892,109894,109896],{"class":469,"line":3820},[151,109875,109876],{"class":503},"n_980 ",[151,109878,1876],{"class":1869},[151,109880,109881],{"class":503}," plt.scatter(n980.clock_speed_in_mhz, n980.avg, ",[151,109883,79362],{"class":15210},[151,109885,19865],{"class":1869},[151,109887,99184],{"class":503},[151,109889,6557],{"class":477},[151,109891,60308],{"class":503},[151,109893,55630],{"class":15210},[151,109895,1876],{"class":1869},[151,109897,105370],{"class":503},[151,109899,109900,109903,109905,109908,109910,109912,109914,109916,109918,109920,109922],{"class":469,"line":7084},[151,109901,109902],{"class":503},"n_970 ",[151,109904,1876],{"class":1869},[151,109906,109907],{"class":503}," plt.scatter(n970.clock_speed_in_mhz, n970.avg, ",[151,109909,79362],{"class":15210},[151,109911,19865],{"class":1869},[151,109913,99184],{"class":503},[151,109915,9187],{"class":477},[151,109917,60308],{"class":503},[151,109919,55630],{"class":15210},[151,109921,1876],{"class":1869},[151,109923,105370],{"class":503},[151,109925,109926,109929,109931,109934,109936,109938,109940,109942,109944,109946,109948],{"class":469,"line":7148},[151,109927,109928],{"class":503},"n_960 ",[151,109930,1876],{"class":1869},[151,109932,109933],{"class":503}," plt.scatter(n960.clock_speed_in_mhz, n960.avg, ",[151,109935,79362],{"class":15210},[151,109937,19865],{"class":1869},[151,109939,99184],{"class":503},[151,109941,24380],{"class":477},[151,109943,60308],{"class":503},[151,109945,55630],{"class":15210},[151,109947,1876],{"class":1869},[151,109949,105370],{"class":503},[151,109951,109952,109955,109957,109960,109962,109964,109966,109968,109970,109972,109974],{"class":469,"line":7211},[151,109953,109954],{"class":503},"n_950 ",[151,109956,1876],{"class":1869},[151,109958,109959],{"class":503}," plt.scatter(n950.clock_speed_in_mhz, n950.avg, ",[151,109961,79362],{"class":15210},[151,109963,19865],{"class":1869},[151,109965,99184],{"class":503},[151,109967,25038],{"class":477},[151,109969,60308],{"class":503},[151,109971,55630],{"class":15210},[151,109973,1876],{"class":1869},[151,109975,105370],{"class":503},[151,109977,109978],{"class":469,"line":7273},[151,109979,1090],{"emptyLinePlaceholder":609},[151,109981,109982],{"class":469,"line":7335},[151,109983,109984],{"class":503},"plt.legend((n_1080, n_1070, n_1060, n_980, n_970, n_960, n_950),\n",[151,109986,109987,109989,109992,109994,109997,109999,110002,110004,110007,110009,110012,110014,110017,110019,110022],{"class":469,"line":7398},[151,109988,99405],{"class":503},[151,109990,109991],{"class":481},"'1080'",[151,109993,106],{"class":503},[151,109995,109996],{"class":481},"'1070'",[151,109998,106],{"class":503},[151,110000,110001],{"class":481},"'1060'",[151,110003,106],{"class":503},[151,110005,110006],{"class":481},"'980'",[151,110008,106],{"class":503},[151,110010,110011],{"class":481},"'970'",[151,110013,106],{"class":503},[151,110015,110016],{"class":481},"'960'",[151,110018,106],{"class":503},[151,110020,110021],{"class":481},"'950'",[151,110023,37985],{"class":503},[151,110025,110026,110028,110030,110033],{"class":469,"line":7462},[151,110027,71582],{"class":15210},[151,110029,19865],{"class":1869},[151,110031,110032],{"class":481}," 'NVIDIA GeForce'",[151,110034,9417],{"class":503},[151,110036,110037,110039,110041,110043],{"class":469,"line":7467},[151,110038,99445],{"class":15210},[151,110040,1876],{"class":1869},[151,110042,6557],{"class":477},[151,110044,9417],{"class":503},[151,110046,110047,110049,110051,110053],{"class":469,"line":7532},[151,110048,99456],{"class":15210},[151,110050,1876],{"class":1869},[151,110052,99461],{"class":481},[151,110054,9417],{"class":503},[151,110056,110057,110059,110061,110063],{"class":469,"line":7537},[151,110058,99468],{"class":15210},[151,110060,1876],{"class":1869},[151,110062,6760],{"class":477},[151,110064,9417],{"class":503},[151,110066,110067,110069,110071,110073],{"class":469,"line":7603},[151,110068,99479],{"class":15210},[151,110070,1876],{"class":1869},[151,110072,67140],{"class":477},[151,110074,3640],{"class":503},[151,110076,110077,110079,110081,110083,110085],{"class":469,"line":7608},[151,110078,65163],{"class":503},[151,110080,99065],{"class":15210},[151,110082,1876],{"class":1869},[151,110084,42327],{"class":477},[151,110086,3640],{"class":503},[151,110088,110089,110091,110093,110095,110097],{"class":469,"line":7673},[151,110090,99502],{"class":503},[151,110092,99065],{"class":15210},[151,110094,1876],{"class":1869},[151,110096,42327],{"class":477},[151,110098,3640],{"class":503},[151,110100,110101],{"class":469,"line":7678},[151,110102,109316],{"class":503},[151,110104,110105],{"class":469,"line":7708},[151,110106,44415],{"class":503},[151,110108,110109,110111,110114],{"class":469,"line":7713},[151,110110,93826],{"class":503},[151,110112,110113],{"class":481},"'~/Documents/GitHub/briancaffey.github.io/img/gpu/9_vs_10.png'",[151,110115,12451],{"class":503},[11,110117,110118],{},[2718,110119],{"alt":20386,"src":110120},"/static/pcpp/gpu/9_vs_10.png",[11,110122,110123],{},"Finally, as we did for CPUs, we can look at the core memory clock and the boost clock for GPUs:",[459,110125,110127],{"className":13136,"code":110126,"language":12886,"meta":464,"style":464},"df2 = df[(df.avg>0)&(df.clock_speed_in_mhz>0)&(df.boost_clock_speed_mhz>0)&(df.tdp>0)]\n\nfrom sklearn.linear_model import LinearRegression\nlreg = LinearRegression()\n\nx = df2.clock_speed_in_mhz\nY = df2.boost_clock_speed_mhz\n\nx = x.values.reshape(-1,1)\nY = Y.values.reshape(-1,1)\n\nlreg.fit(x, Y, sample_weight=None)\ns = df2.tdp*3\na = 0.3\nsns.set_style('whitegrid')\nplt.figure(figsize=(12,8))\nplt.plot(x,lreg.predict(x), c='y')\nplt.scatter(df2[df.make=='NVIDIA'].clock_speed_in_mhz, df2[df.make=='NVIDIA'].boost_clock_speed_mhz, s=s, c='green', alpha=a)\nplt.scatter(df2[df.make=='AMD'].clock_speed_in_mhz, df2[df.make=='AMD'].boost_clock_speed_mhz, s=s, c='r', alpha=a)\nplt.title('Core Clock vs. Boost Clock and TDP (diameter)', fontsize=14)\nplt.xlabel('Core Clock Speed (MHz)', fontsize=14)\nplt.ylabel('Boost Clock Speed (MHz)', fontsize=14)\nplt.xticks(fontsize=13)\nplt.yticks(fontsize=13)\nplt.axis([600,2100,600,2100])\nx_points = [700,1000,1500,2000]\ny_points = [700,1000,1500,2000]\n\nplt.plot(x_points,y_points, c='blue')\nplt.legend([ 'Line of best fit', 'y = x', 'Core vs. Boost Clock (NVIDIA)', 'Core vs. Boost Clock (AMD)'], loc='upper left', fontsize=13)\nplt.savefig(os.path.expanduser('~/Documents/GitHub/briancaffey.github.io/img/gpu/gpu_clock_vs_boost.png'))\n",[30,110128,110129,110175,110179,110189,110197,110201,110210,110220,110224,110243,110262,110266,110279,110292,110301,110309,110327,110340,110380,110416,110433,110450,110467,110479,110491,110512,110537,110562,110566,110579,110620],{"__ignoreMap":464},[151,110130,110131,110133,110135,110137,110139,110141,110143,110145,110147,110149,110151,110153,110155,110158,110160,110162,110164,110166,110169,110171,110173],{"class":469,"line":470},[151,110132,87049],{"class":503},[151,110134,1876],{"class":1869},[151,110136,100420],{"class":503},[151,110138,3663],{"class":1869},[151,110140,9181],{"class":477},[151,110142,748],{"class":503},[151,110144,54214],{"class":1869},[151,110146,107945],{"class":503},[151,110148,3663],{"class":1869},[151,110150,9181],{"class":477},[151,110152,748],{"class":503},[151,110154,54214],{"class":1869},[151,110156,110157],{"class":503},"(df.boost_clock_speed_mhz",[151,110159,3663],{"class":1869},[151,110161,9181],{"class":477},[151,110163,748],{"class":503},[151,110165,54214],{"class":1869},[151,110167,110168],{"class":503},"(df.tdp",[151,110170,3663],{"class":1869},[151,110172,9181],{"class":477},[151,110174,44576],{"class":503},[151,110176,110177],{"class":469,"line":488},[151,110178,1090],{"emptyLinePlaceholder":609},[151,110180,110181,110183,110185,110187],{"class":469,"line":500},[151,110182,16853],{"class":1869},[151,110184,100477],{"class":503},[151,110186,16859],{"class":1869},[151,110188,100482],{"class":503},[151,110190,110191,110193,110195],{"class":469,"line":509},[151,110192,104916],{"class":503},[151,110194,1876],{"class":1869},[151,110196,100492],{"class":503},[151,110198,110199],{"class":469,"line":517},[151,110200,1090],{"emptyLinePlaceholder":609},[151,110202,110203,110205,110207],{"class":469,"line":534},[151,110204,87578],{"class":503},[151,110206,1876],{"class":1869},[151,110208,110209],{"class":503}," df2.clock_speed_in_mhz\n",[151,110211,110212,110215,110217],{"class":469,"line":1413},[151,110213,110214],{"class":503},"Y ",[151,110216,1876],{"class":1869},[151,110218,110219],{"class":503}," df2.boost_clock_speed_mhz\n",[151,110221,110222],{"class":469,"line":1418},[151,110223,1090],{"emptyLinePlaceholder":609},[151,110225,110226,110228,110230,110233,110235,110237,110239,110241],{"class":469,"line":2462},[151,110227,87578],{"class":503},[151,110229,1876],{"class":1869},[151,110231,110232],{"class":503}," x.values.reshape(",[151,110234,12445],{"class":1869},[151,110236,6760],{"class":477},[151,110238,3634],{"class":503},[151,110240,6760],{"class":477},[151,110242,3640],{"class":503},[151,110244,110245,110247,110249,110252,110254,110256,110258,110260],{"class":469,"line":2471},[151,110246,110214],{"class":503},[151,110248,1876],{"class":1869},[151,110250,110251],{"class":503}," Y.values.reshape(",[151,110253,12445],{"class":1869},[151,110255,6760],{"class":477},[151,110257,3634],{"class":503},[151,110259,6760],{"class":477},[151,110261,3640],{"class":503},[151,110263,110264],{"class":469,"line":2480},[151,110265,1090],{"emptyLinePlaceholder":609},[151,110267,110268,110271,110273,110275,110277],{"class":469,"line":2489},[151,110269,110270],{"class":503},"lreg.fit(x, Y, ",[151,110272,104986],{"class":15210},[151,110274,1876],{"class":1869},[151,110276,15437],{"class":477},[151,110278,3640],{"class":503},[151,110280,110281,110283,110285,110288,110290],{"class":469,"line":2497},[151,110282,74751],{"class":503},[151,110284,1876],{"class":1869},[151,110286,110287],{"class":503}," df2.tdp",[151,110289,23268],{"class":1869},[151,110291,64255],{"class":477},[151,110293,110294,110296,110298],{"class":469,"line":3140},[151,110295,61268],{"class":503},[151,110297,1876],{"class":1869},[151,110299,110300],{"class":477}," 0.3\n",[151,110302,110303,110305,110307],{"class":469,"line":3149},[151,110304,87588],{"class":503},[151,110306,87591],{"class":481},[151,110308,3640],{"class":503},[151,110310,110311,110313,110315,110317,110319,110321,110323,110325],{"class":469,"line":3158},[151,110312,44355],{"class":503},[151,110314,44358],{"class":15210},[151,110316,1876],{"class":1869},[151,110318,12386],{"class":503},[151,110320,42360],{"class":477},[151,110322,3634],{"class":503},[151,110324,24369],{"class":477},[151,110326,12451],{"class":503},[151,110328,110329,110332,110334,110336,110338],{"class":469,"line":3167},[151,110330,110331],{"class":503},"plt.plot(x,lreg.predict(x), ",[151,110333,65290],{"class":15210},[151,110335,1876],{"class":1869},[151,110337,71616],{"class":481},[151,110339,3640],{"class":503},[151,110341,110342,110345,110347,110350,110353,110355,110357,110360,110362,110364,110366,110368,110370,110372,110374,110376,110378],{"class":469,"line":3175},[151,110343,110344],{"class":503},"plt.scatter(df2[df.make",[151,110346,17223],{"class":1869},[151,110348,110349],{"class":481},"'NVIDIA'",[151,110351,110352],{"class":503},"].clock_speed_in_mhz, df2[df.make",[151,110354,17223],{"class":1869},[151,110356,110349],{"class":481},[151,110358,110359],{"class":503},"].boost_clock_speed_mhz, ",[151,110361,55630],{"class":15210},[151,110363,1876],{"class":1869},[151,110365,105899],{"class":503},[151,110367,65290],{"class":15210},[151,110369,1876],{"class":1869},[151,110371,105604],{"class":481},[151,110373,106],{"class":503},[151,110375,26256],{"class":15210},[151,110377,1876],{"class":1869},[151,110379,105906],{"class":503},[151,110381,110382,110384,110386,110388,110390,110392,110394,110396,110398,110400,110402,110404,110406,110408,110410,110412,110414],{"class":469,"line":3184},[151,110383,110344],{"class":503},[151,110385,17223],{"class":1869},[151,110387,102952],{"class":481},[151,110389,110352],{"class":503},[151,110391,17223],{"class":1869},[151,110393,102952],{"class":481},[151,110395,110359],{"class":503},[151,110397,55630],{"class":15210},[151,110399,1876],{"class":1869},[151,110401,105899],{"class":503},[151,110403,65290],{"class":15210},[151,110405,1876],{"class":1869},[151,110407,44149],{"class":481},[151,110409,106],{"class":503},[151,110411,26256],{"class":15210},[151,110413,1876],{"class":1869},[151,110415,105906],{"class":503},[151,110417,110418,110420,110423,110425,110427,110429,110431],{"class":469,"line":3193},[151,110419,65123],{"class":503},[151,110421,110422],{"class":481},"'Core Clock vs. Boost Clock and TDP (diameter)'",[151,110424,106],{"class":503},[151,110426,99065],{"class":15210},[151,110428,1876],{"class":1869},[151,110430,67140],{"class":477},[151,110432,3640],{"class":503},[151,110434,110435,110437,110440,110442,110444,110446,110448],{"class":469,"line":3720},[151,110436,65133],{"class":503},[151,110438,110439],{"class":481},"'Core Clock Speed (MHz)'",[151,110441,106],{"class":503},[151,110443,99065],{"class":15210},[151,110445,1876],{"class":1869},[151,110447,67140],{"class":477},[151,110449,3640],{"class":503},[151,110451,110452,110454,110457,110459,110461,110463,110465],{"class":469,"line":3729},[151,110453,65143],{"class":503},[151,110455,110456],{"class":481},"'Boost Clock Speed (MHz)'",[151,110458,106],{"class":503},[151,110460,99065],{"class":15210},[151,110462,1876],{"class":1869},[151,110464,67140],{"class":477},[151,110466,3640],{"class":503},[151,110468,110469,110471,110473,110475,110477],{"class":469,"line":3735},[151,110470,65163],{"class":503},[151,110472,99065],{"class":15210},[151,110474,1876],{"class":1869},[151,110476,42327],{"class":477},[151,110478,3640],{"class":503},[151,110480,110481,110483,110485,110487,110489],{"class":469,"line":3745},[151,110482,99502],{"class":503},[151,110484,99065],{"class":15210},[151,110486,1876],{"class":1869},[151,110488,42327],{"class":477},[151,110490,3640],{"class":503},[151,110492,110493,110495,110497,110499,110502,110504,110506,110508,110510],{"class":469,"line":3754},[151,110494,99036],{"class":503},[151,110496,44836],{"class":477},[151,110498,3634],{"class":503},[151,110500,110501],{"class":477},"2100",[151,110503,3634],{"class":503},[151,110505,44836],{"class":477},[151,110507,3634],{"class":503},[151,110509,110501],{"class":477},[151,110511,38820],{"class":503},[151,110513,110514,110517,110519,110521,110523,110525,110527,110529,110531,110533,110535],{"class":469,"line":3760},[151,110515,110516],{"class":503},"x_points ",[151,110518,1876],{"class":1869},[151,110520,6604],{"class":503},[151,110522,74027],{"class":477},[151,110524,3634],{"class":503},[151,110526,45779],{"class":477},[151,110528,3634],{"class":503},[151,110530,103631],{"class":477},[151,110532,3634],{"class":503},[151,110534,107267],{"class":477},[151,110536,3691],{"class":503},[151,110538,110539,110542,110544,110546,110548,110550,110552,110554,110556,110558,110560],{"class":469,"line":3773},[151,110540,110541],{"class":503},"y_points ",[151,110543,1876],{"class":1869},[151,110545,6604],{"class":503},[151,110547,74027],{"class":477},[151,110549,3634],{"class":503},[151,110551,45779],{"class":477},[151,110553,3634],{"class":503},[151,110555,103631],{"class":477},[151,110557,3634],{"class":503},[151,110559,107267],{"class":477},[151,110561,3691],{"class":503},[151,110563,110564],{"class":469,"line":3782},[151,110565,1090],{"emptyLinePlaceholder":609},[151,110567,110568,110571,110573,110575,110577],{"class":469,"line":3791},[151,110569,110570],{"class":503},"plt.plot(x_points,y_points, ",[151,110572,65290],{"class":15210},[151,110574,1876],{"class":1869},[151,110576,102887],{"class":481},[151,110578,3640],{"class":503},[151,110580,110581,110584,110587,110589,110592,110594,110597,110599,110602,110604,110606,110608,110610,110612,110614,110616,110618],{"class":469,"line":3803},[151,110582,110583],{"class":503},"plt.legend([ ",[151,110585,110586],{"class":481},"'Line of best fit'",[151,110588,106],{"class":503},[151,110590,110591],{"class":481},"'y = x'",[151,110593,106],{"class":503},[151,110595,110596],{"class":481},"'Core vs. Boost Clock (NVIDIA)'",[151,110598,106],{"class":503},[151,110600,110601],{"class":481},"'Core vs. Boost Clock (AMD)'",[151,110603,60308],{"class":503},[151,110605,104219],{"class":15210},[151,110607,1876],{"class":1869},[151,110609,99461],{"class":481},[151,110611,106],{"class":503},[151,110613,99065],{"class":15210},[151,110615,1876],{"class":1869},[151,110617,42327],{"class":477},[151,110619,3640],{"class":503},[151,110621,110622,110624,110627],{"class":469,"line":3811},[151,110623,93826],{"class":503},[151,110625,110626],{"class":481},"'~/Documents/GitHub/briancaffey.github.io/img/gpu/gpu_clock_vs_boost.png'",[151,110628,12451],{"class":503},[11,110630,110631],{},[2718,110632],{"alt":20386,"src":110633},"/static/pcpp/gpu/gpu_clock_vs_boost.png",[11,110635,110636,110637,643],{},"The latest generation of GPUs at much higher clock speeds than previous generations are able to boost clocks even faster than previous generations, on average. This is shown by the slightly steeper curve of the line of best fit as compared with ",[30,110638,110639],{},"y = x",[14063,110641,110643],{"id":110642},"hard-drives","Hard Drives",[11,110645,110646,110647,110652],{},"Hard drive sizes have been growing at an exponential rate since the 1950s. This ",[20,110648,110651],{"href":110649,"rel":110650},"https://en.wikipedia.org/wiki/File:Hard_drive_capacity_over_time.svg",[24],"chart from Wikipedia"," shows the growth of storage sizes on a logarithmic scale in recent decades:",[11,110654,110655],{},[2718,110656],{"alt":20386,"src":110657},"/static/pcpp/storage/storage_growth.png",[11,110659,110660,110661,110666],{},"Linear growth on a logarithmic scale corresponds to exponential growth on a linear scale, which is the basis of Moore's Law. Hard drive disks follow a growth pattern similar to Moore's Law called ",[20,110662,110665],{"href":110663,"rel":110664},"https://en.wikipedia.org/wiki/Mark_Kryder",[24],"Kryder's Law",".\nHere is a scatter plot of storage drive sizes and prices:",[459,110668,110670],{"className":13136,"code":110669,"language":12886,"meta":464,"style":464},"sns.set_style('whitegrid')\nplt.figure(figsize=(12,8))\nplt.axis([0,10500,0,1500])\nssd = df[(df.avg>0)&(df.is_ssd==\"Yes\")]\nhdd = df[(df.avg>0)&(df.is_ssd==\"No\")]\nplt.title('Hard Drive Sizes (GB) vs. Prices ', fontsize=14)\ns=75\nplt.scatter(ssd.storage_gb, ssd.avg, c='black', s=s, alpha=.3)\nplt.scatter(hdd.storage_gb, hdd.avg, c='red', s=s, alpha=.3)\nplt.legend(['SSD', 'HDD'], loc='upper right', fontsize=14)\nplt.xlabel('Hard Drive size (GB)', fontsize=14)\nplt.ylabel('Price', fontsize=14)\nplt.xticks(fontsize=13)\nplt.yticks(fontsize=13)\nplt.savefig(os.path.expanduser('~/Documents/GitHub/briancaffey.github.io/img/storage/storage_vs_price.png'))\n",[30,110671,110672,110680,110698,110719,110745,110770,110787,110795,110823,110850,110880,110897,110913,110925,110937],{"__ignoreMap":464},[151,110673,110674,110676,110678],{"class":469,"line":470},[151,110675,87588],{"class":503},[151,110677,87591],{"class":481},[151,110679,3640],{"class":503},[151,110681,110682,110684,110686,110688,110690,110692,110694,110696],{"class":469,"line":488},[151,110683,44355],{"class":503},[151,110685,44358],{"class":15210},[151,110687,1876],{"class":1869},[151,110689,12386],{"class":503},[151,110691,42360],{"class":477},[151,110693,3634],{"class":503},[151,110695,24369],{"class":477},[151,110697,12451],{"class":503},[151,110699,110700,110702,110704,110706,110709,110711,110713,110715,110717],{"class":469,"line":500},[151,110701,99036],{"class":503},[151,110703,9181],{"class":477},[151,110705,3634],{"class":503},[151,110707,110708],{"class":477},"10500",[151,110710,3634],{"class":503},[151,110712,9181],{"class":477},[151,110714,3634],{"class":503},[151,110716,103631],{"class":477},[151,110718,38820],{"class":503},[151,110720,110721,110724,110726,110728,110730,110732,110734,110736,110739,110741,110743],{"class":469,"line":509},[151,110722,110723],{"class":503},"ssd ",[151,110725,1876],{"class":1869},[151,110727,100420],{"class":503},[151,110729,3663],{"class":1869},[151,110731,9181],{"class":477},[151,110733,748],{"class":503},[151,110735,54214],{"class":1869},[151,110737,110738],{"class":503},"(df.is_ssd",[151,110740,17223],{"class":1869},[151,110742,103704],{"class":481},[151,110744,44576],{"class":503},[151,110746,110747,110750,110752,110754,110756,110758,110760,110762,110764,110766,110768],{"class":469,"line":517},[151,110748,110749],{"class":503},"hdd ",[151,110751,1876],{"class":1869},[151,110753,100420],{"class":503},[151,110755,3663],{"class":1869},[151,110757,9181],{"class":477},[151,110759,748],{"class":503},[151,110761,54214],{"class":1869},[151,110763,110738],{"class":503},[151,110765,17223],{"class":1869},[151,110767,104345],{"class":481},[151,110769,44576],{"class":503},[151,110771,110772,110774,110777,110779,110781,110783,110785],{"class":469,"line":534},[151,110773,65123],{"class":503},[151,110775,110776],{"class":481},"'Hard Drive Sizes (GB) vs. Prices '",[151,110778,106],{"class":503},[151,110780,99065],{"class":15210},[151,110782,1876],{"class":1869},[151,110784,67140],{"class":477},[151,110786,3640],{"class":503},[151,110788,110789,110791,110793],{"class":469,"line":1413},[151,110790,55630],{"class":503},[151,110792,1876],{"class":1869},[151,110794,105244],{"class":477},[151,110796,110797,110800,110802,110804,110806,110808,110810,110812,110814,110816,110818,110821],{"class":469,"line":1418},[151,110798,110799],{"class":503},"plt.scatter(ssd.storage_gb, ssd.avg, ",[151,110801,65290],{"class":15210},[151,110803,1876],{"class":1869},[151,110805,45401],{"class":481},[151,110807,106],{"class":503},[151,110809,55630],{"class":15210},[151,110811,1876],{"class":1869},[151,110813,105899],{"class":503},[151,110815,26256],{"class":15210},[151,110817,1876],{"class":1869},[151,110819,110820],{"class":477},".3",[151,110822,3640],{"class":503},[151,110824,110825,110828,110830,110832,110834,110836,110838,110840,110842,110844,110846,110848],{"class":469,"line":2462},[151,110826,110827],{"class":503},"plt.scatter(hdd.storage_gb, hdd.avg, ",[151,110829,65290],{"class":15210},[151,110831,1876],{"class":1869},[151,110833,80832],{"class":481},[151,110835,106],{"class":503},[151,110837,55630],{"class":15210},[151,110839,1876],{"class":1869},[151,110841,105899],{"class":503},[151,110843,26256],{"class":15210},[151,110845,1876],{"class":1869},[151,110847,110820],{"class":477},[151,110849,3640],{"class":503},[151,110851,110852,110854,110857,110859,110862,110864,110866,110868,110870,110872,110874,110876,110878],{"class":469,"line":2471},[151,110853,102944],{"class":503},[151,110855,110856],{"class":481},"'SSD'",[151,110858,106],{"class":503},[151,110860,110861],{"class":481},"'HDD'",[151,110863,60308],{"class":503},[151,110865,104219],{"class":15210},[151,110867,1876],{"class":1869},[151,110869,104431],{"class":481},[151,110871,106],{"class":503},[151,110873,99065],{"class":15210},[151,110875,1876],{"class":1869},[151,110877,67140],{"class":477},[151,110879,3640],{"class":503},[151,110881,110882,110884,110887,110889,110891,110893,110895],{"class":469,"line":2480},[151,110883,65133],{"class":503},[151,110885,110886],{"class":481},"'Hard Drive size (GB)'",[151,110888,106],{"class":503},[151,110890,99065],{"class":15210},[151,110892,1876],{"class":1869},[151,110894,67140],{"class":477},[151,110896,3640],{"class":503},[151,110898,110899,110901,110903,110905,110907,110909,110911],{"class":469,"line":2489},[151,110900,65143],{"class":503},[151,110902,99095],{"class":481},[151,110904,106],{"class":503},[151,110906,99065],{"class":15210},[151,110908,1876],{"class":1869},[151,110910,67140],{"class":477},[151,110912,3640],{"class":503},[151,110914,110915,110917,110919,110921,110923],{"class":469,"line":2497},[151,110916,65163],{"class":503},[151,110918,99065],{"class":15210},[151,110920,1876],{"class":1869},[151,110922,42327],{"class":477},[151,110924,3640],{"class":503},[151,110926,110927,110929,110931,110933,110935],{"class":469,"line":3140},[151,110928,99502],{"class":503},[151,110930,99065],{"class":15210},[151,110932,1876],{"class":1869},[151,110934,42327],{"class":477},[151,110936,3640],{"class":503},[151,110938,110939,110941,110944],{"class":469,"line":3149},[151,110940,93826],{"class":503},[151,110942,110943],{"class":481},"'~/Documents/GitHub/briancaffey.github.io/img/storage/storage_vs_price.png'",[151,110945,12451],{"class":503},[11,110947,110948],{},[2718,110949],{"alt":20386,"src":110950},"/static/pcpp/storage/storage_vs_price.png",[11,110952,110953],{},"HDD refers to a spinning hard drive disk. Electromechanical magnetic disks spin at high freuencies and a physical arm reads and writes data to and from the spinning disks. SSDs, or Solid state drives, have a number of advantages over spinning drives. SSDs have no moving parts, which makes them more shock resistant and quiter than HDDs. SSDs also have lower access time and lower latency, but they come at a much higher price per GB of storage than HDDs.",[11,110955,110956],{},"The next two graphs show SSDs and HDDs, respectively.",[459,110958,110960],{"className":13136,"code":110959,"language":12886,"meta":464,"style":464},"plt.figure(figsize=(12,8))\nplt.axis([0,2000,0,1500])\nssd = df[(df.avg>0)&(df.is_ssd==\"Yes\")]\n\nplt.scatter(ssd.storage_gb, ssd.avg, s=75, alpha=.2)\nplt.title('SSD Size vs. Price', fontsize=14)\nplt.xlabel('SSD Hard Drive size (GB)', fontsize=14)\nplt.ylabel('Price', fontsize=14)\nplt.xticks(fontsize=13)\nplt.yticks(fontsize=13)\nplt.savefig(os.path.expanduser('~/Documents/GitHub/briancaffey.github.io/img/storage/ssd_storage_vs_price.png'))\n",[30,110961,110962,110980,111000,111024,111028,111048,111065,111082,111098,111110,111122],{"__ignoreMap":464},[151,110963,110964,110966,110968,110970,110972,110974,110976,110978],{"class":469,"line":470},[151,110965,44355],{"class":503},[151,110967,44358],{"class":15210},[151,110969,1876],{"class":1869},[151,110971,12386],{"class":503},[151,110973,42360],{"class":477},[151,110975,3634],{"class":503},[151,110977,24369],{"class":477},[151,110979,12451],{"class":503},[151,110981,110982,110984,110986,110988,110990,110992,110994,110996,110998],{"class":469,"line":488},[151,110983,99036],{"class":503},[151,110985,9181],{"class":477},[151,110987,3634],{"class":503},[151,110989,107267],{"class":477},[151,110991,3634],{"class":503},[151,110993,9181],{"class":477},[151,110995,3634],{"class":503},[151,110997,103631],{"class":477},[151,110999,38820],{"class":503},[151,111001,111002,111004,111006,111008,111010,111012,111014,111016,111018,111020,111022],{"class":469,"line":500},[151,111003,110723],{"class":503},[151,111005,1876],{"class":1869},[151,111007,100420],{"class":503},[151,111009,3663],{"class":1869},[151,111011,9181],{"class":477},[151,111013,748],{"class":503},[151,111015,54214],{"class":1869},[151,111017,110738],{"class":503},[151,111019,17223],{"class":1869},[151,111021,103704],{"class":481},[151,111023,44576],{"class":503},[151,111025,111026],{"class":469,"line":509},[151,111027,1090],{"emptyLinePlaceholder":609},[151,111029,111030,111032,111034,111036,111038,111040,111042,111044,111046],{"class":469,"line":517},[151,111031,110799],{"class":503},[151,111033,55630],{"class":15210},[151,111035,1876],{"class":1869},[151,111037,88018],{"class":477},[151,111039,106],{"class":503},[151,111041,26256],{"class":15210},[151,111043,1876],{"class":1869},[151,111045,102878],{"class":477},[151,111047,3640],{"class":503},[151,111049,111050,111052,111055,111057,111059,111061,111063],{"class":469,"line":534},[151,111051,65123],{"class":503},[151,111053,111054],{"class":481},"'SSD Size vs. Price'",[151,111056,106],{"class":503},[151,111058,99065],{"class":15210},[151,111060,1876],{"class":1869},[151,111062,67140],{"class":477},[151,111064,3640],{"class":503},[151,111066,111067,111069,111072,111074,111076,111078,111080],{"class":469,"line":1413},[151,111068,65133],{"class":503},[151,111070,111071],{"class":481},"'SSD Hard Drive size (GB)'",[151,111073,106],{"class":503},[151,111075,99065],{"class":15210},[151,111077,1876],{"class":1869},[151,111079,67140],{"class":477},[151,111081,3640],{"class":503},[151,111083,111084,111086,111088,111090,111092,111094,111096],{"class":469,"line":1418},[151,111085,65143],{"class":503},[151,111087,99095],{"class":481},[151,111089,106],{"class":503},[151,111091,99065],{"class":15210},[151,111093,1876],{"class":1869},[151,111095,67140],{"class":477},[151,111097,3640],{"class":503},[151,111099,111100,111102,111104,111106,111108],{"class":469,"line":2462},[151,111101,65163],{"class":503},[151,111103,99065],{"class":15210},[151,111105,1876],{"class":1869},[151,111107,42327],{"class":477},[151,111109,3640],{"class":503},[151,111111,111112,111114,111116,111118,111120],{"class":469,"line":2471},[151,111113,99502],{"class":503},[151,111115,99065],{"class":15210},[151,111117,1876],{"class":1869},[151,111119,42327],{"class":477},[151,111121,3640],{"class":503},[151,111123,111124,111126,111129],{"class":469,"line":2480},[151,111125,93826],{"class":503},[151,111127,111128],{"class":481},"'~/Documents/GitHub/briancaffey.github.io/img/storage/ssd_storage_vs_price.png'",[151,111130,12451],{"class":503},[11,111132,111133],{},[2718,111134],{"alt":20386,"src":111135},"/static/pcpp/storage/ssd_storage_vs_price.png",[11,111137,111138],{},"For HDDs, the color of each point represent the speed in rotations per minute of the spinning disk:",[459,111140,111142],{"className":13136,"code":111141,"language":12886,"meta":464,"style":464},"plt.figure(figsize=(12,7))\nplt.axis([0,11000,0,650])\nhdd = df[(df.avg>0)&(df.is_ssd==\"No\")&(df.RPM.notnull())]\nplt.scatter(hdd.storage_gb, hdd.avg, c=hdd.RPM, s=40, cmap=\"Blues_r\")\nplt.colorbar(label='RPM')\nplt.title('HDD Size vs. Price and RPM (color)', fontsize=14)\nplt.xlabel('HDD Hard Drive size (GB)', fontsize=14)\nplt.ylabel('Price', fontsize=14)\nplt.xticks(fontsize=13)\nplt.yticks(fontsize=13)\nplt.savefig(os.path.expanduser('~/Documents/GitHub/briancaffey.github.io/img/storage/hdd_storage_vs_price.png'))\n",[30,111143,111144,111162,111183,111216,111248,111261,111278,111295,111311,111323,111335],{"__ignoreMap":464},[151,111145,111146,111148,111150,111152,111154,111156,111158,111160],{"class":469,"line":470},[151,111147,44355],{"class":503},[151,111149,44358],{"class":15210},[151,111151,1876],{"class":1869},[151,111153,12386],{"class":503},[151,111155,42360],{"class":477},[151,111157,3634],{"class":503},[151,111159,25043],{"class":477},[151,111161,12451],{"class":503},[151,111163,111164,111166,111168,111170,111173,111175,111177,111179,111181],{"class":469,"line":488},[151,111165,99036],{"class":503},[151,111167,9181],{"class":477},[151,111169,3634],{"class":503},[151,111171,111172],{"class":477},"11000",[151,111174,3634],{"class":503},[151,111176,9181],{"class":477},[151,111178,3634],{"class":503},[151,111180,73958],{"class":477},[151,111182,38820],{"class":503},[151,111184,111185,111187,111189,111191,111193,111195,111197,111199,111201,111203,111205,111207,111209,111211,111213],{"class":469,"line":500},[151,111186,110749],{"class":503},[151,111188,1876],{"class":1869},[151,111190,100420],{"class":503},[151,111192,3663],{"class":1869},[151,111194,9181],{"class":477},[151,111196,748],{"class":503},[151,111198,54214],{"class":1869},[151,111200,110738],{"class":503},[151,111202,17223],{"class":1869},[151,111204,104345],{"class":481},[151,111206,748],{"class":503},[151,111208,54214],{"class":1869},[151,111210,102008],{"class":503},[151,111212,97576],{"class":477},[151,111214,111215],{"class":503},".notnull())]\n",[151,111217,111218,111220,111222,111224,111227,111229,111231,111233,111235,111237,111239,111241,111243,111246],{"class":469,"line":509},[151,111219,110827],{"class":503},[151,111221,65290],{"class":15210},[151,111223,1876],{"class":1869},[151,111225,111226],{"class":503},"hdd.",[151,111228,97576],{"class":477},[151,111230,106],{"class":503},[151,111232,55630],{"class":15210},[151,111234,1876],{"class":1869},[151,111236,44365],{"class":477},[151,111238,106],{"class":503},[151,111240,103551],{"class":15210},[151,111242,1876],{"class":1869},[151,111244,111245],{"class":481},"\"Blues_r\"",[151,111247,3640],{"class":503},[151,111249,111250,111252,111254,111256,111259],{"class":469,"line":517},[151,111251,103563],{"class":503},[151,111253,103566],{"class":15210},[151,111255,1876],{"class":1869},[151,111257,111258],{"class":481},"'RPM'",[151,111260,3640],{"class":503},[151,111262,111263,111265,111268,111270,111272,111274,111276],{"class":469,"line":534},[151,111264,65123],{"class":503},[151,111266,111267],{"class":481},"'HDD Size vs. Price and RPM (color)'",[151,111269,106],{"class":503},[151,111271,99065],{"class":15210},[151,111273,1876],{"class":1869},[151,111275,67140],{"class":477},[151,111277,3640],{"class":503},[151,111279,111280,111282,111285,111287,111289,111291,111293],{"class":469,"line":1413},[151,111281,65133],{"class":503},[151,111283,111284],{"class":481},"'HDD Hard Drive size (GB)'",[151,111286,106],{"class":503},[151,111288,99065],{"class":15210},[151,111290,1876],{"class":1869},[151,111292,67140],{"class":477},[151,111294,3640],{"class":503},[151,111296,111297,111299,111301,111303,111305,111307,111309],{"class":469,"line":1418},[151,111298,65143],{"class":503},[151,111300,99095],{"class":481},[151,111302,106],{"class":503},[151,111304,99065],{"class":15210},[151,111306,1876],{"class":1869},[151,111308,67140],{"class":477},[151,111310,3640],{"class":503},[151,111312,111313,111315,111317,111319,111321],{"class":469,"line":2462},[151,111314,65163],{"class":503},[151,111316,99065],{"class":15210},[151,111318,1876],{"class":1869},[151,111320,42327],{"class":477},[151,111322,3640],{"class":503},[151,111324,111325,111327,111329,111331,111333],{"class":469,"line":2471},[151,111326,99502],{"class":503},[151,111328,99065],{"class":15210},[151,111330,1876],{"class":1869},[151,111332,42327],{"class":477},[151,111334,3640],{"class":503},[151,111336,111337,111339,111342],{"class":469,"line":2480},[151,111338,93826],{"class":503},[151,111340,111341],{"class":481},"'~/Documents/GitHub/briancaffey.github.io/img/storage/hdd_storage_vs_price.png'",[151,111343,12451],{"class":503},[11,111345,111346],{},[2718,111347],{"alt":20386,"src":111348},"/static/pcpp/storage/hdd_storage_vs_price.png",[11,111350,111351],{},"This graph shows the price and the price per GB for SSDs and HDDs:",[459,111353,111355],{"className":13136,"code":111354,"language":12886,"meta":464,"style":464},"plt.figure(figsize=(12,8))\nplt.axis([0,5100,0,3])\na=.3\nplt.scatter(ssd.storage_gb,ssd.ppgb, s= 50, alpha=a, color='black')\nplt.scatter(hdd.storage_gb,hdd.ppgb, s= 50, alpha=a, color='red')\nplt.title('Price per GB of Hard Drive vs. Storage Capacity', fontsize=14)\nplt.xlabel('Storage Capacity (GB)', fontsize=14)\nplt.ylabel('Price per GB', fontsize=14)\nplt.xticks(fontsize=13)\nplt.yticks(fontsize=13)\nplt.legend(['SSD', 'HDD'], loc='upper right', fontsize=14)\nplt.savefig(os.path.expanduser('~/Documents/GitHub/briancaffey.github.io/img/storage/storage_vs_ppgb.png'))\n",[30,111356,111357,111375,111396,111405,111434,111461,111478,111495,111511,111523,111535,111563],{"__ignoreMap":464},[151,111358,111359,111361,111363,111365,111367,111369,111371,111373],{"class":469,"line":470},[151,111360,44355],{"class":503},[151,111362,44358],{"class":15210},[151,111364,1876],{"class":1869},[151,111366,12386],{"class":503},[151,111368,42360],{"class":477},[151,111370,3634],{"class":503},[151,111372,24369],{"class":477},[151,111374,12451],{"class":503},[151,111376,111377,111379,111381,111383,111386,111388,111390,111392,111394],{"class":469,"line":488},[151,111378,99036],{"class":503},[151,111380,9181],{"class":477},[151,111382,3634],{"class":503},[151,111384,111385],{"class":477},"5100",[151,111387,3634],{"class":503},[151,111389,9181],{"class":477},[151,111391,3634],{"class":503},[151,111393,6557],{"class":477},[151,111395,38820],{"class":503},[151,111397,111398,111400,111402],{"class":469,"line":500},[151,111399,20],{"class":503},[151,111401,1876],{"class":1869},[151,111403,111404],{"class":477},".3\n",[151,111406,111407,111410,111412,111414,111417,111419,111421,111423,111426,111428,111430,111432],{"class":469,"line":509},[151,111408,111409],{"class":503},"plt.scatter(ssd.storage_gb,ssd.ppgb, ",[151,111411,55630],{"class":15210},[151,111413,1876],{"class":1869},[151,111415,111416],{"class":477}," 50",[151,111418,106],{"class":503},[151,111420,26256],{"class":15210},[151,111422,1876],{"class":1869},[151,111424,111425],{"class":503},"a, ",[151,111427,79362],{"class":15210},[151,111429,1876],{"class":1869},[151,111431,45401],{"class":481},[151,111433,3640],{"class":503},[151,111435,111436,111439,111441,111443,111445,111447,111449,111451,111453,111455,111457,111459],{"class":469,"line":517},[151,111437,111438],{"class":503},"plt.scatter(hdd.storage_gb,hdd.ppgb, ",[151,111440,55630],{"class":15210},[151,111442,1876],{"class":1869},[151,111444,111416],{"class":477},[151,111446,106],{"class":503},[151,111448,26256],{"class":15210},[151,111450,1876],{"class":1869},[151,111452,111425],{"class":503},[151,111454,79362],{"class":15210},[151,111456,1876],{"class":1869},[151,111458,80832],{"class":481},[151,111460,3640],{"class":503},[151,111462,111463,111465,111468,111470,111472,111474,111476],{"class":469,"line":534},[151,111464,65123],{"class":503},[151,111466,111467],{"class":481},"'Price per GB of Hard Drive vs. Storage Capacity'",[151,111469,106],{"class":503},[151,111471,99065],{"class":15210},[151,111473,1876],{"class":1869},[151,111475,67140],{"class":477},[151,111477,3640],{"class":503},[151,111479,111480,111482,111485,111487,111489,111491,111493],{"class":469,"line":1413},[151,111481,65133],{"class":503},[151,111483,111484],{"class":481},"'Storage Capacity (GB)'",[151,111486,106],{"class":503},[151,111488,99065],{"class":15210},[151,111490,1876],{"class":1869},[151,111492,67140],{"class":477},[151,111494,3640],{"class":503},[151,111496,111497,111499,111501,111503,111505,111507,111509],{"class":469,"line":1418},[151,111498,65143],{"class":503},[151,111500,105661],{"class":481},[151,111502,106],{"class":503},[151,111504,99065],{"class":15210},[151,111506,1876],{"class":1869},[151,111508,67140],{"class":477},[151,111510,3640],{"class":503},[151,111512,111513,111515,111517,111519,111521],{"class":469,"line":2462},[151,111514,65163],{"class":503},[151,111516,99065],{"class":15210},[151,111518,1876],{"class":1869},[151,111520,42327],{"class":477},[151,111522,3640],{"class":503},[151,111524,111525,111527,111529,111531,111533],{"class":469,"line":2471},[151,111526,99502],{"class":503},[151,111528,99065],{"class":15210},[151,111530,1876],{"class":1869},[151,111532,42327],{"class":477},[151,111534,3640],{"class":503},[151,111536,111537,111539,111541,111543,111545,111547,111549,111551,111553,111555,111557,111559,111561],{"class":469,"line":2480},[151,111538,102944],{"class":503},[151,111540,110856],{"class":481},[151,111542,106],{"class":503},[151,111544,110861],{"class":481},[151,111546,60308],{"class":503},[151,111548,104219],{"class":15210},[151,111550,1876],{"class":1869},[151,111552,104431],{"class":481},[151,111554,106],{"class":503},[151,111556,99065],{"class":15210},[151,111558,1876],{"class":1869},[151,111560,67140],{"class":477},[151,111562,3640],{"class":503},[151,111564,111565,111567,111570],{"class":469,"line":2489},[151,111566,93826],{"class":503},[151,111568,111569],{"class":481},"'~/Documents/GitHub/briancaffey.github.io/img/storage/storage_vs_ppgb.png'",[151,111571,12451],{"class":503},[11,111573,111574],{},[2718,111575],{"alt":20386,"src":111576},"/static/pcpp/storage/storage_vs_ppgb.png",[11,111578,111579],{},"Both HDDs and SSDs become less expensive per GB as they scale.",[14063,111581,111583],{"id":111582},"case-fans","Case Fans",[11,111585,111586],{},"Case fans are an important consideration for enthusiast-level PC buidlers. Case fan features include maximum RPM, maximum sound (dbA, or A-weighted decibels), maximum airflow (measured in cubic feet per minute, or CFM), fan size (mm) and static pressure (mm/H2O, a measure of pressure). These fan features are determined by the power of the fan motor and the shape, angle and spacing of the fan blades. Here are some of these features on scatter plots:",[459,111588,111590],{"className":13136,"code":111589,"language":12886,"meta":464,"style":464},"df1 = df[(df.noise>0)&(df.max_flow>0)&(df.rpm_max>0)&(df.avg>0)&(df.avg\u003C75)]\nplt.figure(figsize=(12,8))\nplt.scatter(df1.rpm_max, df1.noise, c=df1.avg, cmap='Blues', s=100)\nplt.colorbar(label='Price')\nplt.axis([0,10000,0,70])\nplt.xlabel('Maximum RPM', fontsize=14)\nplt.ylabel('Maximum Noise (dbA)', fontsize=14)\nplt.title('Case Fan Maximum RPM vs. Maximum Noise (dbA) and Price (color)', fontsize=14)\nplt.savefig(os.path.expanduser('~/Documents/GitHub/briancaffey.github.io/img/fan/rpm_vs_noise.png'))\n",[30,111591,111592,111649,111667,111695,111707,111727,111743,111760,111777],{"__ignoreMap":464},[151,111593,111594,111596,111598,111601,111603,111605,111607,111609,111612,111614,111616,111618,111620,111623,111625,111627,111629,111631,111633,111635,111637,111639,111641,111643,111645,111647],{"class":469,"line":470},[151,111595,86777],{"class":503},[151,111597,1876],{"class":1869},[151,111599,111600],{"class":503}," df[(df.noise",[151,111602,3663],{"class":1869},[151,111604,9181],{"class":477},[151,111606,748],{"class":503},[151,111608,54214],{"class":1869},[151,111610,111611],{"class":503},"(df.max_flow",[151,111613,3663],{"class":1869},[151,111615,9181],{"class":477},[151,111617,748],{"class":503},[151,111619,54214],{"class":1869},[151,111621,111622],{"class":503},"(df.rpm_max",[151,111624,3663],{"class":1869},[151,111626,9181],{"class":477},[151,111628,748],{"class":503},[151,111630,54214],{"class":1869},[151,111632,100313],{"class":503},[151,111634,3663],{"class":1869},[151,111636,9181],{"class":477},[151,111638,748],{"class":503},[151,111640,54214],{"class":1869},[151,111642,100313],{"class":503},[151,111644,3613],{"class":1869},[151,111646,88018],{"class":477},[151,111648,44576],{"class":503},[151,111650,111651,111653,111655,111657,111659,111661,111663,111665],{"class":469,"line":488},[151,111652,44355],{"class":503},[151,111654,44358],{"class":15210},[151,111656,1876],{"class":1869},[151,111658,12386],{"class":503},[151,111660,42360],{"class":477},[151,111662,3634],{"class":503},[151,111664,24369],{"class":477},[151,111666,12451],{"class":503},[151,111668,111669,111672,111674,111676,111679,111681,111683,111685,111687,111689,111691,111693],{"class":469,"line":500},[151,111670,111671],{"class":503},"plt.scatter(df1.rpm_max, df1.noise, ",[151,111673,65290],{"class":15210},[151,111675,1876],{"class":1869},[151,111677,111678],{"class":503},"df1.avg, ",[151,111680,103551],{"class":15210},[151,111682,1876],{"class":1869},[151,111684,103556],{"class":481},[151,111686,106],{"class":503},[151,111688,55630],{"class":15210},[151,111690,1876],{"class":1869},[151,111692,71821],{"class":477},[151,111694,3640],{"class":503},[151,111696,111697,111699,111701,111703,111705],{"class":469,"line":509},[151,111698,103563],{"class":503},[151,111700,103566],{"class":15210},[151,111702,1876],{"class":1869},[151,111704,99095],{"class":481},[151,111706,3640],{"class":503},[151,111708,111709,111711,111713,111715,111717,111719,111721,111723,111725],{"class":469,"line":517},[151,111710,99036],{"class":503},[151,111712,9181],{"class":477},[151,111714,3634],{"class":503},[151,111716,45984],{"class":477},[151,111718,3634],{"class":503},[151,111720,9181],{"class":477},[151,111722,3634],{"class":503},[151,111724,73169],{"class":477},[151,111726,38820],{"class":503},[151,111728,111729,111731,111733,111735,111737,111739,111741],{"class":469,"line":534},[151,111730,65133],{"class":503},[151,111732,104806],{"class":481},[151,111734,106],{"class":503},[151,111736,99065],{"class":15210},[151,111738,1876],{"class":1869},[151,111740,67140],{"class":477},[151,111742,3640],{"class":503},[151,111744,111745,111747,111750,111752,111754,111756,111758],{"class":469,"line":1413},[151,111746,65143],{"class":503},[151,111748,111749],{"class":481},"'Maximum Noise (dbA)'",[151,111751,106],{"class":503},[151,111753,99065],{"class":15210},[151,111755,1876],{"class":1869},[151,111757,67140],{"class":477},[151,111759,3640],{"class":503},[151,111761,111762,111764,111767,111769,111771,111773,111775],{"class":469,"line":1418},[151,111763,65123],{"class":503},[151,111765,111766],{"class":481},"'Case Fan Maximum RPM vs. Maximum Noise (dbA) and Price (color)'",[151,111768,106],{"class":503},[151,111770,99065],{"class":15210},[151,111772,1876],{"class":1869},[151,111774,67140],{"class":477},[151,111776,3640],{"class":503},[151,111778,111779,111781,111784],{"class":469,"line":2462},[151,111780,93826],{"class":503},[151,111782,111783],{"class":481},"'~/Documents/GitHub/briancaffey.github.io/img/fan/rpm_vs_noise.png'",[151,111785,12451],{"class":503},[11,111787,111788],{},[2718,111789],{"alt":20386,"src":111790},"/static/pcpp/fan/rpm_vs_noise.png",[11,111792,111793],{},"260 of 1192 case fans in this dataset include a rating for static pressure. Static pressure can be thought of as the force by which air is pushed out of the fan. If you put your hand in front of a fan with low static pressure, you will feel a gentle flow of air. Fans with high static pressure have stronger airflow, but not necessarily more airflow, as measured in CFM.",[11,111795,111796],{},"This graph shows air flow plotted against static pressure, and the weak correlation between the two variables:",[459,111798,111800],{"className":13136,"code":111799,"language":12886,"meta":464,"style":464},"df5 = df[(df.avg>0)&(df.max_flow>0)&(df.static_pressure>0)&(df.rpm_max>0)&(df.static_pressure\u003C15)]\n\nfrom sklearn.linear_model import LinearRegression\nlreg = LinearRegression()\n\nX = df5.max_flow.reshape(df5.max_flow.shape[0],1)\ny = df5.static_pressure.reshape(df5.static_pressure.shape[0],1)\nlreg.fit(X, y, sample_weight=None)\n\nplt.figure(figsize=(12,8))\nplt.scatter(df5.max_flow, df5.static_pressure, s=75)\nplt.title('Air Flow (CFM) vs. Static Pressure (mm/H2O)', fontsize=14)\nplt.xlabel('Air Flow in CFM (cubic feet per minute)', fontsize=14)\nplt.ylabel('Static Pressure (mm/H2O)', fontsize=14)\nplt.xticks(fontsize=13)\nplt.yticks(fontsize=13)\nplt.axis([0,200,0,8])\n\n#plot regression line\nflow = df5.max_flow.reshape(df5.max_flow.shape[0],1)\npred = lreg.predict(df5.max_flow.reshape(df5.max_flow.shape[0],1))\nplt.plot(flow ,pred, color='red')\nplt.savefig(os.path.expanduser('~/Documents/GitHub/briancaffey.github.io/img/fan/air_flow_v_static_pressure.png'))\n\nprint lreg.score(X, y, sample_weight=None)\n",[30,111801,111802,111858,111862,111872,111880,111884,111901,111918,111930,111934,111952,111965,111982,111999,112016,112028,112040,112060,112064,112068,112085,112102,112115,112124,112128],{"__ignoreMap":464},[151,111803,111804,111807,111809,111811,111813,111815,111817,111819,111821,111823,111825,111827,111829,111832,111834,111836,111838,111840,111842,111844,111846,111848,111850,111852,111854,111856],{"class":469,"line":470},[151,111805,111806],{"class":503},"df5 ",[151,111808,1876],{"class":1869},[151,111810,100420],{"class":503},[151,111812,3663],{"class":1869},[151,111814,9181],{"class":477},[151,111816,748],{"class":503},[151,111818,54214],{"class":1869},[151,111820,111611],{"class":503},[151,111822,3663],{"class":1869},[151,111824,9181],{"class":477},[151,111826,748],{"class":503},[151,111828,54214],{"class":1869},[151,111830,111831],{"class":503},"(df.static_pressure",[151,111833,3663],{"class":1869},[151,111835,9181],{"class":477},[151,111837,748],{"class":503},[151,111839,54214],{"class":1869},[151,111841,111622],{"class":503},[151,111843,3663],{"class":1869},[151,111845,9181],{"class":477},[151,111847,748],{"class":503},[151,111849,54214],{"class":1869},[151,111851,111831],{"class":503},[151,111853,3613],{"class":1869},[151,111855,42310],{"class":477},[151,111857,44576],{"class":503},[151,111859,111860],{"class":469,"line":488},[151,111861,1090],{"emptyLinePlaceholder":609},[151,111863,111864,111866,111868,111870],{"class":469,"line":500},[151,111865,16853],{"class":1869},[151,111867,100477],{"class":503},[151,111869,16859],{"class":1869},[151,111871,100482],{"class":503},[151,111873,111874,111876,111878],{"class":469,"line":509},[151,111875,104916],{"class":503},[151,111877,1876],{"class":1869},[151,111879,100492],{"class":503},[151,111881,111882],{"class":469,"line":517},[151,111883,1090],{"emptyLinePlaceholder":609},[151,111885,111886,111888,111890,111893,111895,111897,111899],{"class":469,"line":534},[151,111887,87698],{"class":503},[151,111889,1876],{"class":1869},[151,111891,111892],{"class":503}," df5.max_flow.reshape(df5.max_flow.shape[",[151,111894,9181],{"class":477},[151,111896,104974],{"class":503},[151,111898,6760],{"class":477},[151,111900,3640],{"class":503},[151,111902,111903,111905,111907,111910,111912,111914,111916],{"class":469,"line":1413},[151,111904,98878],{"class":503},[151,111906,1876],{"class":1869},[151,111908,111909],{"class":503}," df5.static_pressure.reshape(df5.static_pressure.shape[",[151,111911,9181],{"class":477},[151,111913,104974],{"class":503},[151,111915,6760],{"class":477},[151,111917,3640],{"class":503},[151,111919,111920,111922,111924,111926,111928],{"class":469,"line":1418},[151,111921,104983],{"class":503},[151,111923,104986],{"class":15210},[151,111925,1876],{"class":1869},[151,111927,15437],{"class":477},[151,111929,3640],{"class":503},[151,111931,111932],{"class":469,"line":2462},[151,111933,1090],{"emptyLinePlaceholder":609},[151,111935,111936,111938,111940,111942,111944,111946,111948,111950],{"class":469,"line":2471},[151,111937,44355],{"class":503},[151,111939,44358],{"class":15210},[151,111941,1876],{"class":1869},[151,111943,12386],{"class":503},[151,111945,42360],{"class":477},[151,111947,3634],{"class":503},[151,111949,24369],{"class":477},[151,111951,12451],{"class":503},[151,111953,111954,111957,111959,111961,111963],{"class":469,"line":2480},[151,111955,111956],{"class":503},"plt.scatter(df5.max_flow, df5.static_pressure, ",[151,111958,55630],{"class":15210},[151,111960,1876],{"class":1869},[151,111962,88018],{"class":477},[151,111964,3640],{"class":503},[151,111966,111967,111969,111972,111974,111976,111978,111980],{"class":469,"line":2489},[151,111968,65123],{"class":503},[151,111970,111971],{"class":481},"'Air Flow (CFM) vs. Static Pressure (mm/H2O)'",[151,111973,106],{"class":503},[151,111975,99065],{"class":15210},[151,111977,1876],{"class":1869},[151,111979,67140],{"class":477},[151,111981,3640],{"class":503},[151,111983,111984,111986,111989,111991,111993,111995,111997],{"class":469,"line":2497},[151,111985,65133],{"class":503},[151,111987,111988],{"class":481},"'Air Flow in CFM (cubic feet per minute)'",[151,111990,106],{"class":503},[151,111992,99065],{"class":15210},[151,111994,1876],{"class":1869},[151,111996,67140],{"class":477},[151,111998,3640],{"class":503},[151,112000,112001,112003,112006,112008,112010,112012,112014],{"class":469,"line":3140},[151,112002,65143],{"class":503},[151,112004,112005],{"class":481},"'Static Pressure (mm/H2O)'",[151,112007,106],{"class":503},[151,112009,99065],{"class":15210},[151,112011,1876],{"class":1869},[151,112013,67140],{"class":477},[151,112015,3640],{"class":503},[151,112017,112018,112020,112022,112024,112026],{"class":469,"line":3149},[151,112019,65163],{"class":503},[151,112021,99065],{"class":15210},[151,112023,1876],{"class":1869},[151,112025,42327],{"class":477},[151,112027,3640],{"class":503},[151,112029,112030,112032,112034,112036,112038],{"class":469,"line":3158},[151,112031,99502],{"class":503},[151,112033,99065],{"class":15210},[151,112035,1876],{"class":1869},[151,112037,42327],{"class":477},[151,112039,3640],{"class":503},[151,112041,112042,112044,112046,112048,112050,112052,112054,112056,112058],{"class":469,"line":3167},[151,112043,99036],{"class":503},[151,112045,9181],{"class":477},[151,112047,3634],{"class":503},[151,112049,41624],{"class":477},[151,112051,3634],{"class":503},[151,112053,9181],{"class":477},[151,112055,3634],{"class":503},[151,112057,24369],{"class":477},[151,112059,38820],{"class":503},[151,112061,112062],{"class":469,"line":3175},[151,112063,1090],{"emptyLinePlaceholder":609},[151,112065,112066],{"class":469,"line":3184},[151,112067,105134],{"class":1527},[151,112069,112070,112073,112075,112077,112079,112081,112083],{"class":469,"line":3193},[151,112071,112072],{"class":503},"flow ",[151,112074,1876],{"class":1869},[151,112076,111892],{"class":503},[151,112078,9181],{"class":477},[151,112080,104974],{"class":503},[151,112082,6760],{"class":477},[151,112084,3640],{"class":503},[151,112086,112087,112089,112091,112094,112096,112098,112100],{"class":469,"line":3720},[151,112088,105157],{"class":503},[151,112090,1876],{"class":1869},[151,112092,112093],{"class":503}," lreg.predict(df5.max_flow.reshape(df5.max_flow.shape[",[151,112095,9181],{"class":477},[151,112097,104974],{"class":503},[151,112099,6760],{"class":477},[151,112101,12451],{"class":503},[151,112103,112104,112107,112109,112111,112113],{"class":469,"line":3729},[151,112105,112106],{"class":503},"plt.plot(flow ,pred, ",[151,112108,79362],{"class":15210},[151,112110,1876],{"class":1869},[151,112112,80832],{"class":481},[151,112114,3640],{"class":503},[151,112116,112117,112119,112122],{"class":469,"line":3735},[151,112118,93826],{"class":503},[151,112120,112121],{"class":481},"'~/Documents/GitHub/briancaffey.github.io/img/fan/air_flow_v_static_pressure.png'",[151,112123,12451],{"class":503},[151,112125,112126],{"class":469,"line":3745},[151,112127,1090],{"emptyLinePlaceholder":609},[151,112129,112130,112132,112135,112137,112139,112141],{"class":469,"line":3754},[151,112131,18513],{"class":2226},[151,112133,112134],{"class":503}," lreg.score(X, y, ",[151,112136,104986],{"class":15210},[151,112138,1876],{"class":1869},[151,112140,15437],{"class":477},[151,112142,3640],{"class":503},[11,112144,112145],{},[30,112146,112147],{},"0.0345903950712",[11,112149,112150],{},[2718,112151],{"alt":20386,"src":112152},"/static/pcpp/fan/air_flow_v_static_pressure.png",[11,112154,112155],{},"Two fan features with a strong correlation are maximum RPM and static pressure:",[459,112157,112159],{"className":13136,"code":112158,"language":12886,"meta":464,"style":464},"df5 = df[(df.avg>0)&(df.max_flow>0)&(df.static_pressure>0)&(df.rpm_max>0)&(df.static_pressure\u003C15)]\n\nfrom sklearn.linear_model import LinearRegression\nlreg = LinearRegression()\nX = df5.rpm_max.reshape(df5.rpm_max.shape[0],1)\ny = df5.static_pressure.reshape(df5.static_pressure.shape[0],1)\nlreg.fit(X, y, sample_weight=None)\n\nplt.figure(figsize=(12,8))\nplt.scatter(df5.rpm_max, df5.static_pressure, s=75)\n\nplt.axis([0,8000,0,16])\nplt.title('Maximum RPM vs. Static Pressure (mm/H2O)', fontsize=14)\nplt.xlabel('Maximum RPM', fontsize=14)\nplt.ylabel('Static Pressure (mm/H2O)', fontsize=14)\nplt.xticks(fontsize=13)\nplt.yticks(fontsize=13)\n\n#plot regression line\nsize = df5.rpm_max.reshape(df5.rpm_max.shape[0],1)\npred = lreg.predict(df5.rpm_max.reshape(df5.rpm_max.shape[0],1))\nplt.plot(size ,pred, color='red')\nplt.savefig(os.path.expanduser('~/Documents/GitHub/briancaffey.github.io/img/fan/rpm_max_vs_static_pressure.png'))\nprint lreg.score(X,y,sample_weight=None)\n",[30,112160,112161,112215,112219,112229,112237,112254,112270,112282,112286,112304,112317,112321,112341,112358,112374,112390,112402,112414,112418,112422,112438,112455,112467,112476],{"__ignoreMap":464},[151,112162,112163,112165,112167,112169,112171,112173,112175,112177,112179,112181,112183,112185,112187,112189,112191,112193,112195,112197,112199,112201,112203,112205,112207,112209,112211,112213],{"class":469,"line":470},[151,112164,111806],{"class":503},[151,112166,1876],{"class":1869},[151,112168,100420],{"class":503},[151,112170,3663],{"class":1869},[151,112172,9181],{"class":477},[151,112174,748],{"class":503},[151,112176,54214],{"class":1869},[151,112178,111611],{"class":503},[151,112180,3663],{"class":1869},[151,112182,9181],{"class":477},[151,112184,748],{"class":503},[151,112186,54214],{"class":1869},[151,112188,111831],{"class":503},[151,112190,3663],{"class":1869},[151,112192,9181],{"class":477},[151,112194,748],{"class":503},[151,112196,54214],{"class":1869},[151,112198,111622],{"class":503},[151,112200,3663],{"class":1869},[151,112202,9181],{"class":477},[151,112204,748],{"class":503},[151,112206,54214],{"class":1869},[151,112208,111831],{"class":503},[151,112210,3613],{"class":1869},[151,112212,42310],{"class":477},[151,112214,44576],{"class":503},[151,112216,112217],{"class":469,"line":488},[151,112218,1090],{"emptyLinePlaceholder":609},[151,112220,112221,112223,112225,112227],{"class":469,"line":500},[151,112222,16853],{"class":1869},[151,112224,100477],{"class":503},[151,112226,16859],{"class":1869},[151,112228,100482],{"class":503},[151,112230,112231,112233,112235],{"class":469,"line":509},[151,112232,104916],{"class":503},[151,112234,1876],{"class":1869},[151,112236,100492],{"class":503},[151,112238,112239,112241,112243,112246,112248,112250,112252],{"class":469,"line":517},[151,112240,87698],{"class":503},[151,112242,1876],{"class":1869},[151,112244,112245],{"class":503}," df5.rpm_max.reshape(df5.rpm_max.shape[",[151,112247,9181],{"class":477},[151,112249,104974],{"class":503},[151,112251,6760],{"class":477},[151,112253,3640],{"class":503},[151,112255,112256,112258,112260,112262,112264,112266,112268],{"class":469,"line":534},[151,112257,98878],{"class":503},[151,112259,1876],{"class":1869},[151,112261,111909],{"class":503},[151,112263,9181],{"class":477},[151,112265,104974],{"class":503},[151,112267,6760],{"class":477},[151,112269,3640],{"class":503},[151,112271,112272,112274,112276,112278,112280],{"class":469,"line":1413},[151,112273,104983],{"class":503},[151,112275,104986],{"class":15210},[151,112277,1876],{"class":1869},[151,112279,15437],{"class":477},[151,112281,3640],{"class":503},[151,112283,112284],{"class":469,"line":1418},[151,112285,1090],{"emptyLinePlaceholder":609},[151,112287,112288,112290,112292,112294,112296,112298,112300,112302],{"class":469,"line":2462},[151,112289,44355],{"class":503},[151,112291,44358],{"class":15210},[151,112293,1876],{"class":1869},[151,112295,12386],{"class":503},[151,112297,42360],{"class":477},[151,112299,3634],{"class":503},[151,112301,24369],{"class":477},[151,112303,12451],{"class":503},[151,112305,112306,112309,112311,112313,112315],{"class":469,"line":2471},[151,112307,112308],{"class":503},"plt.scatter(df5.rpm_max, df5.static_pressure, ",[151,112310,55630],{"class":15210},[151,112312,1876],{"class":1869},[151,112314,88018],{"class":477},[151,112316,3640],{"class":503},[151,112318,112319],{"class":469,"line":2480},[151,112320,1090],{"emptyLinePlaceholder":609},[151,112322,112323,112325,112327,112329,112331,112333,112335,112337,112339],{"class":469,"line":2489},[151,112324,99036],{"class":503},[151,112326,9181],{"class":477},[151,112328,3634],{"class":503},[151,112330,13103],{"class":477},[151,112332,3634],{"class":503},[151,112334,9181],{"class":477},[151,112336,3634],{"class":503},[151,112338,87061],{"class":477},[151,112340,38820],{"class":503},[151,112342,112343,112345,112348,112350,112352,112354,112356],{"class":469,"line":2497},[151,112344,65123],{"class":503},[151,112346,112347],{"class":481},"'Maximum RPM vs. Static Pressure (mm/H2O)'",[151,112349,106],{"class":503},[151,112351,99065],{"class":15210},[151,112353,1876],{"class":1869},[151,112355,67140],{"class":477},[151,112357,3640],{"class":503},[151,112359,112360,112362,112364,112366,112368,112370,112372],{"class":469,"line":3140},[151,112361,65133],{"class":503},[151,112363,104806],{"class":481},[151,112365,106],{"class":503},[151,112367,99065],{"class":15210},[151,112369,1876],{"class":1869},[151,112371,67140],{"class":477},[151,112373,3640],{"class":503},[151,112375,112376,112378,112380,112382,112384,112386,112388],{"class":469,"line":3149},[151,112377,65143],{"class":503},[151,112379,112005],{"class":481},[151,112381,106],{"class":503},[151,112383,99065],{"class":15210},[151,112385,1876],{"class":1869},[151,112387,67140],{"class":477},[151,112389,3640],{"class":503},[151,112391,112392,112394,112396,112398,112400],{"class":469,"line":3158},[151,112393,65163],{"class":503},[151,112395,99065],{"class":15210},[151,112397,1876],{"class":1869},[151,112399,42327],{"class":477},[151,112401,3640],{"class":503},[151,112403,112404,112406,112408,112410,112412],{"class":469,"line":3167},[151,112405,99502],{"class":503},[151,112407,99065],{"class":15210},[151,112409,1876],{"class":1869},[151,112411,42327],{"class":477},[151,112413,3640],{"class":503},[151,112415,112416],{"class":469,"line":3175},[151,112417,1090],{"emptyLinePlaceholder":609},[151,112419,112420],{"class":469,"line":3184},[151,112421,105134],{"class":1527},[151,112423,112424,112426,112428,112430,112432,112434,112436],{"class":469,"line":3193},[151,112425,105139],{"class":503},[151,112427,1876],{"class":1869},[151,112429,112245],{"class":503},[151,112431,9181],{"class":477},[151,112433,104974],{"class":503},[151,112435,6760],{"class":477},[151,112437,3640],{"class":503},[151,112439,112440,112442,112444,112447,112449,112451,112453],{"class":469,"line":3720},[151,112441,105157],{"class":503},[151,112443,1876],{"class":1869},[151,112445,112446],{"class":503}," lreg.predict(df5.rpm_max.reshape(df5.rpm_max.shape[",[151,112448,9181],{"class":477},[151,112450,104974],{"class":503},[151,112452,6760],{"class":477},[151,112454,12451],{"class":503},[151,112456,112457,112459,112461,112463,112465],{"class":469,"line":3729},[151,112458,105175],{"class":503},[151,112460,79362],{"class":15210},[151,112462,1876],{"class":1869},[151,112464,80832],{"class":481},[151,112466,3640],{"class":503},[151,112468,112469,112471,112474],{"class":469,"line":3735},[151,112470,93826],{"class":503},[151,112472,112473],{"class":481},"'~/Documents/GitHub/briancaffey.github.io/img/fan/rpm_max_vs_static_pressure.png'",[151,112475,12451],{"class":503},[151,112477,112478,112480,112483,112485,112487,112489],{"class":469,"line":3745},[151,112479,18513],{"class":2226},[151,112481,112482],{"class":503}," lreg.score(X,y,",[151,112484,104986],{"class":15210},[151,112486,1876],{"class":1869},[151,112488,15437],{"class":477},[151,112490,3640],{"class":503},[11,112492,112493],{},[30,112494,112495],{},"0.711316685518",[11,112497,112498],{},[2718,112499],{"alt":20386,"src":112500},"/static/pcpp/fan/rpm_max_vs_static_pressure.png",[11,112502,112503],{},"The stronger force of the air flow from high static pressure fans is better at blowing hot air off of hot surfaces, so these fans are often attached to liquid CPU cooler radiators and CPU cooler heat blocks.",[11,112505,112506],{},"As we discussed TDP and CPU coolers, we know that managing heat well will improve performance. Every part of a PC generates heat. Some memory sticks even include heat shrouds to help keep temperatures down. Software utilities are available to monitor the temperature of every component in a PC build, and dynamically change the fan speeds to help keep temperatures low and framerates high.",[14063,112508,112510],{"id":112509},"monitors","Monitors",[11,112512,112513],{},"Monitors come in all shapes and sizes. Here's a look at the different dimensions of monitors listed:",[459,112515,112517],{"className":13136,"code":112516,"language":12886,"meta":464,"style":464},"plt.figure(figsize=(12,8))\nplt.scatter(df.screen_x, df.screen_y, alpha=0.01, s=150)\nplt.gca().set_aspect('equal', adjustable='box')\nplt.axis([0,35,0,35./2])\nplt.title('Screen Size')\nplt.xlabel('Screen Width (inches)', fontsize=14)\nplt.ylabel('Screen Height (inches)', fontsize=14)\nplt.xticks(fontsize=13)\nplt.yticks(fontsize=13)\nplt.savefig(os.path.expanduser('~/Documents/GitHub/briancaffey.github.io/img/monitor/screen_size.png'))\n",[30,112518,112519,112537,112559,112579,112605,112614,112631,112648,112660,112672],{"__ignoreMap":464},[151,112520,112521,112523,112525,112527,112529,112531,112533,112535],{"class":469,"line":470},[151,112522,44355],{"class":503},[151,112524,44358],{"class":15210},[151,112526,1876],{"class":1869},[151,112528,12386],{"class":503},[151,112530,42360],{"class":477},[151,112532,3634],{"class":503},[151,112534,24369],{"class":477},[151,112536,12451],{"class":503},[151,112538,112539,112542,112544,112546,112549,112551,112553,112555,112557],{"class":469,"line":488},[151,112540,112541],{"class":503},"plt.scatter(df.screen_x, df.screen_y, ",[151,112543,26256],{"class":15210},[151,112545,1876],{"class":1869},[151,112547,112548],{"class":477},"0.01",[151,112550,106],{"class":503},[151,112552,55630],{"class":15210},[151,112554,1876],{"class":1869},[151,112556,45949],{"class":477},[151,112558,3640],{"class":503},[151,112560,112561,112564,112567,112569,112572,112574,112577],{"class":469,"line":500},[151,112562,112563],{"class":503},"plt.gca().set_aspect(",[151,112565,112566],{"class":481},"'equal'",[151,112568,106],{"class":503},[151,112570,112571],{"class":15210},"adjustable",[151,112573,1876],{"class":1869},[151,112575,112576],{"class":481},"'box'",[151,112578,3640],{"class":503},[151,112580,112581,112583,112585,112587,112589,112591,112593,112595,112597,112599,112601,112603],{"class":469,"line":509},[151,112582,99036],{"class":503},[151,112584,9181],{"class":477},[151,112586,3634],{"class":503},[151,112588,41984],{"class":477},[151,112590,3634],{"class":503},[151,112592,9181],{"class":477},[151,112594,3634],{"class":503},[151,112596,41984],{"class":477},[151,112598,643],{"class":503},[151,112600,19883],{"class":1869},[151,112602,6619],{"class":477},[151,112604,38820],{"class":503},[151,112606,112607,112609,112612],{"class":469,"line":517},[151,112608,65123],{"class":503},[151,112610,112611],{"class":481},"'Screen Size'",[151,112613,3640],{"class":503},[151,112615,112616,112618,112621,112623,112625,112627,112629],{"class":469,"line":534},[151,112617,65133],{"class":503},[151,112619,112620],{"class":481},"'Screen Width (inches)'",[151,112622,106],{"class":503},[151,112624,99065],{"class":15210},[151,112626,1876],{"class":1869},[151,112628,67140],{"class":477},[151,112630,3640],{"class":503},[151,112632,112633,112635,112638,112640,112642,112644,112646],{"class":469,"line":1413},[151,112634,65143],{"class":503},[151,112636,112637],{"class":481},"'Screen Height (inches)'",[151,112639,106],{"class":503},[151,112641,99065],{"class":15210},[151,112643,1876],{"class":1869},[151,112645,67140],{"class":477},[151,112647,3640],{"class":503},[151,112649,112650,112652,112654,112656,112658],{"class":469,"line":1418},[151,112651,65163],{"class":503},[151,112653,99065],{"class":15210},[151,112655,1876],{"class":1869},[151,112657,42327],{"class":477},[151,112659,3640],{"class":503},[151,112661,112662,112664,112666,112668,112670],{"class":469,"line":2462},[151,112663,99502],{"class":503},[151,112665,99065],{"class":15210},[151,112667,1876],{"class":1869},[151,112669,42327],{"class":477},[151,112671,3640],{"class":503},[151,112673,112674,112676,112679],{"class":469,"line":2471},[151,112675,93826],{"class":503},[151,112677,112678],{"class":481},"'~/Documents/GitHub/briancaffey.github.io/img/monitor/screen_size.png'",[151,112680,12451],{"class":503},[11,112682,112683],{},[2718,112684],{"alt":20386,"src":112685},"/static/pcpp/monitor/screen_size.png",[11,112687,112688,112689,112691,112692,112695,112696,112699,112700,112702],{},"Monitors are measured not only in size, but in a number of different measures related to ",[51,112690,59358],{},". Framerate refers to how many frames per second are rendered in programs, typically games. A display monitor can show as many frames per second as its ",[51,112693,112694],{},"refresh rate"," allows. Most modern games and monitors use a technology called V-Sync (vertical-sync), which limits the ",[51,112697,112698],{},"frame rate"," to the maximum ",[51,112701,112694],{}," that the monitor can handle.",[11,112704,112705,112706,208],{},"Input lag and response time are two other important monitor metrics, particullarly for competitive gamers. Input lag refers to the amount of time it takes for user input from peripherals, such as mouse and keyboard, to be reflected on the screen. Input lag typically ranges from (range), and can make all of the difference in FPS games (first-person shooters). Here's an explination of response time from an enthusiast monitor site, ",[20,112707,112710],{"href":112708,"rel":112709},"http://www.144hzmonitors.com/knowledge-base/what-does-response-time/",[24],"www.144hzmonitors.com",[210,112712,112713],{},[11,112714,112715],{},"Response time is a measure of quickly a pixel can display a change from either black to white or from one shade of gray to another. Lower response times are better. Normal response time right now is 1ms for TN panels and 4ms for IPS panels.",[11,112717,112718],{},"Response time is an important metric for playing fast-paced games and watching action movies. If pixels are not able to fully change color in between the time of each from (typically 17 ms for 60Hz monitors), then images displayed may appear with blurred motion trailing certain objects displayed on screen.",[11,112720,112721],{},"There are lots of features for monitors that would seem to be good predictors of price, but there is too much missing data to draw meaningful conclusions. Instead, we can look at another dimension of the pricing that we still haven't explored.",[56,112723,112725],{"id":112724},"vendor-data","Vendor Data",[11,112727,112728],{},"In the graphs and models used above for exploring various types of PC components, the price associated with an individual part is the average of prices offered by any number of vendors. For example, one CPU may be sold by Amazon for $150.00, NewEgg for $145.00 and NCIX US for $170.00. This CPU would be priced at $155 in the dataset.",[11,112730,112731],{},"I'm very interested in understanding how the different vendors compare to one another on product offering prices. My intuition and (experience using the site) tells me that Amazon has the best deals overall, but I would like to find a way to show this, and possibly rank and score the vendors against one another.",[11,112733,112734],{},"Here's a simple method I've devised for comparing vendors:",[459,112736,112738],{"className":13136,"code":112737,"language":12886,"meta":464,"style":464},"df['Pricing_'] = [ast.literal_eval(x) for x in df.Pricing]\n\ndef score_vendors(prices):\n    mean = np.mean([prices[x] for x in prices])\n    std = np.std([prices[x] for x in prices])\n    def z_score(std, mean, price):\n        diff = float(price - mean)\n        z_score = diff/std\n        return z_score\n    vendor_dict = {}\n    for x in prices:\n        vendor_z_score = z_score(std, mean, prices[x])\n        vendor_dict[x] = vendor_z_score\n    return vendor_dict\n\nvendor_z_score_list = []\nfor x in df.Pricing_:\n    z_score_dict = score_vendors(x)\n    vendor_z_score_list.append(z_score_dict)\n\nvend_df = pd.DataFrame(vendor_z_score_list)\n\nvendor_avg = {}\nfor x in vend_df.columns:\n    vendor_avg[x] = vend_df[x].dropna().sum()/vend_df[x].dropna().count()\n\nvendor_avg_chart = pd.DataFrame(vendor_avg, index=[0])\n\nvendor_avg_chart.T.sort_values(by=0, ascending=True).plot(kind='bar', figsize=(12,8))\nplt.title('Average Z-score for Monitor prices by Vendor', fontsize=14)\nplt.ylabel('Count', fontsize=14)\nplt.xlabel('Vendor', fontsize=14)\nplt.xticks(fontsize=13)\nplt.yticks(fontsize=13)\nplt.legend(['Average Z-score'], loc='upper left', fontsize=14)\n",[30,112739,112740,112763,112767,112781,112800,112818,112841,112858,112873,112880,112889,112900,112910,112920,112927,112931,112940,112951,112961,112966,112970,112980,112984,112993,113004,113019,113023,113043,113047,113091,113108,113124,113141,113153,113165],{"__ignoreMap":464},[151,112741,112742,112744,112747,112749,112751,112754,112756,112758,112760],{"class":469,"line":470},[151,112743,70736],{"class":503},[151,112745,112746],{"class":481},"'Pricing_'",[151,112748,16654],{"class":503},[151,112750,1876],{"class":1869},[151,112752,112753],{"class":503}," [ast.literal_eval(x) ",[151,112755,16732],{"class":1869},[151,112757,44552],{"class":503},[151,112759,16417],{"class":1869},[151,112761,112762],{"class":503}," df.Pricing]\n",[151,112764,112765],{"class":469,"line":488},[151,112766,1090],{"emptyLinePlaceholder":609},[151,112768,112769,112771,112774,112776,112779],{"class":469,"line":500},[151,112770,16925],{"class":12347},[151,112772,112773],{"class":473}," score_vendors",[151,112775,12386],{"class":503},[151,112777,112778],{"class":15232},"prices",[151,112780,15264],{"class":503},[151,112782,112783,112786,112788,112791,112793,112795,112797],{"class":469,"line":509},[151,112784,112785],{"class":503},"    mean ",[151,112787,1876],{"class":1869},[151,112789,112790],{"class":503}," np.mean([prices[x] ",[151,112792,16732],{"class":1869},[151,112794,44552],{"class":503},[151,112796,16417],{"class":1869},[151,112798,112799],{"class":503}," prices])\n",[151,112801,112802,112805,112807,112810,112812,112814,112816],{"class":469,"line":517},[151,112803,112804],{"class":503},"    std ",[151,112806,1876],{"class":1869},[151,112808,112809],{"class":503}," np.std([prices[x] ",[151,112811,16732],{"class":1869},[151,112813,44552],{"class":503},[151,112815,16417],{"class":1869},[151,112817,112799],{"class":503},[151,112819,112820,112822,112825,112827,112829,112831,112834,112836,112839],{"class":469,"line":534},[151,112821,16566],{"class":12347},[151,112823,112824],{"class":473}," z_score",[151,112826,12386],{"class":503},[151,112828,23276],{"class":15232},[151,112830,106],{"class":503},[151,112832,112833],{"class":15232},"mean",[151,112835,106],{"class":503},[151,112837,112838],{"class":15232},"price",[151,112840,15264],{"class":503},[151,112842,112843,112846,112848,112850,112853,112855],{"class":469,"line":1413},[151,112844,112845],{"class":503},"        diff ",[151,112847,1876],{"class":1869},[151,112849,96658],{"class":6205},[151,112851,112852],{"class":503},"(price ",[151,112854,12445],{"class":1869},[151,112856,112857],{"class":503}," mean)\n",[151,112859,112860,112863,112865,112868,112870],{"class":469,"line":1418},[151,112861,112862],{"class":503},"        z_score ",[151,112864,1876],{"class":1869},[151,112866,112867],{"class":503}," diff",[151,112869,19883],{"class":1869},[151,112871,112872],{"class":503},"std\n",[151,112874,112875,112877],{"class":469,"line":2462},[151,112876,16833],{"class":1869},[151,112878,112879],{"class":503}," z_score\n",[151,112881,112882,112885,112887],{"class":469,"line":2471},[151,112883,112884],{"class":503},"    vendor_dict ",[151,112886,1876],{"class":1869},[151,112888,16634],{"class":503},[151,112890,112891,112893,112895,112897],{"class":469,"line":2480},[151,112892,16411],{"class":1869},[151,112894,44552],{"class":503},[151,112896,16417],{"class":1869},[151,112898,112899],{"class":503}," prices:\n",[151,112901,112902,112905,112907],{"class":469,"line":2489},[151,112903,112904],{"class":503},"        vendor_z_score ",[151,112906,1876],{"class":1869},[151,112908,112909],{"class":503}," z_score(std, mean, prices[x])\n",[151,112911,112912,112915,112917],{"class":469,"line":2497},[151,112913,112914],{"class":503},"        vendor_dict[x] ",[151,112916,1876],{"class":1869},[151,112918,112919],{"class":503}," vendor_z_score\n",[151,112921,112922,112924],{"class":469,"line":3140},[151,112923,17496],{"class":1869},[151,112925,112926],{"class":503}," vendor_dict\n",[151,112928,112929],{"class":469,"line":3149},[151,112930,1090],{"emptyLinePlaceholder":609},[151,112932,112933,112936,112938],{"class":469,"line":3158},[151,112934,112935],{"class":503},"vendor_z_score_list ",[151,112937,1876],{"class":1869},[151,112939,16606],{"class":503},[151,112941,112942,112944,112946,112948],{"class":469,"line":3167},[151,112943,16732],{"class":1869},[151,112945,44552],{"class":503},[151,112947,16417],{"class":1869},[151,112949,112950],{"class":503}," df.Pricing_:\n",[151,112952,112953,112956,112958],{"class":469,"line":3175},[151,112954,112955],{"class":503},"    z_score_dict ",[151,112957,1876],{"class":1869},[151,112959,112960],{"class":503}," score_vendors(x)\n",[151,112962,112963],{"class":469,"line":3184},[151,112964,112965],{"class":503},"    vendor_z_score_list.append(z_score_dict)\n",[151,112967,112968],{"class":469,"line":3193},[151,112969,1090],{"emptyLinePlaceholder":609},[151,112971,112972,112975,112977],{"class":469,"line":3720},[151,112973,112974],{"class":503},"vend_df ",[151,112976,1876],{"class":1869},[151,112978,112979],{"class":503}," pd.DataFrame(vendor_z_score_list)\n",[151,112981,112982],{"class":469,"line":3729},[151,112983,1090],{"emptyLinePlaceholder":609},[151,112985,112986,112989,112991],{"class":469,"line":3735},[151,112987,112988],{"class":503},"vendor_avg ",[151,112990,1876],{"class":1869},[151,112992,16634],{"class":503},[151,112994,112995,112997,112999,113001],{"class":469,"line":3745},[151,112996,16732],{"class":1869},[151,112998,44552],{"class":503},[151,113000,16417],{"class":1869},[151,113002,113003],{"class":503}," vend_df.columns:\n",[151,113005,113006,113009,113011,113014,113016],{"class":469,"line":3754},[151,113007,113008],{"class":503},"    vendor_avg[x] ",[151,113010,1876],{"class":1869},[151,113012,113013],{"class":503}," vend_df[x].dropna().sum()",[151,113015,19883],{"class":1869},[151,113017,113018],{"class":503},"vend_df[x].dropna().count()\n",[151,113020,113021],{"class":469,"line":3760},[151,113022,1090],{"emptyLinePlaceholder":609},[151,113024,113025,113028,113030,113033,113035,113037,113039,113041],{"class":469,"line":3773},[151,113026,113027],{"class":503},"vendor_avg_chart ",[151,113029,1876],{"class":1869},[151,113031,113032],{"class":503}," pd.DataFrame(vendor_avg, ",[151,113034,86494],{"class":15210},[151,113036,1876],{"class":1869},[151,113038,6698],{"class":503},[151,113040,9181],{"class":477},[151,113042,38820],{"class":503},[151,113044,113045],{"class":469,"line":3782},[151,113046,1090],{"emptyLinePlaceholder":609},[151,113048,113049,113052,113054,113056,113058,113060,113062,113064,113066,113069,113071,113073,113075,113077,113079,113081,113083,113085,113087,113089],{"class":469,"line":3791},[151,113050,113051],{"class":503},"vendor_avg_chart.T.sort_values(",[151,113053,65808],{"class":15210},[151,113055,1876],{"class":1869},[151,113057,9181],{"class":477},[151,113059,106],{"class":503},[151,113061,65817],{"class":15210},[151,113063,1876],{"class":1869},[151,113065,36962],{"class":477},[151,113067,113068],{"class":503},").plot(",[151,113070,100637],{"class":15210},[151,113072,1876],{"class":1869},[151,113074,100642],{"class":481},[151,113076,106],{"class":503},[151,113078,44358],{"class":15210},[151,113080,1876],{"class":1869},[151,113082,12386],{"class":503},[151,113084,42360],{"class":477},[151,113086,3634],{"class":503},[151,113088,24369],{"class":477},[151,113090,12451],{"class":503},[151,113092,113093,113095,113098,113100,113102,113104,113106],{"class":469,"line":3803},[151,113094,65123],{"class":503},[151,113096,113097],{"class":481},"'Average Z-score for Monitor prices by Vendor'",[151,113099,106],{"class":503},[151,113101,99065],{"class":15210},[151,113103,1876],{"class":1869},[151,113105,67140],{"class":477},[151,113107,3640],{"class":503},[151,113109,113110,113112,113114,113116,113118,113120,113122],{"class":469,"line":3811},[151,113111,65143],{"class":503},[151,113113,87648],{"class":481},[151,113115,106],{"class":503},[151,113117,99065],{"class":15210},[151,113119,1876],{"class":1869},[151,113121,67140],{"class":477},[151,113123,3640],{"class":503},[151,113125,113126,113128,113131,113133,113135,113137,113139],{"class":469,"line":3820},[151,113127,65133],{"class":503},[151,113129,113130],{"class":481},"'Vendor'",[151,113132,106],{"class":503},[151,113134,99065],{"class":15210},[151,113136,1876],{"class":1869},[151,113138,67140],{"class":477},[151,113140,3640],{"class":503},[151,113142,113143,113145,113147,113149,113151],{"class":469,"line":7084},[151,113144,65163],{"class":503},[151,113146,99065],{"class":15210},[151,113148,1876],{"class":1869},[151,113150,42327],{"class":477},[151,113152,3640],{"class":503},[151,113154,113155,113157,113159,113161,113163],{"class":469,"line":7148},[151,113156,99502],{"class":503},[151,113158,99065],{"class":15210},[151,113160,1876],{"class":1869},[151,113162,42327],{"class":477},[151,113164,3640],{"class":503},[151,113166,113167,113169,113172,113174,113176,113178,113180,113182,113184,113186,113188],{"class":469,"line":7211},[151,113168,102944],{"class":503},[151,113170,113171],{"class":481},"'Average Z-score'",[151,113173,60308],{"class":503},[151,113175,104219],{"class":15210},[151,113177,1876],{"class":1869},[151,113179,99461],{"class":481},[151,113181,106],{"class":503},[151,113183,99065],{"class":15210},[151,113185,1876],{"class":1869},[151,113187,67140],{"class":477},[151,113189,3640],{"class":503},[11,113191,113192],{},[2718,113193],{"alt":20386,"src":113194},"/static/pcpp/monitor/average_z_score_by_vendor.png",[11,113196,113197],{},"One problem with this method of scoring is that there is a very small number of price observations for each part, and the price observations generally are not normally distributed. Also, some vendors offer significantly more products than others. However, it does support my guess that Amazon has the most competitive product offerings overall.",[11,113199,113200],{},"The products with the fewest offerings tend to have the highest prices. Here's a look at how many monitors each vendor offers:",[459,113202,113204],{"className":13136,"code":113203,"language":12886,"meta":464,"style":464},"vendor_count = {}\nfor x in vend_df.columns:\n    vendor_count[x] = vend_df[x].dropna().count()\n\nvendor_count_chart = pd.DataFrame(vendor_count, index=[0])\n\nvendor_count_chart.T.sort_values(by=0, ascending=True).plot(kind='bar', figsize=(12,8))\nplt.title('Product count by Vendor', fontsize=14)\nplt.ylabel('Count', fontsize=14)\nplt.xlabel('Vendor', fontsize=14)\nplt.xticks(fontsize=13)\nplt.yticks(fontsize=13)\nplt.legend(['Count'], fontsize=14, loc='upper left')\n",[30,113205,113206,113215,113225,113235,113239,113259,113263,113306,113323,113339,113355,113367,113379],{"__ignoreMap":464},[151,113207,113208,113211,113213],{"class":469,"line":470},[151,113209,113210],{"class":503},"vendor_count ",[151,113212,1876],{"class":1869},[151,113214,16634],{"class":503},[151,113216,113217,113219,113221,113223],{"class":469,"line":488},[151,113218,16732],{"class":1869},[151,113220,44552],{"class":503},[151,113222,16417],{"class":1869},[151,113224,113003],{"class":503},[151,113226,113227,113230,113232],{"class":469,"line":500},[151,113228,113229],{"class":503},"    vendor_count[x] ",[151,113231,1876],{"class":1869},[151,113233,113234],{"class":503}," vend_df[x].dropna().count()\n",[151,113236,113237],{"class":469,"line":509},[151,113238,1090],{"emptyLinePlaceholder":609},[151,113240,113241,113244,113246,113249,113251,113253,113255,113257],{"class":469,"line":517},[151,113242,113243],{"class":503},"vendor_count_chart ",[151,113245,1876],{"class":1869},[151,113247,113248],{"class":503}," pd.DataFrame(vendor_count, ",[151,113250,86494],{"class":15210},[151,113252,1876],{"class":1869},[151,113254,6698],{"class":503},[151,113256,9181],{"class":477},[151,113258,38820],{"class":503},[151,113260,113261],{"class":469,"line":534},[151,113262,1090],{"emptyLinePlaceholder":609},[151,113264,113265,113268,113270,113272,113274,113276,113278,113280,113282,113284,113286,113288,113290,113292,113294,113296,113298,113300,113302,113304],{"class":469,"line":1413},[151,113266,113267],{"class":503},"vendor_count_chart.T.sort_values(",[151,113269,65808],{"class":15210},[151,113271,1876],{"class":1869},[151,113273,9181],{"class":477},[151,113275,106],{"class":503},[151,113277,65817],{"class":15210},[151,113279,1876],{"class":1869},[151,113281,36962],{"class":477},[151,113283,113068],{"class":503},[151,113285,100637],{"class":15210},[151,113287,1876],{"class":1869},[151,113289,100642],{"class":481},[151,113291,106],{"class":503},[151,113293,44358],{"class":15210},[151,113295,1876],{"class":1869},[151,113297,12386],{"class":503},[151,113299,42360],{"class":477},[151,113301,3634],{"class":503},[151,113303,24369],{"class":477},[151,113305,12451],{"class":503},[151,113307,113308,113310,113313,113315,113317,113319,113321],{"class":469,"line":1418},[151,113309,65123],{"class":503},[151,113311,113312],{"class":481},"'Product count by Vendor'",[151,113314,106],{"class":503},[151,113316,99065],{"class":15210},[151,113318,1876],{"class":1869},[151,113320,67140],{"class":477},[151,113322,3640],{"class":503},[151,113324,113325,113327,113329,113331,113333,113335,113337],{"class":469,"line":2462},[151,113326,65143],{"class":503},[151,113328,87648],{"class":481},[151,113330,106],{"class":503},[151,113332,99065],{"class":15210},[151,113334,1876],{"class":1869},[151,113336,67140],{"class":477},[151,113338,3640],{"class":503},[151,113340,113341,113343,113345,113347,113349,113351,113353],{"class":469,"line":2471},[151,113342,65133],{"class":503},[151,113344,113130],{"class":481},[151,113346,106],{"class":503},[151,113348,99065],{"class":15210},[151,113350,1876],{"class":1869},[151,113352,67140],{"class":477},[151,113354,3640],{"class":503},[151,113356,113357,113359,113361,113363,113365],{"class":469,"line":2480},[151,113358,65163],{"class":503},[151,113360,99065],{"class":15210},[151,113362,1876],{"class":1869},[151,113364,42327],{"class":477},[151,113366,3640],{"class":503},[151,113368,113369,113371,113373,113375,113377],{"class":469,"line":2489},[151,113370,99502],{"class":503},[151,113372,99065],{"class":15210},[151,113374,1876],{"class":1869},[151,113376,42327],{"class":477},[151,113378,3640],{"class":503},[151,113380,113381,113383,113385,113387,113389,113391,113393,113395,113397,113399,113401],{"class":469,"line":2497},[151,113382,102944],{"class":503},[151,113384,87648],{"class":481},[151,113386,60308],{"class":503},[151,113388,99065],{"class":15210},[151,113390,1876],{"class":1869},[151,113392,67140],{"class":477},[151,113394,106],{"class":503},[151,113396,104219],{"class":15210},[151,113398,1876],{"class":1869},[151,113400,99461],{"class":481},[151,113402,3640],{"class":503},[11,113404,113405],{},[2718,113406],{"alt":20386,"src":113407},"/static/pcpp/monitor/monitors_by_vendor.png",[56,113409,113411],{"id":113410},"user-reviews","User Reviews",[11,113413,113414],{},"One other interesting dimension of the PC part data is user reviews. Users are able to leave reviews for parts they included in their PCs. Reviews include a short text description with a star-rating (between 1 and 5 stars). Here's a short sample of some of this data:",[459,113416,113418],{"className":13136,"code":113417,"language":12886,"meta":464,"style":464},"df.Stars.value_counts().plot(kind='bar', figsize=(12,8), rot=0)\nplt.title('GPU User Review Star Ratings', fontsize=14)\nplt.xlabel('Star Rating', fontsize=14)\nplt.ylabel('Count', fontsize=14)\nplt.xticks(fontsize=13)\nplt.yticks(fontsize=13)\nplt.savefig(os.path.expanduser('~/Documents/GitHub/briancaffey.github.io/img/gpu/star_ratings.png'))\n",[30,113419,113420,113455,113472,113489,113505,113517,113529],{"__ignoreMap":464},[151,113421,113422,113425,113427,113429,113431,113433,113435,113437,113439,113441,113443,113445,113447,113449,113451,113453],{"class":469,"line":470},[151,113423,113424],{"class":503},"df.Stars.value_counts().plot(",[151,113426,100637],{"class":15210},[151,113428,1876],{"class":1869},[151,113430,100642],{"class":481},[151,113432,106],{"class":503},[151,113434,44358],{"class":15210},[151,113436,1876],{"class":1869},[151,113438,12386],{"class":503},[151,113440,42360],{"class":477},[151,113442,3634],{"class":503},[151,113444,24369],{"class":477},[151,113446,24817],{"class":503},[151,113448,101309],{"class":15210},[151,113450,1876],{"class":1869},[151,113452,9181],{"class":477},[151,113454,3640],{"class":503},[151,113456,113457,113459,113462,113464,113466,113468,113470],{"class":469,"line":488},[151,113458,65123],{"class":503},[151,113460,113461],{"class":481},"'GPU User Review Star Ratings'",[151,113463,106],{"class":503},[151,113465,99065],{"class":15210},[151,113467,1876],{"class":1869},[151,113469,67140],{"class":477},[151,113471,3640],{"class":503},[151,113473,113474,113476,113479,113481,113483,113485,113487],{"class":469,"line":500},[151,113475,65133],{"class":503},[151,113477,113478],{"class":481},"'Star Rating'",[151,113480,106],{"class":503},[151,113482,99065],{"class":15210},[151,113484,1876],{"class":1869},[151,113486,67140],{"class":477},[151,113488,3640],{"class":503},[151,113490,113491,113493,113495,113497,113499,113501,113503],{"class":469,"line":509},[151,113492,65143],{"class":503},[151,113494,87648],{"class":481},[151,113496,106],{"class":503},[151,113498,99065],{"class":15210},[151,113500,1876],{"class":1869},[151,113502,67140],{"class":477},[151,113504,3640],{"class":503},[151,113506,113507,113509,113511,113513,113515],{"class":469,"line":517},[151,113508,65163],{"class":503},[151,113510,99065],{"class":15210},[151,113512,1876],{"class":1869},[151,113514,42327],{"class":477},[151,113516,3640],{"class":503},[151,113518,113519,113521,113523,113525,113527],{"class":469,"line":534},[151,113520,99502],{"class":503},[151,113522,99065],{"class":15210},[151,113524,1876],{"class":1869},[151,113526,42327],{"class":477},[151,113528,3640],{"class":503},[151,113530,113531,113533,113536],{"class":469,"line":1413},[151,113532,93826],{"class":503},[151,113534,113535],{"class":481},"'~/Documents/GitHub/briancaffey.github.io/img/gpu/star_ratings.png'",[151,113537,12451],{"class":503},[11,113539,113540],{},[2718,113541],{"alt":20386,"src":113542},"/static/pcpp/gpu/star_ratings.png",[11,113544,113545],{},"Here's one text rating from each of the 5 categories:",[736,113547,113549],{"id":113548},"_5-star-rating","5 Star Rating",[210,113551,113552],{},[11,113553,113554],{},"Amazing GPU; 4 GB VRAM, and it processes data faster than I thought it would. Not a fancy 900-series one but still very powerful.",[736,113556,113558],{"id":113557},"_4-star-rating","4 Star Rating",[210,113560,113561],{},[11,113562,113563],{},"Runs cool, aesthetically pleasing. -1 star for the VRAM fiasco with Nvidia.",[736,113565,113567],{"id":113566},"_3-star-rating","3 Star Rating",[210,113569,113570],{},[11,113571,113572],{},"I like the aesthetics of this card, but I don’t like that I couldn’t get a 960 like I hoped. I for sure will let people not to go to the store to get a GPU, as they are overpriced at both Best Buy and Micro Center. Also, if you are planning to use this with an analog monitor, be sure that the monitor has a DisplayPort/HDMI/DVI input (and you have a cable of one of those types), as this card DOES NOT in ANY WAY accept analog signals.",[736,113574,113576],{"id":113575},"_2-star-rating","2 Star Rating",[210,113578,113579],{},[11,113580,113581],{},"It's a potato. You cannot play any game at any decent framerates even with a FX-8320.",[736,113583,113585],{"id":113584},"_1-star-rating","1 Star Rating",[210,113587,113588],{},[11,113589,113590],{},"CRAP",[11,113592,113593],{},"With this labeled text data we can use natural language processing (NLP) techniques to predict the sentiment (the star rating in this case) for a new text review. At a very basic level, this works by assigning a probabilities to each word in a review and then generating a binary prediction based on a statistical model.",[11,113595,113596],{},"To simplify the problem, we can have our model predict not how many stars a text review would have, but whether or not the text rating is a 5-star rating. This reduces the complexity of the task.",[11,113598,113599],{},"Before we do any statistical modeling, it is important that we make a very simple prediction based on the most common star rating. For GPUs, 67% of the ratings tend to be 5-star ratings, then we could expect to have an accuracy of 67% percent if we predicted that any new ratings are 5-stars, regardless of the accompanying text. It will be important to check our NLP model against this \"baseline\" prediction; hopefully we can significantly improve on it.",[14063,113601,113603],{"id":113602},"naive-bayes-text-classification","Naive Bayes text classification",[11,113605,113606,113607,113612],{},"I will be using techniques introduced in the 'Working With Text Data' tutorial introduced in the ",[20,113608,113611],{"href":113609,"rel":113610},"http://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html",[24],"scikit-learn documentation"," to classify text reviews. First we will put the labeled review data from the different part types into one DataFrame and map the 5-star rating to a binary variable where 5 stars ratings are mapped to 1 and ratings less than 5 stars are mapped to 0.",[459,113614,113616],{"className":13136,"code":113615,"language":12886,"meta":464,"style":464},"os.chdir(os.path.expanduser('~/Documents/Projects/Data/PCPP/parts/x_comment_csv_files/'))\nfiles = os.listdir(os.getcwd())\ndf = pd.DataFrame(index=[0])\nfor x in files:\n    df1 = pd.read_csv(x)\n    df= df.append(df1)\n\n    df = df.dropna()\n\ndf['five'] = ['five' if x == 5 else 'not' for x in df.Stars]\ndf['label'] = df.five.map({'not':0, 'five':1})\nprint df.label.value_counts()\n",[30,113617,113618,113627,113635,113654,113664,113674,113684,113688,113697,113701,113738,113769],{"__ignoreMap":464},[151,113619,113620,113622,113625],{"class":469,"line":470},[151,113621,86611],{"class":503},[151,113623,113624],{"class":481},"'~/Documents/Projects/Data/PCPP/parts/x_comment_csv_files/'",[151,113626,12451],{"class":503},[151,113628,113629,113631,113633],{"class":469,"line":488},[151,113630,64699],{"class":503},[151,113632,1876],{"class":1869},[151,113634,96095],{"class":503},[151,113636,113637,113639,113641,113644,113646,113648,113650,113652],{"class":469,"line":500},[151,113638,70720],{"class":503},[151,113640,1876],{"class":1869},[151,113642,113643],{"class":503}," pd.DataFrame(",[151,113645,86494],{"class":15210},[151,113647,1876],{"class":1869},[151,113649,6698],{"class":503},[151,113651,9181],{"class":477},[151,113653,38820],{"class":503},[151,113655,113656,113658,113660,113662],{"class":469,"line":509},[151,113657,16732],{"class":1869},[151,113659,44552],{"class":503},[151,113661,16417],{"class":1869},[151,113663,65212],{"class":503},[151,113665,113666,113669,113671],{"class":469,"line":517},[151,113667,113668],{"class":503},"    df1 ",[151,113670,1876],{"class":1869},[151,113672,113673],{"class":503}," pd.read_csv(x)\n",[151,113675,113676,113679,113681],{"class":469,"line":534},[151,113677,113678],{"class":503},"    df",[151,113680,1876],{"class":1869},[151,113682,113683],{"class":503}," df.append(df1)\n",[151,113685,113686],{"class":469,"line":1413},[151,113687,1090],{"emptyLinePlaceholder":609},[151,113689,113690,113692,113694],{"class":469,"line":1418},[151,113691,65019],{"class":503},[151,113693,1876],{"class":1869},[151,113695,113696],{"class":503}," df.dropna()\n",[151,113698,113699],{"class":469,"line":2462},[151,113700,1090],{"emptyLinePlaceholder":609},[151,113702,113703,113705,113708,113710,113712,113714,113716,113718,113720,113722,113724,113726,113729,113731,113733,113735],{"class":469,"line":2471},[151,113704,70736],{"class":503},[151,113706,113707],{"class":481},"'five'",[151,113709,16654],{"class":503},[151,113711,1876],{"class":1869},[151,113713,6604],{"class":503},[151,113715,113707],{"class":481},[151,113717,3435],{"class":1869},[151,113719,44552],{"class":503},[151,113721,17223],{"class":1869},[151,113723,546],{"class":477},[151,113725,17229],{"class":1869},[151,113727,113728],{"class":481}," 'not'",[151,113730,2235],{"class":1869},[151,113732,44552],{"class":503},[151,113734,16417],{"class":1869},[151,113736,113737],{"class":503}," df.Stars]\n",[151,113739,113740,113742,113745,113747,113749,113752,113755,113757,113759,113761,113763,113765,113767],{"class":469,"line":2480},[151,113741,70736],{"class":503},[151,113743,113744],{"class":481},"'label'",[151,113746,16654],{"class":503},[151,113748,1876],{"class":1869},[151,113750,113751],{"class":503}," df.five.map({",[151,113753,113754],{"class":481},"'not'",[151,113756,208],{"class":503},[151,113758,9181],{"class":477},[151,113760,106],{"class":503},[151,113762,113707],{"class":481},[151,113764,208],{"class":503},[151,113766,6760],{"class":477},[151,113768,19610],{"class":503},[151,113770,113771,113773],{"class":469,"line":2489},[151,113772,18513],{"class":2226},[151,113774,113775],{"class":503}," df.label.value_counts()\n",[459,113777,113780],{"className":113778,"code":113779,"language":997,"meta":464},[995],"1    1197\n0     769\nName: label, dtype: int64\n",[30,113781,113779],{"__ignoreMap":464},[11,113783,113784,113785,748],{},"With 1197 positive reviews and 769 negative reviews, our baseline prediction would would predict that all ratings are postive and we would expect an accuracy of about 60.8% (",[30,113786,113787],{},"1197/(1197+769)",[11,113789,113790,113791,113794],{},"The next step is to split the data into training and testing data, tokenize the text data and transform the structure of the data into a format that we can use to train our model. I'll describe this process in detail in ",[20,113792,113793],{"href":464},"another post",", but here is how we do this with scikit-learn:",[459,113796,113798],{"className":13136,"code":113797,"language":12886,"meta":464,"style":464},"X = df.Comments\ny = df.label\n\nfrom sklearn.cross_validation import train_test_split\nX_train, X_test, y_train, y_test = train_test_split(X, y, stratify=df.label, test_size=0.2, random_state=4)\n\nfrom sklearn.feature_extraction.text import CountVectorizer\nvect = CountVectorizer()\n\nX_train_dtm = vect.fit_transform(X_train)\nX_test_dtm = vect.transform(X_test)\n",[30,113799,113800,113809,113818,113822,113833,113868,113872,113884,113894,113898,113908],{"__ignoreMap":464},[151,113801,113802,113804,113806],{"class":469,"line":470},[151,113803,87698],{"class":503},[151,113805,1876],{"class":1869},[151,113807,113808],{"class":503}," df.Comments\n",[151,113810,113811,113813,113815],{"class":469,"line":488},[151,113812,98878],{"class":503},[151,113814,1876],{"class":1869},[151,113816,113817],{"class":503}," df.label\n",[151,113819,113820],{"class":469,"line":500},[151,113821,1090],{"emptyLinePlaceholder":609},[151,113823,113824,113826,113828,113830],{"class":469,"line":509},[151,113825,16853],{"class":1869},[151,113827,100010],{"class":503},[151,113829,16859],{"class":1869},[151,113831,113832],{"class":503}," train_test_split\n",[151,113834,113835,113838,113840,113843,113846,113848,113851,113854,113856,113858,113860,113862,113864,113866],{"class":469,"line":517},[151,113836,113837],{"class":503},"X_train, X_test, y_train, y_test ",[151,113839,1876],{"class":1869},[151,113841,113842],{"class":503}," train_test_split(X, y, ",[151,113844,113845],{"class":15210},"stratify",[151,113847,1876],{"class":1869},[151,113849,113850],{"class":503},"df.label, ",[151,113852,113853],{"class":15210},"test_size",[151,113855,1876],{"class":1869},[151,113857,71641],{"class":477},[151,113859,106],{"class":503},[151,113861,71775],{"class":15210},[151,113863,1876],{"class":1869},[151,113865,9187],{"class":477},[151,113867,3640],{"class":503},[151,113869,113870],{"class":469,"line":534},[151,113871,1090],{"emptyLinePlaceholder":609},[151,113873,113874,113876,113879,113881],{"class":469,"line":1413},[151,113875,16853],{"class":1869},[151,113877,113878],{"class":503}," sklearn.feature_extraction.text ",[151,113880,16859],{"class":1869},[151,113882,113883],{"class":503}," CountVectorizer\n",[151,113885,113886,113889,113891],{"class":469,"line":1418},[151,113887,113888],{"class":503},"vect ",[151,113890,1876],{"class":1869},[151,113892,113893],{"class":503}," CountVectorizer()\n",[151,113895,113896],{"class":469,"line":2462},[151,113897,1090],{"emptyLinePlaceholder":609},[151,113899,113900,113903,113905],{"class":469,"line":2471},[151,113901,113902],{"class":503},"X_train_dtm ",[151,113904,1876],{"class":1869},[151,113906,113907],{"class":503}," vect.fit_transform(X_train)\n",[151,113909,113910,113913,113915],{"class":469,"line":2480},[151,113911,113912],{"class":503},"X_test_dtm ",[151,113914,1876],{"class":1869},[151,113916,113917],{"class":503}," vect.transform(X_test)\n",[11,113919,113920],{},"Finally, we train and evaluate our model using a Multinomial Naive Bayes model:",[459,113922,113924],{"className":13136,"code":113923,"language":12886,"meta":464,"style":464},"# train a Naive Bayes model using X_train_dtm\nfrom sklearn.naive_bayes import MultinomialNB\nnb = MultinomialNB()\nnb.fit(X_train_dtm, y_train)\n\n# make class predictions for X_test_dtm\ny_pred_class = nb.predict(X_test_dtm)\n\n\n# calculate accuracy of class predictions\nfrom sklearn import metrics\nprint 'Accuracy score: ' + str(metrics.accuracy_score(y_test, y_pred_class))\n\n# confusion matrix\nprint \"Confusion Matrix: \\n\" + str(metrics.confusion_matrix(y_test, y_pred_class))\n",[30,113925,113926,113931,113943,113953,113958,113962,113967,113977,113981,113985,113990,114000,114014,114018,114023],{"__ignoreMap":464},[151,113927,113928],{"class":469,"line":470},[151,113929,113930],{"class":1527},"# train a Naive Bayes model using X_train_dtm\n",[151,113932,113933,113935,113938,113940],{"class":469,"line":488},[151,113934,16853],{"class":1869},[151,113936,113937],{"class":503}," sklearn.naive_bayes ",[151,113939,16859],{"class":1869},[151,113941,113942],{"class":503}," MultinomialNB\n",[151,113944,113945,113948,113950],{"class":469,"line":500},[151,113946,113947],{"class":503},"nb ",[151,113949,1876],{"class":1869},[151,113951,113952],{"class":503}," MultinomialNB()\n",[151,113954,113955],{"class":469,"line":509},[151,113956,113957],{"class":503},"nb.fit(X_train_dtm, y_train)\n",[151,113959,113960],{"class":469,"line":517},[151,113961,1090],{"emptyLinePlaceholder":609},[151,113963,113964],{"class":469,"line":534},[151,113965,113966],{"class":1527},"# make class predictions for X_test_dtm\n",[151,113968,113969,113972,113974],{"class":469,"line":1413},[151,113970,113971],{"class":503},"y_pred_class ",[151,113973,1876],{"class":1869},[151,113975,113976],{"class":503}," nb.predict(X_test_dtm)\n",[151,113978,113979],{"class":469,"line":1418},[151,113980,1090],{"emptyLinePlaceholder":609},[151,113982,113983],{"class":469,"line":2462},[151,113984,1090],{"emptyLinePlaceholder":609},[151,113986,113987],{"class":469,"line":2471},[151,113988,113989],{"class":1527},"# calculate accuracy of class predictions\n",[151,113991,113992,113994,113996,113998],{"class":469,"line":2480},[151,113993,16853],{"class":1869},[151,113995,83841],{"class":503},[151,113997,16859],{"class":1869},[151,113999,102361],{"class":503},[151,114001,114002,114004,114007,114009,114011],{"class":469,"line":2489},[151,114003,18513],{"class":2226},[151,114005,114006],{"class":481}," 'Accuracy score: '",[151,114008,23378],{"class":1869},[151,114010,84112],{"class":6205},[151,114012,114013],{"class":503},"(metrics.accuracy_score(y_test, y_pred_class))\n",[151,114015,114016],{"class":469,"line":2497},[151,114017,1090],{"emptyLinePlaceholder":609},[151,114019,114020],{"class":469,"line":3140},[151,114021,114022],{"class":1527},"# confusion matrix\n",[151,114024,114025,114027,114030,114032,114034,114036,114038],{"class":469,"line":3149},[151,114026,18513],{"class":2226},[151,114028,114029],{"class":481}," \"Confusion Matrix: ",[151,114031,8043],{"class":477},[151,114033,8592],{"class":481},[151,114035,23378],{"class":1869},[151,114037,84112],{"class":6205},[151,114039,114040],{"class":503},"(metrics.confusion_matrix(y_test, y_pred_class))\n",[459,114042,114045],{"className":114043,"code":114044,"language":997,"meta":464},[995],"Accuracy score: 0.723350253807\nConfusion Matrix:\n[[ 84  70]\n [ 39 201]]\n",[30,114046,114044],{"__ignoreMap":464},[11,114048,114049,114050,114053,114054,114057],{},"Here's a quick overview of what we just did. First, we split the labeled text data into training data and test data. We trained our model on the training data and then used the trained model to predict the label of 20% percent of our data set. One important thing to do in the training process is to split the train and test data into equal proportions of positive and negative reviews. This is accomplished by including ",[30,114051,114052],{},"stratify=df.label"," as a parameter for our ",[30,114055,114056],{},"train_test_split"," function.",[11,114059,114060],{},"We correctly predicted the label for 285 reviews of the 394 text reviews in our testing data, resulting in an accuracy score of 72.3%, which is a significant improvement over our 60% baseline prediction (predicting that all ratings are 5 star ratings). The confusion matrix adds more dimensions to our accuracy score. Here's what the numbers in the confusion matrix correspond to:",[76,114062,114063,114068,114073,114078],{},[79,114064,114065,114067],{},[15,114066,45410],{},": 84 text reviews were correctly predicted to have less than 5 stars. (true positives)",[79,114069,114070,114072],{},[15,114071,73169],{},": 70 text reviews were predicted to have 5 stars but had less than 5 stars. (false negatives)",[79,114074,114075,114077],{},[15,114076,89825],{},": 39 text reviews were predicted to have less than 5 stars had 5 stars. (false positives)",[79,114079,114080,114083],{},[15,114081,114082],{},"201",": 201 text reviews were correctly predicted to have 5 stars. (true negatives)",[11,114085,114086],{},"Let's take a look at a few of the text reviews for false negatives found in our results (reviews that were predicted to have 5 stars but had less than 5 stars):",[210,114088,114089],{},[11,114090,114091],{},"Great GPU",[210,114093,114094],{},[11,114095,114096],{},"So far so good.",[210,114098,114099],{},[11,114100,114101],{},"I don't play too many graphics-heavy games, so this card is pretty good for me.",[11,114103,114104],{},"And here are a few false positives (reviews predicted to have less than 5 star but rated 5 stars):",[210,114106,114107],{},[11,114108,114109],{},"I bought it because it was yellow",[210,114111,114112],{},[11,114113,114114],{},"Works great, speed is as-advertised. It's a bit boring for putting on display in your case, but at least it's not covered in some lame sticker.",[210,114116,114117],{},[11,114118,114119],{},"RAM is RAM. End of story.",[11,114121,114122],{},"It's easy to see the limitations of such a model, but I think that the accuracy than can be achieved from this Naive Bayes approach is simply amazing given the nuance of natural language. A major limitation is the amount of text data available to train the model, and also the fact that some of the text reviews were not as carefully written as reviews you may see on Amazon.",[11,114124,114125,114126,114131],{},"There is a lot of exciting work being done in the area of sentiment analysis. Google uses similar principles when filtering out spam emails from regular emails in your inbox. A better approach would be to use a Recurrent Neural Network with Long Short Term Memory architure like the famous example of IMDB movie review sentiment analysis on ",[20,114127,114130],{"href":114128,"rel":114129},"http://deeplearning.net/tutorial/lstm.html",[24],"DeepLearning.net",". I'm curious to see how an advanced model would perform against a Naive Bayes model with a limited amount of training data.",[11,114133,114134],{},"We will do more text analysis with the user descriptions of PC builds. Instead of classifying builds, we will attempt to cluster them into distinct categories based on the language used in their descriptions.",[56,114136,114138],{"id":114137},"pc-builds","PC Builds",[11,114140,114141,114142,208],{},"We can now revisit data from the collection of PC builds. Each row in the builds DataFrame contains several links to the parts that are included in the. To do this, we will be merging the part data frames with the PC builds data frame. Here's a bried description of database-style DataFrame joining/merging from the ",[20,114143,114146],{"href":114144,"rel":114145},"http://pandas.pydata.org/pandas-docs/stable/merging.html",[24],"pandas documentation",[210,114148,114149],{},[11,114150,114151],{},"pandas has full-featured, high performance in-memory join operations idiomatically very similar to relational databases like SQL. These methods perform significantly better (in some cases well over an order of magnitude better) than other open source implementations (like base::merge.data.frame in R). The reason for this is careful algorithmic design and internal layout of the data in DataFrame.",[11,114153,114154],{},"This will allow us to add additional information (as new columns) for each of the parts in the builds DataFrame. For example, we can add a column indicating the number of watts for the power supply of each build. Let's start with this simple example. Here's how we would perform this operation in pandas:",[459,114156,114158],{"className":13136,"code":114157,"language":12886,"meta":464,"style":464},"#navigate to the builds DataFrame\nos.chdir('/Users/andrewcaffey/Documents/Projects/Data/PCPP/builds/')\n\n#read in the builds DataFrame\ndf = pd.read_csv('master_build_csv.csv', low_memory=False)\n\n#Setup a new DataFrame that contains only unique IDs for the PSU of each build\ndf1 = df[['Power_Supply_1_link']]\n\n#navigate to the PSU DataFrame\nos.chdir(os.path.expanduser('~/Documents/Projects/Data/PCPP/parts/x_csv_files/'))\n\n#read in psu DataFrame\npsu_df = pd.read_csv('psu_csv.csv')\n\n#Filter for certain columns we are interested in\npsu_columns = [u'Modular', u'Name', u'avg', u'short_link', u'power', u'eff_rank', u'ppw', u'Manufacturer', u'Efficiency Certification']\n\n#redefine the DataFrame to include those columns only\npsu_df = psu_df[psu_columns]\n\n#merge DataFrames\ndf1 = pd.merge(df1, psu_df, how='left',left_on='Power_Supply_1_link', right_on='short_link')\n",[30,114159,114160,114165,114173,114177,114182,114204,114208,114213,114227,114231,114236,114245,114249,114254,114268,114272,114277,114341,114345,114350,114359,114363,114368],{"__ignoreMap":464},[151,114161,114162],{"class":469,"line":470},[151,114163,114164],{"class":1527},"#navigate to the builds DataFrame\n",[151,114166,114167,114169,114171],{"class":469,"line":488},[151,114168,92835],{"class":503},[151,114170,97286],{"class":481},[151,114172,3640],{"class":503},[151,114174,114175],{"class":469,"line":500},[151,114176,1090],{"emptyLinePlaceholder":609},[151,114178,114179],{"class":469,"line":509},[151,114180,114181],{"class":1527},"#read in the builds DataFrame\n",[151,114183,114184,114186,114188,114190,114193,114195,114198,114200,114202],{"class":469,"line":517},[151,114185,70720],{"class":503},[151,114187,1876],{"class":1869},[151,114189,86782],{"class":503},[151,114191,114192],{"class":481},"'master_build_csv.csv'",[151,114194,106],{"class":503},[151,114196,114197],{"class":15210},"low_memory",[151,114199,1876],{"class":1869},[151,114201,39461],{"class":477},[151,114203,3640],{"class":503},[151,114205,114206],{"class":469,"line":534},[151,114207,1090],{"emptyLinePlaceholder":609},[151,114209,114210],{"class":469,"line":1413},[151,114211,114212],{"class":1527},"#Setup a new DataFrame that contains only unique IDs for the PSU of each build\n",[151,114214,114215,114217,114219,114222,114225],{"class":469,"line":1418},[151,114216,86777],{"class":503},[151,114218,1876],{"class":1869},[151,114220,114221],{"class":503}," df[[",[151,114223,114224],{"class":481},"'Power_Supply_1_link'",[151,114226,87779],{"class":503},[151,114228,114229],{"class":469,"line":2462},[151,114230,1090],{"emptyLinePlaceholder":609},[151,114232,114233],{"class":469,"line":2471},[151,114234,114235],{"class":1527},"#navigate to the PSU DataFrame\n",[151,114237,114238,114240,114243],{"class":469,"line":2480},[151,114239,86611],{"class":503},[151,114241,114242],{"class":481},"'~/Documents/Projects/Data/PCPP/parts/x_csv_files/'",[151,114244,12451],{"class":503},[151,114246,114247],{"class":469,"line":2489},[151,114248,1090],{"emptyLinePlaceholder":609},[151,114250,114251],{"class":469,"line":2497},[151,114252,114253],{"class":1527},"#read in psu DataFrame\n",[151,114255,114256,114259,114261,114263,114266],{"class":469,"line":3140},[151,114257,114258],{"class":503},"psu_df ",[151,114260,1876],{"class":1869},[151,114262,86782],{"class":503},[151,114264,114265],{"class":481},"'psu_csv.csv'",[151,114267,3640],{"class":503},[151,114269,114270],{"class":469,"line":3149},[151,114271,1090],{"emptyLinePlaceholder":609},[151,114273,114274],{"class":469,"line":3158},[151,114275,114276],{"class":1527},"#Filter for certain columns we are interested in\n",[151,114278,114279,114282,114284,114286,114288,114290,114292,114294,114296,114298,114300,114302,114304,114306,114309,114311,114313,114315,114317,114319,114321,114323,114325,114327,114329,114331,114333,114335,114337,114339],{"class":469,"line":3167},[151,114280,114281],{"class":503},"psu_columns ",[151,114283,1876],{"class":1869},[151,114285,6604],{"class":503},[151,114287,68688],{"class":12347},[151,114289,99579],{"class":481},[151,114291,106],{"class":503},[151,114293,68688],{"class":12347},[151,114295,98005],{"class":481},[151,114297,106],{"class":503},[151,114299,68688],{"class":12347},[151,114301,99593],{"class":481},[151,114303,106],{"class":503},[151,114305,68688],{"class":12347},[151,114307,114308],{"class":481},"'short_link'",[151,114310,106],{"class":503},[151,114312,68688],{"class":12347},[151,114314,98818],{"class":481},[151,114316,106],{"class":503},[151,114318,68688],{"class":12347},[151,114320,98993],{"class":481},[151,114322,106],{"class":503},[151,114324,68688],{"class":12347},[151,114326,98867],{"class":481},[151,114328,106],{"class":503},[151,114330,68688],{"class":12347},[151,114332,99572],{"class":481},[151,114334,106],{"class":503},[151,114336,68688],{"class":12347},[151,114338,99002],{"class":481},[151,114340,3691],{"class":503},[151,114342,114343],{"class":469,"line":3175},[151,114344,1090],{"emptyLinePlaceholder":609},[151,114346,114347],{"class":469,"line":3184},[151,114348,114349],{"class":1527},"#redefine the DataFrame to include those columns only\n",[151,114351,114352,114354,114356],{"class":469,"line":3193},[151,114353,114258],{"class":503},[151,114355,1876],{"class":1869},[151,114357,114358],{"class":503}," psu_df[psu_columns]\n",[151,114360,114361],{"class":469,"line":3720},[151,114362,1090],{"emptyLinePlaceholder":609},[151,114364,114365],{"class":469,"line":3729},[151,114366,114367],{"class":1527},"#merge DataFrames\n",[151,114369,114370,114372,114374,114377,114379,114381,114384,114386,114389,114391,114393,114395,114398,114400,114402],{"class":469,"line":3735},[151,114371,86777],{"class":503},[151,114373,1876],{"class":1869},[151,114375,114376],{"class":503}," pd.merge(df1, psu_df, ",[151,114378,638],{"class":15210},[151,114380,1876],{"class":1869},[151,114382,114383],{"class":481},"'left'",[151,114385,3634],{"class":503},[151,114387,114388],{"class":15210},"left_on",[151,114390,1876],{"class":1869},[151,114392,114224],{"class":481},[151,114394,106],{"class":503},[151,114396,114397],{"class":15210},"right_on",[151,114399,1876],{"class":1869},[151,114401,114308],{"class":481},[151,114403,3640],{"class":503},[11,114405,114406,114407,114410,114411,114414,114415,18952,114418,187,114420,114423,114424,187,114426,114428,114429,114432,114433,114435,114436,643],{},"The arguments of ",[30,114408,114409],{},"pd.merge()"," define how we merge the information from ",[30,114412,114413],{},"psu_df"," onto ",[30,114416,114417],{},"df1",[30,114419,114388],{},[30,114421,114422],{},"right_one"," define the columns that we will use in the ",[30,114425,114417],{},[30,114427,114413],{}," DataFrames to match information. ",[30,114430,114431],{},"how='left'"," essentially specifies that we want to keep all of the original rows of ",[30,114434,114417],{},", even if the PSU links for those rows are missing. There is more information and examples on DataFrame merging in the ",[20,114437,114439],{"href":114144,"rel":114438},[24],"documentation",[11,114441,114442,114444],{},[30,114443,114417],{}," is now a DataFrame that displays the PSU data for each unique build in the collection of nearly 26,000 PCs. This allows us to ask questions about the PC build data that we couldn't ask with link data alone, such as: what does the distribution of power ratings look like for PSUs across all PC builds?",[11,114446,114447],{},"In the next post I will merge all of the individual PC part data frames with the PC builds data frame so we can have a more granular look at the collection of computers and their parts.",[56,114449,114451],{"id":114450},"my-recent-pc-builds","My Recent PC Builds",[11,114453,114454],{},"As promised, here are the two builds that I put together last summer:",[14063,114456,95238],{"id":114457},"ascension-i",[11,114459,114460,10289,114465],{},[20,114461,114464],{"href":114462,"rel":114463},"https://pcpartpicker.com/list/fRx8d6",[24],"PCPartPicker part list",[20,114466,114469],{"href":114467,"rel":114468},"https://pcpartpicker.com/list/fRx8d6/by_merchant/",[24],"Price breakdown by merchant",[1131,114471,114472,114485],{},[1134,114473,114474],{},[1137,114475,114476,114479,114482],{},[1140,114477,99977],{"align":114478},"left",[1140,114480,114481],{"align":114478},"Item",[1140,114483,114484],{"align":114478},"Price",[1153,114486,114487,114503,114519,114535,114551,114568,114581,114597,114614,114630,114647,114664,114681,114692,114706],{},[1137,114488,114489,114493,114500],{},[1158,114490,114491],{"align":114478},[15,114492,72752],{},[1158,114494,114495],{"align":114478},[20,114496,114499],{"href":114497,"rel":114498},"https://pcpartpicker.com/product/tdmxFT/intel-cpu-bx80662i76700k",[24],"Intel Core i7-6700K 4.0GHz Quad-Core Processor",[1158,114501,114502],{"align":114478},"$329.25 @ OutletPC",[1137,114504,114505,114509,114516],{},[1158,114506,114507],{"align":114478},[15,114508,104295],{},[1158,114510,114511],{"align":114478},[20,114512,114515],{"href":114513,"rel":114514},"https://pcpartpicker.com/product/CrDzK8/corsair-cpu-cooler-cw9060025ww",[24],"Corsair H100i v2 70.7 CFM Liquid CPU Cooler",[1158,114517,114518],{"align":114478},"$99.99 @ B&H",[1137,114520,114521,114525,114532],{},[1158,114522,114523],{"align":114478},[15,114524,97475],{},[1158,114526,114527],{"align":114478},[20,114528,114531],{"href":114529,"rel":114530},"https://pcpartpicker.com/product/tBZ2FT/asus-motherboard-maximusviiihero",[24],"Asus MAXIMUS VIII HERO ATX LGA1151 Motherboard",[1158,114533,114534],{"align":114478},"$209.99 @ B&H",[1137,114536,114537,114541,114548],{},[1158,114538,114539],{"align":114478},[15,114540,97465],{},[1158,114542,114543],{"align":114478},[20,114544,114547],{"href":114545,"rel":114546},"https://pcpartpicker.com/product/dNLypg/crucial-memory-bls2k8g4d240fsa",[24],"Crucial Ballistix Sport 16GB (2 x 8GB) DDR4-2400 Memory",[1158,114549,114550],{"align":114478},"$109.64 @ B&H",[1137,114552,114553,114558,114565],{},[1158,114554,114555],{"align":114478},[15,114556,114557],{},"Storage",[1158,114559,114560],{"align":114478},[20,114561,114564],{"href":114562,"rel":114563},"https://pcpartpicker.com/product/3kL7YJ/samsung-internal-hard-drive-mz75e250bam",[24],"Samsung 850 EVO-Series 250GB 2.5\" Solid State Drive",[1158,114566,114567],{"align":114478},"$97.88 @ OutletPC",[1137,114569,114570,114574,114579],{},[1158,114571,114572],{"align":114478},[15,114573,114557],{},[1158,114575,114576],{"align":114478},[20,114577,114564],{"href":114562,"rel":114578},[24],[1158,114580,114567],{"align":114478},[1137,114582,114583,114587,114594],{},[1158,114584,114585],{"align":114478},[15,114586,114557],{},[1158,114588,114589],{"align":114478},[20,114590,114593],{"href":114591,"rel":114592},"https://pcpartpicker.com/product/Fz2kcf/western-digital-internal-hard-drive-wd1003fzex",[24],"Western Digital BLACK SERIES 1TB 3.5\" 7200RPM Internal Hard Drive",[1158,114595,114596],{"align":114478},"$69.00 @ B&H",[1137,114598,114599,114604,114611],{},[1158,114600,114601],{"align":114478},[15,114602,114603],{},"Video Card",[1158,114605,114606],{"align":114478},[20,114607,114610],{"href":114608,"rel":114609},"https://pcpartpicker.com/product/gRvZxr/msi-video-card-geforcegtx1080foundersedition",[24],"MSI GeForce GTX 1080 8GB Founders Edition Video Card",[1158,114612,114613],{"align":114478},"$660.31 @ Amazon",[1137,114615,114616,114620,114627],{},[1158,114617,114618],{"align":114478},[15,114619,97437],{},[1158,114621,114622],{"align":114478},[20,114623,114626],{"href":114624,"rel":114625},"https://pcpartpicker.com/product/9JvRsY/corsair-case-cc9011049ww",[24],"Corsair 450D ATX Mid Tower Case",[1158,114628,114629],{"align":114478},"$109.99 @ Newegg",[1137,114631,114632,114637,114644],{},[1158,114633,114634],{"align":114478},[15,114635,114636],{},"Power Supply",[1158,114638,114639],{"align":114478},[20,114640,114643],{"href":114641,"rel":114642},"https://pcpartpicker.com/product/DmPzK8/corsair-power-supply-cp9020086",[24],"Corsair 850W 80+ Gold Certified Semi-Modular ATX Power Supply",[1158,114645,114646],{"align":114478},"$120.98 @ Newegg",[1137,114648,114649,114654,114661],{},[1158,114650,114651],{"align":114478},[15,114652,114653],{},"Operating System",[1158,114655,114656],{"align":114478},[20,114657,114660],{"href":114658,"rel":114659},"https://pcpartpicker.com/product/MfH48d/microsoft-os-fqc08930",[24],"Microsoft Windows 10 Pro OEM 64-bit",[1158,114662,114663],{"align":114478},"$94.00 @ Amazon",[1137,114665,114666,114671,114678],{},[1158,114667,114668],{"align":114478},[15,114669,114670],{},"Software",[1158,114672,114673],{"align":114478},[20,114674,114677],{"href":114675,"rel":114676},"https://pcpartpicker.com/product/XwzZxr/eset-software-esshn111rbx2016",[24],"ESET Smart Security 2016 (1 Year Subscription) Software",[1158,114679,114680],{"align":114478},"$62.98 @ Newegg",[1137,114682,114683,114688,114690],{},[1158,114684,114685],{"align":114478},[51,114686,114687],{},"Prices include shipping, taxes, rebates, and discounts",[1158,114689],{"align":114478},[1158,114691],{"align":114478},[1137,114693,114694,114699,114704],{},[1158,114695,114696],{"align":114478},[15,114697,114698],{},"Total",[1158,114700,114701],{"align":114478},[15,114702,114703],{},"$2061.89",[1158,114705],{"align":114478},[1137,114707,114708,114716,114718],{},[1158,114709,114710,114711,114715],{"align":114478},"Generated by ",[20,114712,95225],{"href":114713,"rel":114714},"http://pcpartpicker.com",[24]," 2017-02-11 21:29 EST-0500",[1158,114717],{"align":114478},[1158,114719],{"align":114478},[14063,114721,114723],{"id":114722},"beastmode-ii-bm2","Beastmode II (BM2)",[11,114725,114726,10289,114730],{},[20,114727,114464],{"href":114728,"rel":114729},"https://pcpartpicker.com/list/D83rWX",[24],[20,114731,114469],{"href":114732,"rel":114733},"https://pcpartpicker.com/list/D83rWX/by_merchant/",[24],[1131,114735,114736,114746],{},[1134,114737,114738],{},[1137,114739,114740,114742,114744],{},[1140,114741,99977],{"align":114478},[1140,114743,114481],{"align":114478},[1140,114745,114484],{"align":114478},[1153,114747,114748,114764,114777,114793,114808,114821,114834,114850,114863,114880,114893,114909,114925,114938,114954,114964,114974,114984,114997],{},[1137,114749,114750,114754,114761],{},[1158,114751,114752],{"align":114478},[15,114753,72752],{},[1158,114755,114756],{"align":114478},[20,114757,114760],{"href":114758,"rel":114759},"https://pcpartpicker.com/product/Td98TW/intel-cpu-bx80671i76800k",[24],"Intel Core i7-6800K 3.4GHz 6-Core Processor",[1158,114762,114763],{"align":114478},"$408.58 @ OutletPC",[1137,114765,114766,114770,114775],{},[1158,114767,114768],{"align":114478},[15,114769,104295],{},[1158,114771,114772],{"align":114478},[20,114773,114515],{"href":114513,"rel":114774},[24],[1158,114776,114518],{"align":114478},[1137,114778,114779,114783,114790],{},[1158,114780,114781],{"align":114478},[15,114782,97475],{},[1158,114784,114785],{"align":114478},[20,114786,114789],{"href":114787,"rel":114788},"https://pcpartpicker.com/product/wtnG3C/gigabyte-ga-x99-designare-ex-atx-lga2011-3-motherboard-ga-x99-designare-ex",[24],"Gigabyte GA-X99-Designare EX ATX LGA2011-3 Motherboard",[1158,114791,114792],{"align":114478},"$418.95 @ B&H",[1137,114794,114795,114799,114805],{},[1158,114796,114797],{"align":114478},[15,114798,97465],{},[1158,114800,114801],{"align":114478},[20,114802,114804],{"href":97648,"rel":114803},[24],"Corsair Vengeance LPX 16GB (2 x 8GB) DDR4-3000 Memory",[1158,114806,114807],{"align":114478},"$124.99 @ Newegg",[1137,114809,114810,114814,114819],{},[1158,114811,114812],{"align":114478},[15,114813,97465],{},[1158,114815,114816],{"align":114478},[20,114817,114804],{"href":97648,"rel":114818},[24],[1158,114820,114807],{"align":114478},[1137,114822,114823,114827,114832],{},[1158,114824,114825],{"align":114478},[15,114826,114557],{},[1158,114828,114829],{"align":114478},[20,114830,114564],{"href":114562,"rel":114831},[24],[1158,114833,114567],{"align":114478},[1137,114835,114836,114840,114847],{},[1158,114837,114838],{"align":114478},[15,114839,114557],{},[1158,114841,114842],{"align":114478},[20,114843,114846],{"href":114844,"rel":114845},"https://pcpartpicker.com/product/XtjG3C/western-digital-internal-hard-drive-wd2003fzex",[24],"Western Digital BLACK SERIES 2TB 3.5\" 7200RPM Internal Hard Drive",[1158,114848,114849],{"align":114478},"$117.99 @ SuperBiiz",[1137,114851,114852,114856,114861],{},[1158,114853,114854],{"align":114478},[15,114855,114557],{},[1158,114857,114858],{"align":114478},[20,114859,114846],{"href":114844,"rel":114860},[24],[1158,114862,114849],{"align":114478},[1137,114864,114865,114869,114877],{},[1158,114866,114867],{"align":114478},[15,114868,114603],{},[1158,114870,114871,114876],{"align":114478},[20,114872,114875],{"href":114873,"rel":114874},"https://pcpartpicker.com/product/FwcMnQ/msi-video-card-geforcegtx1080armor8goc",[24],"MSI GeForce GTX 1080 8GB Video Card"," (2-Way SLI)",[1158,114878,114879],{"align":114478},"$598.45 @ Amazon",[1137,114881,114882,114886,114891],{},[1158,114883,114884],{"align":114478},[15,114885,114603],{},[1158,114887,114888,114876],{"align":114478},[20,114889,114875],{"href":114873,"rel":114890},[24],[1158,114892,114879],{"align":114478},[1137,114894,114895,114899,114906],{},[1158,114896,114897],{"align":114478},[15,114898,97437],{},[1158,114900,114901],{"align":114478},[20,114902,114905],{"href":114903,"rel":114904},"https://pcpartpicker.com/product/yLvRsY/corsair-case-760tblack",[24],"Corsair 760T Black ATX Full Tower Case",[1158,114907,114908],{"align":114478},"$176.33 @ Amazon",[1137,114910,114911,114915,114922],{},[1158,114912,114913],{"align":114478},[15,114914,114636],{},[1158,114916,114917],{"align":114478},[20,114918,114921],{"href":114919,"rel":114920},"https://pcpartpicker.com/product/dJ6BD3/evga-power-supply-220p21000xr",[24],"EVGA SuperNOVA 1000 P2 1000W 80+ Platinum Certified Fully-Modular ATX Power Supply",[1158,114923,114924],{"align":114478},"$183.80 @ OutletPC",[1137,114926,114927,114931,114936],{},[1158,114928,114929],{"align":114478},[15,114930,114653],{},[1158,114932,114933],{"align":114478},[20,114934,114660],{"href":114658,"rel":114935},[24],[1158,114937,114663],{"align":114478},[1137,114939,114940,114944,114951],{},[1158,114941,114942],{"align":114478},[15,114943,114670],{},[1158,114945,114946],{"align":114478},[20,114947,114950],{"href":114948,"rel":114949},"https://pcpartpicker.com/product/HmkwrH/eset-software-eavhn111rbx2016",[24],"ESET NOD32 Antivirus 2016 (1 Year Subscription) Software",[1158,114952,114953],{"align":114478},"$44.99 @ Adorama",[1137,114955,114956,114960,114962],{},[1158,114957,114958],{"align":114478},[51,114959,114687],{},[1158,114961],{"align":114478},[1158,114963],{"align":114478},[1137,114965,114966,114969,114972],{},[1158,114967,114968],{"align":114478},"Total (before mail-in rebates)",[1158,114970,114971],{"align":114478},"$3227.38",[1158,114973],{"align":114478},[1137,114975,114976,114979,114982],{},[1158,114977,114978],{"align":114478},"Mail-in rebates",[1158,114980,114981],{"align":114478},"-$20.00",[1158,114983],{"align":114478},[1137,114985,114986,114990,114995],{},[1158,114987,114988],{"align":114478},[15,114989,114698],{},[1158,114991,114992],{"align":114478},[15,114993,114994],{},"$3207.38",[1158,114996],{"align":114478},[1137,114998,114999,115005,115007],{},[1158,115000,114710,115001,115004],{"align":114478},[20,115002,95225],{"href":114713,"rel":115003},[24]," 2017-02-11 21:46 EST-0500",[1158,115006],{"align":114478},[1158,115008],{"align":114478},[589,115010,115011],{},"html pre.shiki code .sstjo, html code.shiki .sstjo{--shiki-default:#032F62;--shiki-dark:#9ECBFF;--shiki-sepia:#E6DB74}html pre.shiki code .sq6CD, html code.shiki .sq6CD{--shiki-default:#D73A49;--shiki-default-font-style:inherit;--shiki-dark:#F97583;--shiki-dark-font-style:inherit;--shiki-sepia:#66D9EF;--shiki-sepia-font-style:italic}html pre.shiki code .sMOD_, html code.shiki .sMOD_{--shiki-default:#24292E;--shiki-dark:#E1E4E8;--shiki-sepia:#F8F8F2}html pre.shiki code .sC2Qs, html code.shiki .sC2Qs{--shiki-default:#D73A49;--shiki-dark:#F97583;--shiki-sepia:#F92672}html pre.shiki code .srTi1, html code.shiki .srTi1{--shiki-default:#6F42C1;--shiki-dark:#B392F0;--shiki-sepia:#A6E22E}html pre.shiki code .sTHNf, html code.shiki .sTHNf{--shiki-default:#E36209;--shiki-default-font-style:inherit;--shiki-dark:#FFAB70;--shiki-dark-font-style:inherit;--shiki-sepia:#FD971F;--shiki-sepia-font-style:italic}html pre.shiki code .s7F3e, html code.shiki .s7F3e{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#AE81FF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html .sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html.sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html pre.shiki code .s8-w5, html code.shiki .s8-w5{--shiki-default:#6A737D;--shiki-dark:#6A737D;--shiki-sepia:#88846F}html pre.shiki code .sTrkL, html code.shiki .sTrkL{--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#66D9EF}html pre.shiki code .s-m8C, html code.shiki .s-m8C{--shiki-default:#005CC5;--shiki-default-font-style:inherit;--shiki-dark:#79B8FF;--shiki-dark-font-style:inherit;--shiki-sepia:#66D9EF;--shiki-sepia-font-style:italic}html pre.shiki code .so59x, html code.shiki .so59x{--shiki-default:#24292E;--shiki-default-font-style:inherit;--shiki-dark:#E1E4E8;--shiki-dark-font-style:inherit;--shiki-sepia:#FD971F;--shiki-sepia-font-style:italic}",{"title":464,"searchDepth":488,"depth":488,"links":115013},[115014,115015,115016,115017,115018,115019,115020,115027,115028],{"id":95249,"depth":488,"text":95250},{"id":97423,"depth":488,"text":97424},{"id":98797,"depth":488,"text":98798},{"id":97405,"depth":488,"text":97475},{"id":102738,"depth":488,"text":72752},{"id":112724,"depth":488,"text":112725},{"id":113410,"depth":488,"text":113411,"children":115021},[115022,115023,115024,115025,115026],{"id":113548,"depth":500,"text":113549},{"id":113557,"depth":500,"text":113558},{"id":113566,"depth":500,"text":113567},{"id":113575,"depth":500,"text":113576},{"id":113584,"depth":500,"text":113585},{"id":114137,"depth":488,"text":114138},{"id":114450,"depth":488,"text":114451},"2017-01-01","In the summer of 2016 I built two high-end computers, something I haven't done since 2011. I used PCPartPicker to research the components and read about PC builds similar to the ones I had in mind. It's a relatively new site that has a strong community of builders, helpful tools to help with part compatibility as well as extensive user reviews on PC components.",{"layout":48045},"/2017/01/01/pc-data",{"title":95216,"description":115030},"2017/01/01/pc-data",[12886,46089,115036,12355],"machine-learning","e61x5vnMTyN5bqjcTF6Ikx4AKqVNzVycmGh8V9aT5bU",{"id":115039,"title":115040,"body":115041,"comments":609,"date":115300,"description":464,"draft":602,"extension":605,"external":606,"image":115047,"meta":115301,"navigation":609,"path":115302,"seo":115303,"stem":115304,"tags":115305,"__hash__":115308},"blog/2016/04/07/home-media-setup.md","Home media center meta-tutorial with miniDLNA, Raspberry Pi, Deluge and Apple TV",{"type":8,"value":115042,"toc":115293},[115043,115048,115051,115054,115065,115068,115096,115100,115108,115117,115121,115130,115139,115143,115156,115162,115165,115171,115194,115197,115203,115206,115212,115230,115233,115239,115242,115250,115275,115279,115288,115290],[11,115044,115045],{},[2718,115046],{"alt":20386,"src":115047},"/static/media-setup.png",[11,115049,115050],{},"For a few months now I have been enjoying a new home media system that I threw together with my Raspberry Pi. My setup allows me sit in my living room and stream content from my Raspberry Pi to my 4th Generation Apple TV in full 1080p resolution. The Raspberry Pi is a small, inexpensive stand-alone computer, but it can serve as a powerful media server, and I've been blown away by its consistent performance. Rather than write everything from scratch, I've gathered the tutorials I used when setting up my devices and software and sprinkled it with a few bits of knowledge that I wish I knew when I started.",[11,115052,115053],{},"Here is an overview of my current setup:",[76,115055,115056,115059,115062],{},[79,115057,115058],{},"Raspberry Pi Model 2 B with miniDLNA (media server), Deluge (for torrenting media), TorGuard (anonymous VPN service) and omxplayer (built-in Raspberry Pi media player for playing media right on the Raspberry Pi)",[79,115060,115061],{},"Apple TV with VLC app (for streaming content from my network-connected Raspberry Pi via miniDLNA)",[79,115063,115064],{},"iPhone/iPad with Creation 5 + Creation 2 Video Player (for streaming content from my Raspberry Pi to my mobile devices)",[11,115066,115067],{},"Here's what you'll need:",[76,115069,115070,115073,115076,115079,115082,115090,115093],{},[79,115071,115072],{},"Raspberry Pi (Model 2 or 3)",[79,115074,115075],{},"Wifi-connection",[79,115077,115078],{},"Laptop (you will be SSHing into the Raspberry Pi and managing your torrent downloads through a web-browser interface on your laptop)",[79,115080,115081],{},"Sufficiently large microSD card (mine is 64GB), large USB drive (I recommend 256GB) or external hard drive (1TB)",[79,115083,115084,115089],{},[20,115085,115088],{"href":115086,"rel":115087},"https://torguard.net/aff.php?aff=1933",[24],"TorGuard account"," (optional)",[79,115091,115092],{},"Apple TV (optional)",[79,115094,115095],{},"iPhone or iPad with Creation 5 and Creation 2 Video player app installed (optional)",[56,115097,115099],{"id":115098},"virtual-private-network-vpn","Virtual Private Network (VPN)",[11,115101,115102,115103,115107],{},"If you are planning on downloading Copyrighted content from a public tracker, you should be using a virtual private network (VNP). I have been using ",[20,115104,115106],{"href":115086,"rel":115105},[24],"TorGuard"," for about 6 months and have had excellent service, great download speeds and setup was fairly painless.",[11,115109,115110,115111,115116],{},"Follow along with ",[20,115112,115115],{"href":115113,"rel":115114},"https://torguard.net/knowledgebase.php?action=displayarticle&id=174",[24],"this tutorial"," on how to get TorGuard running on your Raspberry Pi.",[56,115118,115120],{"id":115119},"minidlna","miniDLNA",[11,115122,115123,115124,115129],{},"Next, we will install miniDLNA on your Raspberry Pi. miniDLNA stands for (mini) Digital Network Living Alliance and is a protocol that is used in many devices, including consoles, SmartTVs, and mobile devices. To install it, follow along with ",[20,115125,115128],{"href":115126,"rel":115127},"http://www.instructables.com/id/Raspberry-Pi-Media-Server-MiniDLNA/",[24],"this Instructables tutorial",". Step 4 (Mounting the drive on startup) is not absolutely necessary, but it is a good idea if you will be using a dedicated external USB drive or external hard drive to store your content. I don't have a dedicated USB stick that I use with miniDLNA, but it works just fine playing content from my 64GB microSD card.",[11,115131,115132,115133,115138],{},"Download ",[20,115134,115137],{"href":115135,"rel":115136},"http://www.creation.com.es/creation-5-app/",[24],"Creation 5"," and Creation 2 Video Player on your iPhone. These are free apps with very minimal advertising. Through the Creation 5 app, you should be able to select your miniDLNA server as a media source. You should then see your Music, Image and Video folders that you just configured, but they will all be empty.",[56,115140,115142],{"id":115141},"bittorrent-client-deluge","BitTorrent Client (Deluge)",[11,115144,115145,115146,115151,115152,115155],{},"Now that you have your VPN and miniDLNA set up, you will want to try it out with some new content. I use Deluge, a BitTorrent client written in Python, but there are plenty of other great options out there. Deluge is fairly light-weight, so it works well with the Raspberry Pi. There are a number of ways that you can access Deluge, I prefer to use the Deluge WebUI. Here's ",[20,115147,115150],{"href":115148,"rel":115149},"http://www.howtogeek.com/142044/how-to-turn-a-raspberry-pi-into-an-always-on-bittorrent-box/",[24],"one more tutorial"," from How-to Geek that talks about a few different ways to install and configure Deluge. I recommend jumping to the ",[30,115153,115154],{},"Setting up Deluge for WebUI Access"," section and running the three commands you will need to get started:",[459,115157,115160],{"className":115158,"code":115159,"language":997},[995],"$ sudo apt-get install deluged\n$ sudo apt-get install python-mako\n$ sudo apt-get install deluge-web\n",[30,115161,115159],{"__ignoreMap":464},[11,115163,115164],{},"This gets everything installed. To use Deluge, you will need to run two more commands:",[459,115166,115169],{"className":115167,"code":115168,"language":997},[995],"$ deluged\n$ deluge-web&\n",[30,115170,115168],{"__ignoreMap":464},[11,115172,115173,115176,115177,115179,115180,115183,115184,115187,115188],{},[30,115174,115175],{},"deluged"," runs the Deluge daemon (a background process; the ",[30,115178,78271],{}," at the end of ",[30,115181,115182],{},"deluge"," signifies that it is a daemon) that will start Deluge. ",[30,115185,115186],{},"deluge-web&"," starts that web interface that should be available at http://",[10093,115189,115190,115191,115193],{"raspberry":464,"pi":464,"ip":464,"address":464},":8112. The ",[30,115192,54214],{}," simply keeps the command line available for running other commands.",[11,115195,115196],{},"You can set Deluge to save downloaded files directly into your various miniDLNA folders, but you will need to restart miniDLNA with the following commands before they are visible on your network. SSH into your Raspberry Pi by running the following command:",[459,115198,115201],{"className":115199,"code":115200,"language":997},[995],"$ ssh pi@\u003Cyour raspberry pi ip address>\n",[30,115202,115200],{"__ignoreMap":464},[11,115204,115205],{},"If you don't know your Raspberry Pi's IP address, run following command:",[459,115207,115210],{"className":115208,"code":115209,"language":997},[995],"$ ifconfig\n",[30,115211,115209],{"__ignoreMap":464},[11,115213,115214,115215,115218,115219,115222,115223,30583,115226,115229],{},"Look at the output under ",[30,115216,115217],{},"wlan0"," and it will be the address following ",[30,115220,115221],{},"inet addr",", something like ",[30,115224,115225],{},"192.168.1.5",[30,115227,115228],{},"10.0.1.132",". To clarify, this is your Raspberry Pi's internal IP address; it is assigned to your Raspberry Pi by your home network and it has nothing to do with the external IP address that you set up with TorGuard.",[11,115231,115232],{},"Enter your Raspberry Pi's login and password and then enter the following commands once you have established an SSH connection:",[459,115234,115237],{"className":115235,"code":115236,"language":997},[995],"$ sudo service minidlna restart\n$ sudo service minidlna force-reload\n",[30,115238,115236],{"__ignoreMap":464},[11,115240,115241],{},"Now you should be able to see your content in media players that support miniDLNA.",[11,115243,115244,115245,643],{},"Here are some additional resources for ",[20,115246,115249],{"href":115247,"rel":115248},"https://help.ubuntu.com/community/MiniDLNA",[24],"help with miniDLNA on Ubuntu.com",[11,115251,115252,115253,115258,115259,115262,115263,115268,115269,115274],{},"Before you start downloading content, I would use use TorGuard's ",[20,115254,115257],{"href":115255,"rel":115256},"https://torguard.net/checkmytorrentipaddress.php",[24],"Check My Torrent IP Address"," tool. This is a blank ",[30,115260,115261],{},".torrent"," file that you add to Deluge. In the torrent's tracker information you can see the Public IP address that is listed on the public tracker. This would be the address that ",[20,115264,115267],{"href":115265,"rel":115266},"https://en.wikipedia.org/wiki/Digital_Millennium_Copyright_Act",[24],"Digital Millennium Copyright Act (DMCA)"," would report to your ISP if it was your public IP address, so make sure it is not ",[20,115270,115273],{"href":115271,"rel":115272},"http://www.whatsmyip.org/",[24],"your public IP address"," that you use in your home. You can select proxy IP address from a number of different countries in the TorGuard setup.",[56,115276,115278],{"id":115277},"vlc-for-tvos","VLC for tvOS",[11,115280,115281,115282,115287],{},"VLC is a great media player, and you probably have it downloaded on your computer. You might not have know that VLC has an app for the Apple TV. This app supports the playback of media from miniDLNA servers like the one you just set up, so you can stream content directly from your Raspberry Pi onto your Apple TV through the VLC app. One thing to note, however, is that VLC for tvOS ",[20,115283,115286],{"href":115284,"rel":115285},"https://forum.videolan.org/viewtopic.php?t=125032",[24],"does not support AC3 audio encodings",", but you should be good to go for almost any other audio/video format on the planet.",[56,115289,14265],{"id":30030},[11,115291,115292],{},"Using the Raspberry Pi certainly isn't necessary for simply downloading torrents, but having it set as a dedicated machine for downloading content and playing back files is a lot of fun and works better than I would have guessed.I don't like running torrents on my MacBook Air, and sometimes I like to use my laptop when I'm watching TV. I would usually have to plug in my laptop to my TV directly and mirror the laptop display onto my TV, but the solution I have eliminates this headache altogether. My setup is by no means perfect, but it gets the job done with a very tolerable amount of effort on my part. Let me know in the comments if you have any critiques or ideas for how to enhance the setup I have described here. Thanks, and good luck with setting up your own home media system.",{"title":464,"searchDepth":488,"depth":488,"links":115294},[115295,115296,115297,115298,115299],{"id":115098,"depth":488,"text":115099},{"id":115119,"depth":488,"text":115120},{"id":115141,"depth":488,"text":115142},{"id":115277,"depth":488,"text":115278},{"id":30030,"depth":488,"text":14265},"2016-04-07",{"layout":48045},"/2016/04/07/home-media-setup",{"title":115040,"description":464},"2016/04/07/home-media-setup",[115306,115307,115182,83692],"mini-dlna","raspberry-pi","cc9fe4z3Jq-M4HtC8mKshDqzO6eRPj_AH10pTyzb3jw",1781796218142]