# 一部影片看完 Stanford AI 系統課程，從 LLM 到 Agentic Workflow https://www.youtube.com/watch?v=eKW9ITaltWw Translation: en [00:00] 如果你有在關注 AI If you have been paying attention to AI [00:01] 這一兩年最炙手可熱的工作一定是 AI Engineer，AI Builder The hottest job in the past year or two is definitely AI Engineer, AI Builder. [00:05] 大型語言模型越強，這群人越吃香 The stronger large language models become, the more in demand these people are. [00:08] 他們有能力幫公司打造內部工具，優化運營流程 They have the ability to help companies build internal tools and optimize operational processes. [00:11] 也已經有一堆人自己動手把想法做成產品，成立了自己的公司 There are also many people who have turned their ideas into products on their own and founded their own companies. [00:15] 而這種人，現在每間公司都在搶 And these people are now being fought over by every company. [00:17] 想學 AI 的人很多 There are many people who want to learn AI. [00:19] 但大多數人都不知道從哪裡下手 But most people don't know where to start. [00:21] 市面上的資源不是太破碎，就是太過艱澀 The resources available are either too fragmented or too difficult. [00:24] 看完還是不知道自己學到了什麼，或者下一步該往哪走 After finishing, you still don't know what you've learned or where to go next. [00:28] 史丹佛大學有一堂課叫做 Beyond LLM Stanford University has a course called Beyond LLM. [00:32] 我認為這是目前最接近那個答案的東西 I think this is the closest thing to that answer right now. [00:34] 從大型語言模型的本質講起，一路帶到 Prompt Engineering，RAG，Fine-Tuning，Agentic Workflow，還有 Multi-Agent 的框架 It starts from the essence of large language models, then moves on to Prompt Engineering, RAG, Fine-Tuning, Agentic Workflow, and the Multi-Agent framework. [00:42] 這堂課時長兩小時 This course is two hours long. [00:44] 今天我就用一支影片的時間把核心整理給你 Today, I will summarize the core content for you in the time of one video. [00:47] 你看完之後，會對 AI 在商業領域上的應用有一套完整的認知框架 After you watch it, you will have a complete cognitive framework for the application of AI in the business field. [00:51] 知道這些技術怎麼組合在一起，也能規劃自己接下來的學習路線 You will know how these technologies are combined and be able to plan your own learning path going forward. [00:56] 那我們廢話不多說，直接開始 Without further ado, let's get started. [00:58] 首先我們要先知道大型語言模型的限制是什麼 First, we need to understand what the limitations of large language models are. [01:02] LLM 要變強有兩條路 There are two ways for an LLM to become stronger. [01:04] 第一條是橫軸 The first is the horizontal axis. [01:05] 就是換更強的 base model That is, switching to a stronger base model. [01:07] 比方說從 GPT-4 升級到 GPT-5 For example, upgrading from GPT-4 to GPT-5. [01:10] 但橫軸是 OpenAI 跟 Anthropic 在做的 But the horizontal axis is what OpenAI and Anthropic are doing. [01:13] 一般人沒有那個能力 Ordinary people don't have that capability. [01:14] 也沒有那個金錢去訓練自己的大型語言模型 Nor do they have the money to train their own large language models. [01:18] 你能施力的地方是縱軸 The area where you can exert effort is the vertical axis. [01:20] 也就是在現有的 LLM 上面疊各種工程技術 That is, stacking various engineering techniques on top of existing LLMs. [01:23] 又叫做 Augmenting LLM Also called Augmenting LLM. [01:25] 也就是想辦法提升這些 base model 的能力 That is, finding ways to enhance the capabilities of these base models. [01:28] 而這整堂課，包括這支影片 And this entire course, including this video, [01:30] 講的都是縱軸的事情 is all about the vertical axis. [01:32] 如果你只會使用 base model If you only know how to use the base model, [01:34] 在商業場景上一定會撞牆 you will definitely hit a wall in business scenarios. [01:36] 舉個例子，ChatGPT 剛出來的時候 For example, when ChatGPT first came out, [01:39] 沒有連網功能，沒辦法調用工具 it didn't have internet connectivity and couldn't call tools. [01:41] 充其量就是一個很聰明的問答機器人 At best, it was just a very smart Q&A bot. [01:44] 沒辦法實際幫你做完一件事 It couldn't actually help you complete a task. [01:46] Stanford 的教授整理了 base model 幾個常見的限制 A Stanford professor compiled several common limitations of base models. [01:49] 第一是缺乏 domain knowledge First is the lack of domain knowledge. [01:51] 比方說有一組學生在做自動化農業設備 For example, a group of students is working on automated agricultural equipment. [01:54] 要用相機判斷作物有沒有生病 They want to use cameras to determine if crops are diseased. [01:57] 這種農業病害的 dataset 市面上根本找不到 This kind of agricultural disease dataset is simply not available on the market. [01:59] 換句話說你的公司資料，內部文件，產品規格 In other words, your company data, internal documents, product specifications [02:03] base model 都不知道 The base model doesn't even know. [02:04] 它也沒辦法根據這些資料幫你解決問題 It also can't help you solve problems based on this data. [02:07] 第二是資訊落後 Second, the information is outdated. [02:09] 模型不可能每幾個月重訓一次 The model can't be retrained every few months. [02:11] 新詞，新事件，新公司它通通不認識 It doesn't recognize new words, new events, or new companies at all. [02:14] 年輕人在用的網路流行用語 The internet slang that young people use. [02:16] 模型大多也聽不懂 The model mostly can't understand it either. [02:18] 第三個，控制很難 Third, control is difficult. [02:20] LLM 是機率性輸出 LLMs produce probabilistic outputs. [02:22] 同樣的 prompt 兩次跑可能結果不一樣 Running the same prompt twice might yield different results. [02:24] 在 ChatGPT 上聊天當然 OK Chatting on ChatGPT is of course OK. [02:26] 但你想想 production 環境 But think about the production environment. [02:28] 使用者問退費 A user asks about a refund. [02:29] AI 一下說可以一下說不行，公司就麻煩了 If the AI says yes one moment and no the next, the company is in trouble. [02:32] 教授說，就連 OpenAI 跟 xAI 這種資金最多 The professor said that even OpenAI and xAI, which have the most funding, [02:36] 人才最齊的團隊 and the most complete teams of talent, [02:37] 都還沒辦法把 LLM 完全控制好 still haven't been able to fully control LLMs. [02:39] 更何況一般公司 Let alone ordinary companies. [02:41] 第四個，當 context 太長 Fourth, when the context is too long, [02:43] 模型的表現會顯著退步 the model's performance significantly degrades. [02:45] 現在主流模型的 context window The context window of current mainstream models [02:47] 已經拉到 100 萬 token has been extended to 1 million tokens, [02:49] 差不多十幾本書的長度 roughly the length of a dozen books. [02:50] 但在這麼大的 context 裡 But within such a large context, [02:52] 還是會出現 lost in the middle 的現象 the 'lost in the middle' phenomenon still occurs. [02:54] 簡單講 Simply put, [02:55] 你把一個 Gary 午餐吃了一顆蘋果這樣的小細節 if you hide a small detail like 'Gary ate an apple for lunch' [02:57] 藏進公司過去一年的會議記錄裡 in the company's meeting records from the past year, [03:00] 再問模型 Gary 午餐吃了什麼 and then ask the model what Gary ate for lunch, [03:02] 雖然這個狀況現在已經顯著改善 although this situation has now significantly improved. [03:05] 但 LLM 有時候還是會答不出來 But sometimes LLM still cannot answer. [03:07] 想做 AI 產品，光換更強的 base model 不夠 To build an AI product, simply switching to a stronger base model is not enough. [03:10] 你需要在縱軸上施力 You need to apply effort on the vertical axis. [03:13] 接下來這支影片的前半段會告訴你，強化單一 LLM 的三個工具和觀念，也就是 Prompt Engineering、Fine-Tuning，還有 RAG The first half of this video will tell you three tools and concepts for strengthening a single LLM: Prompt Engineering, Fine-Tuning, and RAG. [03:21] 影片後半段會進到系統設計，介紹 Agentic Workflow 跟 evaluation 評估方法 The second half of the video will move into system design, introducing Agentic Workflow and evaluation methods. [03:26] 然後用一個客服 agent 的 case study 把這些東西串起來 Then it will tie these together with a customer service agent case study. [03:29] 最後也會帶到很多人感興趣的 Multi-Agent Finally, it will also cover Multi-Agent, which many people are interested in. [03:32] 那我們直接從 Prompt Engineering 開始講 So let's start directly with Prompt Engineering. [03:34] Stanford 的教授說他不認為 prompt engineer 會是一個職業 A Stanford professor said he doesn't think prompt engineer will be a career. [03:38] 因為提示詞工程應該是每個工程師都該會的基本技能 Because prompt engineering should be a basic skill that every engineer should have. [03:42] 你不會靠 prompt engineering 當飯吃 You won't make a living solely from prompt engineering. [03:44] 但這個技能會讓你在職涯裡用一輩子 But this skill will serve you throughout your career. [03:46] 就像九九乘法表一樣是基本功 It's a fundamental skill, just like the multiplication table. [03:49] 我之前有做過一部影片，專門在講 prompt engineering I previously made a video specifically about prompt engineering. [03:52] 有興趣的朋友可以去看看 Interested friends can go check it out. [03:53] 關於 prompt engineering，有一個很有意思的研究我想分享給你 Regarding prompt engineering, there is a very interesting study I want to share with you. [03:57] 頂級管理顧問公司 BCG 做過一個實驗 Top management consulting firm BCG conducted an experiment. [04:00] 他們把一群顧問分成三組 They divided a group of consultants into three teams. [04:02] 第一組沒有 AI The first team had no AI. [04:03] 第二組可以使用 ChatGPT The second team could use ChatGPT. [04:05] 第三組則是可以使用 ChatGPT 以外 The third group can use ChatGPT and also [04:07] 還有接受撰寫提示詞的訓練 received training in writing prompts. [04:10] 最後研究的結果發現三件事 Finally, the research results found three things. [04:12] 第一個叫 The Jagged Frontier，鋸齒邊界 The first is called The Jagged Frontier. [04:15] 意思是 AI 不是在所有任務上都表現得好 It means AI does not perform well on all tasks. [04:19] 有些任務搭配 AI 顯著加分 Some tasks significantly benefit from AI. [04:21] 但有些任務 AI 反而扯後腿 But on some tasks, AI actually hinders performance. [04:24] 第二個發現叫 Falling asleep at the wheel The second finding is called Falling asleep at the wheel. [04:26] 翻成中文就是在方向盤前打瞌睡 Translated into Chinese, it means dozing off in front of the steering wheel. [04:29] 當你不知道任務剛好是 AI 不擅長的 When you don't know that the task is exactly what AI is not good at, [04:32] 卻太信任它把產出直接送出 but trust it too much and submit its output directly, [04:34] 結果比沒用 AI 還慘 the result is worse than not using AI at all. [04:36] 也就是說，你要知道 AI 的邊界在哪 In other words, you need to know where AI's boundaries are, [04:39] 才不會在它弱的地方踩雷 so you don't step on landmines in its weak areas. [04:41] 第三個現象最有趣 The third phenomenon is the most interesting. [04:42] 他們發現使用 AI 的方式分成兩種 They found that the ways of using AI are divided into two types: [04:45] 分別是 Centaurs 跟 Cyborgs Centaurs and Cyborgs. [04:47] Centaur 是半人馬，也就是分工委派型 Centaur is a half-human, half-horse, meaning a division-of-labor delegation type. [04:50] 丟一個長 prompt 給 AI 叫它做整份簡報 You give AI a long prompt and ask it to make the entire presentation, [04:53] 自己去做別的事 while you go do other things. [04:55] Cyborg 是生化人，是高頻來回型 Cyborg is a cyborg, a high-frequency back-and-forth type. [04:57] 跟模型一句一句對話協作 You collaborate with the model sentence by sentence in dialogue. [04:59] Stanford 的教授說 A Stanford professor said, [05:01] 學生的使用習慣會比較像 cyborg students' usage habits are more like cyborgs, [05:03] 而企業在自動化 workflow 時比較像 centaur while enterprises automating workflows are more like centaurs. [05:06] 那你應該用哪一種？ So which one should you use? [05:07] 關鍵是看任務性質 The key is to look at the nature of the task. [05:10] 重複性高，流程清楚的任務 For tasks with high repetition and clear processes, [05:12] 用 centaur 模式委派出去就好 just delegate them using the centaur mode. [05:14] 需要判斷，需要創意，需要來回校正的任務 For tasks that require judgment, creativity, and iterative correction, [05:18] cyborg 模式才能逼出 AI 的最佳輸出 only the cyborg mode can extract the best output from AI. [05:21] 兩者沒有好壞之分，實務上兩種都會用 There is no good or bad between the two; in practice, both are used. [05:25] 重點是要有意識地切換 The key is to switch consciously. [05:27] 接著進到實作 Next, we move to implementation. [05:28] 通常一個好 prompt 要有三個東西 Usually, a good prompt needs three things: [05:30] 給誰看，產出格式，重點是什麼 who it is for, the output format, and what the focus is. [05:33] 請幫我總結這篇文章是一個很爛的提示詞 "Please summarize this article" is a very poor prompt. [05:36] 因為什麼資訊都沒給 Because it gives no information at all. [05:38] 但如果你寫的是請將這份再生能源論文 But if you write, "Please take this renewable energy paper [05:41] 整理成 5 個重點摘要 and organize it into 5 key points, [05:42] 並且聚焦在其背後的政策意涵上 and focus on the policy implications behind it," [05:45] 模型立刻知道對象是政策制定者 the model immediately knows the audience is policymakers, [05:48] 長度是 5 個點，重點放在政策意涵上 the length is 5 points, and the focus is on policy implications. [05:51] 你的產出品質會立刻提升 Your output quality will immediately improve. [05:54] 但教授說，prompt engineering 裡最常用 But the professor said that the most commonly used [05:56] 最重要的一個技巧不是這些，是 Prompt Chaining and most important technique in prompt engineering is not these, but Prompt Chaining. [06:00] 注意，chaining 跟 Chain of Thought 不一樣 Note, chaining is different from Chain of Thought. [06:03] Chain of Thought 是叫模型 step by step 思考 Chain of Thought tells the model to think step by step. [06:06] Chaining 則是把一個複雜的 prompt 拆成多個獨立的 prompt Chaining, on the other hand, breaks a complex prompt into multiple independent prompts. [06:09] 前一個的 output 餵給下一個 Feed the output of the previous one into the next. [06:11] 有用過 n8n 的朋友應該知道我在說什麼 Friends who have used n8n should know what I'm talking about. [06:14] 舉個例。 For example. [06:14] 你要做客戶投訴回信 You want to write a reply to a customer complaint. [06:17] 單一 prompt 寫法是讀這封投訴信 The single prompt approach is to read this complaint letter. [06:20] 寫一封專業的回應 Write a professional response. [06:22] 這就是一個黑盒子 This is a black box. [06:23] 最後的產出如果有問題 If there is a problem with the final output, [06:25] 你也不會知道該調整哪裡 you won't know where to adjust. [06:27] 但如果使用 prompt chaining But if you use prompt chaining, [06:29] 拆成三個 prompt break it into three prompts. [06:30] 第一個抽出客戶在抱怨什麼 The first extracts what the customer is complaining about. [06:32] 第二個用抽出來的問題起草大綱 The second drafts an outline using the extracted issue. [06:34] 第三個用大綱寫完整回信 The third writes the full reply using the outline. [06:37] 每一步都可以獨立測試，獨立 debug Each step can be tested and debugged independently. [06:40] Chaining 不只讓模型表現更好 Chaining not only makes the model perform better, [06:42] 也讓你得到 observability but also gives you observability. [06:44] 能觀察 LLM 在做什麼，哪個流程出了問題 You can observe what the LLM is doing and which process went wrong. [06:47] 講完 Prompt Engineering，下一個是 Fine-Tuning After discussing Prompt Engineering, the next topic is Fine-Tuning. [06:50] Stanford 教授對 fine-tuning 的立場很簡單 The Stanford professor's stance on fine-tuning is simple. [06:53] 能不做就不做，原因有四個 Avoid it if possible; there are four reasons. [06:56] 第一，要大量的優質數據 First, you need a large amount of high-quality data. [06:58] 如果你想要自己 fine-tune 模型 If you want to fine-tune the model yourself, [07:00] 就要有大量高品質的，標注好的資料 you need a large amount of high-quality, well-labeled data. [07:02] 要做到這件事情的成本對一般人來說太高了 The cost of doing this is too high for the average person. [07:06] 第二，容易 overfit Second, it is easy to overfit. [07:08] 模型容易在你那個特定任務上變很強 The model can easily become very strong on that specific task. [07:11] 但通用問題反而答不出來 But it can't answer general questions. [07:13] 就會失去 base model 原本的廣度 Then it loses the original breadth of the base model. [07:16] 第三，時效性差，這應該是最傷的點 Third, poor timeliness; this should be the most damaging point. [07:19] 你花兩個月 fine-tune 完一個模型上線 You spend two months fine-tuning a model and put it online. [07:21] 下個月新一代 base model 就出來 Next month, a new generation base model comes out. [07:23] 直接打贏你 fine-tune 的版本 It directly beats your fine-tuned version. [07:26] 第四，通常用 prompt engineering 也可以達到一樣的效果 Fourth, you can usually achieve the same effect with prompt engineering. [07:29] 而且成本低很多 And the cost is much lower. [07:30] 你想換新 base model You want to switch to a new base model. [07:31] 原本的 prompt 大多都是 portable 的 Most of the original prompts are portable. [07:34] 但 fine-tuning 的模型不行 But the fine-tuned model is not. [07:36] 當然，針對少數情境 Of course, for a few scenarios, [07:38] fine-tune 還是值得做的 fine-tuning is still worth doing. [07:40] 比方說法律，科學那種需要重複高精度輸出的領域 For example, fields like law and science that require repeated high-precision output. [07:44] 或者 base model 在某個 domain 上表現吃力 Or when the base model struggles in a certain domain. [07:47] 這些情況下，fine-tune 都會有幫助 In these cases, fine-tuning will be helpful. [07:49] 但對大多數的人來說 But for most people, [07:50] 其實不會有太大的效益 it actually won't bring much benefit. [07:52] 所以，如果你想要把特定的 domain knowledge 塞進模型 So, if you want to inject specific domain knowledge into the model, [07:55] 單純的提示詞工程可能塞不下 simple prompt engineering may not be able to contain it. [07:58] fine-tuning 的成本效益又對不起來 And the cost-effectiveness of fine-tuning doesn't add up. [08:00] 那實作端應該怎麼辦呢？ So what should the implementation side do? [08:02] Stanford 教授給出的答案是 RAG The answer given by a Stanford professor is RAG. [08:04] RAG 是 AI 工程師面試最常考的題目之一 RAG is one of the most common topics in AI engineer interviews. [08:08] 面試官常常會請你用 5 歲小孩聽得懂的方式解釋什麼是 RAG Interviewers often ask you to explain what RAG is in a way a 5-year-old can understand. [08:13] 你做 AI Builder 也好，做 AI 工程師也好 Whether you are an AI Builder or an AI engineer, [08:15] 這個是你一定要知道的事情 this is something you must know. [08:17] RAG 解的痛點有幾個 There are several pain points that RAG solves. [08:19] context window 太小，長 context 抓不準 The context window is too small, and long contexts are inaccurate. [08:22] 資訊有時效性，或者模型會產生幻覺 Information has timeliness, or the model may hallucinate. [08:25] RAG 的做法很單純 The approach of RAG is very simple. [08:27] 我用藥物副作用這個需要高度準確的醫療場景來說明 I will use the medical scenario of drug side effects, which requires high accuracy, to explain. [08:31] 通常 RAG 的做法是這樣的 Typically, the RAG approach is like this: [08:33] 先把所有的資料，文件用 embedding 模型 First, take all the data and documents and use an embedding model [08:36] 把這些資訊轉成向量，存進向量資料庫 to convert this information into vectors and store them in a vector database. [08:38] 英文叫做 vector database In English, it's called a vector database. [08:41] 所謂向量，就是把一段文字的語意轉換成一串數字 A vector means converting the semantics of a piece of text into a sequence of numbers. [08:45] 語意相近的文字 Texts with similar semantics [08:46] 轉出來的數字陣列在數學空間裡距離也會比較近 will have their resulting number arrays closer in mathematical space. [08:50] 所以當使用者問藥物 A 的副作用 So when a user asks about the side effects of drug A, [08:53] 我們不是在做關鍵字比對 we are not doing keyword matching, [08:55] 而是在找語意最接近的文件片段 but rather finding the document fragments with the closest semantics. [08:58] 就算文件裡寫的是不良反應而不是副作用 Even if the document says 'adverse reactions' instead of 'side effects,' [09:01] 一樣能找到 it can still be found. [09:02] 使用者的問題也用同個 embedding 模型轉成向量 The user's question is also converted into a vector using the same embedding model. [09:05] 再用距離 metric 從 vector database 裡 Then, using a distance metric, from the vector database, [09:08] 找出最相近的 documents find the most similar documents. [09:10] 最後把這些 documents 加上 system prompt Finally, combine these documents with the system prompt [09:12] 加上 user query，組合成餵給 LLM 的最終 prompt and the user query to form the final prompt fed to the LLM. [09:16] Prompt template 大概長這樣 The prompt template looks roughly like this. [09:18] 根據以下 documents 回答使用者的問題 Answer the user's question based on the following documents. [09:21] 如果 documents 裡沒有答案，就說『我不知道』 If the documents do not contain the answer, say 'I don't know'. [09:23] 這樣設計是為了把模型鎖在你提供的資料範圍內 This design is meant to lock the model within the scope of the data you provide. [09:27] 避免它自由發揮，憑空捏造 To prevent it from freely improvising and fabricating things out of thin air. [09:29] 當然，你還可以要求模型回答時附上這個答案來自第幾頁 Of course, you can also ask the model to include which page the answer comes from when responding. [09:34] 第幾章，第幾行加超連結 Which chapter, which line, and add a hyperlink. [09:36] 這樣使用者可以自行回溯資料源進行驗證 This way, users can trace back to the data source themselves for verification. [09:39] 但是單純這樣做有時候不夠 But doing just this is sometimes not enough. [09:41] 比方說一種藥的相關文件可能足足有 50 頁 For example, the related documents for a certain drug might be as long as 50 pages. [09:45] 整份直接轉成向量，那很多細節遺失其中 If the entire document is directly converted into vectors, many details are lost in the process. [09:49] 所以通常會搭配 chunking So it is usually paired with chunking. [09:51] 最基本的 chunking 是把文件切成固定大小的片段 The most basic chunking is to cut the document into fixed-size segments. [09:54] 每段各自轉成向量 Each segment is converted into a vector separately. [09:56] 更進階的做法是多層次存儲 A more advanced approach is multi-level storage. [09:58] 同時保留整篇，每章，每段的向量 Simultaneously retain the vectors for the whole document, each chapter, and each paragraph. [10:01] retrieval 時可以先找到相關的章節 During retrieval, you can first find the relevant chapter. [10:03] 再往下鑽到精確的段落 Then drill down to the exact paragraph. [10:05] 這樣在大文件裡的命中率會更高 This way, the hit rate in large documents will be higher. [10:08] 還有一個最近比較多人在討論的話題 There is also a topic that many people have been discussing recently. [10:10] 有人說，像現在已經有模型支援超長的 context window Some say that models already support extremely long context windows now. [10:14] 等這個技術成熟，算力夠便宜 Once this technology matures and computing power becomes cheap enough. [10:17] Base model 直接讀完整個資料庫 The base model directly reads the entire database. [10:19] RAG 就沒用了 Then RAG would be useless. [10:20] 但 Stanford 的教授認為這樣的說法理論上對 But a Stanford professor believes that this statement is theoretically correct. [10:23] 實務上錯 But practically wrong. [10:25] 因為實際上你還是會遇到很多問題 Because in reality, you will still encounter many problems. [10:27] 比方說 Latency，你想想看 For example, latency—think about it. [10:30] 每次問問題，模型都要把整個 Google Drive 重讀一次 Every time you ask a question, the model has to read the entire Google Drive again. [10:33] 沒人等得了 No one can wait for that. [10:35] 就像搜尋引擎也是靠預先建好的索引來快速定位資料 Just like search engines rely on pre-built indexes to quickly locate data. [10:38] 不可能每次 query 都把整個網路重新爬過一遍 It's impossible to crawl the entire web again for every query. [10:41] 所以 RAG 除了準確度以外 So, besides accuracy, RAG also has advantages such as retrieval efficiency and the ability to update in real time. [10:43] 還有檢索效率，可即時更新這類的優勢 It also has advantages like retrieval efficiency and real-time updates. [10:46] 而這些優勢在可預見的未來都還是有其存在的價值 And these advantages will still have their value in the foreseeable future. [10:50] 好，到這邊我們講完了影片的第一部分 Okay, so far we have finished the first part of the video. [10:53] 從 base LLM 的限制出發 Starting from the limitations of the base LLM, [10:54] 我們看了 Prompt Engineering we looked at Prompt Engineering, [10:56] Fine-Tuning 還有 RAG Fine-Tuning, and RAG. [10:56] 這三個工具本質上都是在強化單一個 LLM 的能力 These three tools are essentially about enhancing the capability of a single LLM. [11:00] 但單一個 LLM 再強，還是有它做不到的事 But no matter how powerful a single LLM is, there are still things it cannot do. [11:03] 所以接下來進入第二部分 AI 系統設計 So next, we move into the second part: AI system design. [11:06] 我們會講 Agentic Workflow We will talk about Agentic Workflow, [11:08] Evaluation 還有 Multi-Agent Evaluation, and Multi-Agent. [11:09] 這些東西讓你能把 AI 從一個會回答問題的模型 These things allow you to turn AI from a model that answers questions [11:13] 變成一套真正可以運作，實際產出價值的系統 into a system that can truly operate and produce real value. [11:17] 看到這邊，如果你覺得這支影片對你有幫助 Seeing this, if you think this video is helpful to you. [11:20] 想請你幫我按個讚，點個追蹤 I would like to ask you to give me a like and follow. [11:21] 這是我持續創作下去的動力 This is the motivation for me to keep creating. [11:23] 那我們直接進 Part 2 Then let's go directly to Part 2. [11:25] 先講一下 agentic workflow 這個命名 First, let's talk about the naming of agentic workflow. [11:27] 這個詞來自吳恩達 This term comes from Andrew Ng. [11:29] 如果你不認識他 If you don't know him, [11:30] 吳恩達是 Coursera 的共同創辦人 Andrew Ng is the co-founder of Coursera, [11:32] Google Brain 的創始負責人 the founding lead of Google Brain, [11:34] 前百度首席科學家 former chief scientist at Baidu, [11:35] 是在 AI 領域講話有一定份量的男人 a man whose words carry weight in the AI field. [11:38] 吳恩達用 agentic workflow 這個詞 Andrew Ng uses the term agentic workflow [11:40] 是因為 AI Agent 已經被用到爛掉了 because AI Agent has been overused to the point of being worn out. [11:43] 有人寫了一個很長的 prompt，叫 agent Someone writes a very long prompt and calls it an agent, [11:46] 有人做了複雜的 multi-agent 系統 someone builds a complex multi-agent system [11:48] 也叫 agent and also calls it an agent. [11:49] 這個詞什麼都能套，反而什麼都說不清楚 This term can be applied to anything, but instead it explains nothing clearly. [11:52] 所以他用 agentic workflow 來精確描述一件事 So he uses agentic workflow to precisely describe one thing: [11:56] 把一堆提示詞，外部工具，還有各種元件 taking a bunch of prompts, external tools, and various components, [11:59] 組合進一個有結構的工作流程裡，成為一套系統 combining them into a structured workflow to form a system. [12:02] 這就叫做 agentic workflow This is called agentic workflow. [12:05] 還記得我們前面講的 RAG 嗎？ Do you still remember the RAG we talked about earlier? [12:07] RAG 主要做一件事 RAG mainly does one thing: [12:08] 給 LLM 外部資料當參考 providing external data to the LLM as a reference. [12:10] 但 agent 把 RAG 當作工具之一 But the agent treats RAG as one of its tools, [12:13] 外加 tool calls，memory，多步驟決策 plus tool calls, memory, and multi-step decision-making, [12:15] 所以能做到 RAG 單獨做不到的事 so it can do things that RAG alone cannot. [12:18] 舉個例子，使用者說我想退這筆訂單 For example, the user says I want to cancel this order. [12:21] RAG 只能丟政策文件給你 RAG can only throw policy documents at you. [12:24] agent 則是用 RAG retrieve 政策 The agent, on the other hand, uses RAG to retrieve policies. [12:25] 主動問訂單編號，用 tool 查訂單 It proactively asks for the order number and uses a tool to look up the order. [12:28] 確認退費 Confirm the refund. [12:29] 告訴你 3 到 5 個工作天會處理 It tells you it will be processed in 3 to 5 business days. [12:31] RAG 是工具，Agent 是使用 RAG 這個工具的系統 RAG is a tool, and the Agent is the system that uses the RAG tool. [12:35] 要打造 agentic workflow To build an agentic workflow, [12:36] 你的工程心態要先翻轉過來 your engineering mindset must first be flipped. [12:39] 因為傳統 software 跟 agentic AI software Because traditional software and agentic AI software [12:42] 在四個面向都很不一樣 are very different in four aspects. [12:44] 首先是資料 First is data. [12:45] 傳統軟體吃結構化資料 Traditional software consumes structured data. [12:47] JSON，資料庫，表單，格式固定，邊界清楚 JSON, databases, forms — fixed format, clear boundaries. [12:50] Agentic 系統吃的是自由文本 Agentic systems consume free text, [12:53] 圖片，音訊，沒有固定格式 images, audio — no fixed format. [12:56] 第二個是邏輯 Second is logic. [12:57] 傳統 software 是 deterministic Traditional software is deterministic. [12:59] 同樣的 input，永遠給你同樣的 output The same input always gives you the same output. [13:03] 可預測，可重現 Predictable and reproducible. [13:05] Agentic 系統是 fuzzy 的 Agentic systems are fuzzy. [13:07] 意思是同樣的 input Meaning, with the same input, [13:08] 不同時間跑可能給你不同的 output running at different times may give you different outputs. [13:11] 因為 LLM 本身有隨機性 Because the LLM itself has randomness, [13:13] 加上它會根據 context 做判斷 plus it makes judgments based on context. [13:15] 沒有一個固定答案 There is no fixed answer. [13:17] 第三個是架構心態，這個最重要 Third is the architectural mindset — this is the most important. [13:20] 傳統工程師的思維是寫 microservices The traditional engineer's mindset is to write microservices. [13:22] 寫 monolith Write monolith. [13:23] 你精確控制每一步執行路徑 You precisely control every step of the execution path. [13:26] Agentic 系統的思維是 think like a manager The mindset of an agentic system is to think like a manager. [13:28] 你給 AI 一個目標和限制，讓它自己決定怎麼完成 You give the AI a goal and constraints, letting it decide how to accomplish it. [13:32] 你管的是方向和邊界，不是每一行程式碼 What you manage is the direction and boundaries, not every line of code. [13:36] 第四個是測試 The fourth is testing. [13:37] 傳統測試是確定性的，跑一百次結果一樣 Traditional testing is deterministic; running it a hundred times yields the same result. [13:40] Agentic 測試是迭代探索式的 Agentic testing is iterative and exploratory. [13:43] 因為系統本身是非確定性的 Because the system itself is non-deterministic. [13:45] 加上它對 context 非常敏感 Plus, it is very sensitive to context. [13:46] 你沒辦法窮舉所有情況 You cannot exhaustively enumerate all situations. [13:48] 知道這四件事之後 After knowing these four things, [13:49] 第一個落地原則就跟著來了 the first implementation principle follows. [13:51] 能 deterministic 解的問題，就 deterministic 解 For problems that can be solved deterministically, solve them deterministically. [13:54] 剩下 fuzzy 的部分，加上護欄 For the remaining fuzzy parts, add guardrails. [13:58] 教授舉自己的例子 The professor gave his own example. [13:59] 他們做 skills assessment They do skills assessment. [14:01] 選擇題，配對題，拖拉題用 deterministic 算分 Multiple-choice questions, matching questions, and drag-and-drop questions are scored deterministically. [14:04] 因為這些有標準答案，對就是對，錯就是錯 Because these have standard answers; right is right, wrong is wrong. [14:07] 但語音題，語音加 coding 的混合題型 But for voice questions, or mixed question types combining voice and coding, [14:10] 沒有標準答案，沒辦法 deterministic 評分 there are no standard answers, so deterministic scoring is not possible. [14:12] 只能讓 LLM 去判斷 We can only let the LLM judge. [14:14] 這就是 fuzzy scoring This is fuzzy scoring. [14:15] 也就是沒有標準答案的評分 That is, scoring without standard answers. [14:17] 你請 LLM 聽一段語音回答 You ask the LLM to listen to a voice response. [14:20] 它要判斷這個人有沒有真的理解這個概念 It needs to determine whether this person truly understands the concept. [14:22] 表達是否清晰，邏輯有沒有跑掉 Whether the expression is clear and the logic is coherent. [14:24] 這些不是對或錯，是一個程度的判斷 These are not about right or wrong, but a judgment of degree. [14:28] LLM 會給你一個分數 The LLM will give you a score. [14:29] 但這個分數是它覺得合理的答案，不是算出來的 But this score is what it considers a reasonable answer, not something calculated. [14:34] Fuzzy 的問題是它一定會犯錯 The fuzzy problem is that it will inevitably make mistakes. [14:36] LLM 可能誤判一個正確答案 The LLM might misjudge a correct answer. [14:38] 或是對模稜兩可的回答給出偏低的分數 Or give a low score to an ambiguous response. [14:41] 在考試評分這種高風險的場景，這不可接受 In high-stakes scenarios like exam grading, this is unacceptable. [14:44] 所以他們設計了一個 Appeal feature So they designed an appeal feature. [14:46] 受測者可以對 agent 的判分提出申訴 The test taker can appeal the agent's scoring. [14:49] 由真人介入審查並糾正 A human intervenes to review and correct it. [14:52] 這就是護欄的具體形式 This is the concrete form of the guardrail. [14:53] 不是試圖讓 AI 零錯誤 It's not about trying to make AI have zero errors. [14:55] 而是在它出錯的時候有人接得住 But rather having someone to catch it when it makes mistakes. [14:58] 好，原則講完了 Okay, the principles are covered. [15:00] 要實際打造一個 agent，你需要哪些東西？ To actually build an agent, what do you need? [15:03] 教授用訂機票去巴黎當範例 The professor used booking a flight to Paris as an example. [15:05] 整理出三個核心要素 He summarized three core elements. [15:07] 第一個是 Prompts The first is Prompts. [15:08] 前面講的提示詞工程，在 agent 裡就是這一塊 The prompt engineering mentioned earlier is exactly this part in the agent. [15:12] 你怎麼告訴 AI 它的角色是什麼 How do you tell the AI what its role is? [15:14] 它能做什麼，不能做什麼 What it can and cannot do. [15:15] 第二個是 Context Management The second is Context Management. [15:17] 也就是怎麼管理 agent 在每一個當下看得到的資訊 That is, how to manage the information the agent can see at any given moment. [15:21] 這包含幾件事 This includes several things. [15:23] Memory，對話歷史 Memory, conversation history. [15:24] 也可能有 RAG 撈回來的資料 There may also be data retrieved by RAG. [15:26] 通通都要塞進有限的 context window 裡 All of it must be squeezed into a limited context window. [15:28] 你要決定什麼重要，什麼可以丟，什麼要壓縮 You need to decide what is important, what can be discarded, and what should be compressed. [15:32] Context Management 本質上只做一件事 Context Management essentially does only one thing. [15:34] 那就是把對的資訊在對的時間提供給你的 agent That is to provide the right information to your agent at the right time. [15:38] Memory 本身又分兩層 Memory itself is divided into two layers. [15:40] Working memory 是高頻，要快的 Working memory is high-frequency and needs to be fast. [15:42] 比方說使用者的名字，這次的目的地是巴黎 For example, the user's name, the destination this time is Paris. [15:46] Archival memory 是低頻，可以慢一點的 Archival memory is low-frequency and can be slower. [15:49] 比方說使用者過去五年的訂房紀錄 For example, the user's booking records from the past five years. [15:52] 需要的時候再去撈 Retrieve it when needed. [15:54] 第三個是 Tools The third is Tools. [15:55] 也就是 agent 能呼叫的外部能力 That is, the external capabilities that the agent can call. [15:58] 通常分成兩種 It is usually divided into two types. [15:59] 一種是做事的 One type is for doing things. [16:00] 比方說 Flight Search，Hotel Booking，Payment For example, Flight Search, Hotel Booking, Payment. [16:02] 一種是查資料的 One type is for querying data. [16:04] 比方說去 CRM 撈客戶資料 For example, retrieving customer data from CRM. [16:05] 去資料庫查訂單紀錄 Querying order records from the database. [16:07] 另外，課堂上教授把 agent 的自主性分為三層 Additionally, in class, the professor divided the autonomy of agents into three levels. [16:11] 最低是 hardcoded steps The lowest is hardcoded steps. [16:13] 步驟全寫死，先識別用戶的意圖 All steps are fixed; first identify the user's intent. [16:16] 再 lookup history，最後呼叫 API，照順序走 Then look up history, and finally call the API, following the sequence. [16:19] 安全，可預測，但很僵硬 Safe, predictable, but very rigid. [16:22] 遇到預期外的情況就卡住 It gets stuck when encountering unexpected situations. [16:24] 第二層是 hardcoded tools The second layer is hardcoded tools. [16:25] 但讓 agent 自己決定執行步驟 But let the agent decide the execution steps itself. [16:28] 你給它一組工具，告訴它你是 travel agent You give it a set of tools and tell it you are a travel agent. [16:31] 這些是你能用的工具，怎麼用你決定 These are the tools you can use; how to use them is up to you. [16:34] 掌控工具範圍，但給 agent 空間判斷怎麼組合 Control the scope of tools, but give the agent room to decide how to combine them. [16:38] 這是目前最常見的 production setup This is currently the most common production setup. [16:40] 也是教授推薦的起點 It is also the starting point recommended by the professor. [16:42] 第三層是 fully autonomous The third layer is fully autonomous. [16:44] agent 自己決定步驟，甚至自己創工具 The agent decides the steps itself and even creates its own tools. [16:48] 給它 code editor，給它 web search Give it a code editor, give it web search. [16:50] 叫它自己寫 code 解決問題 Tell it to write its own code to solve problems. [16:52] 能力最強，但風險也最高 It has the strongest capability, but also the highest risk. [16:55] 你沒辦法完全預測它會做什麼 You cannot fully predict what it will do. [16:57] 如果 agent 判斷錯誤，自己訂了 100 張機票 If the agent makes a wrong judgment and books 100 tickets on its own, [16:59] 你就完蛋了 you are done for. [17:01] 還有一個關鍵概念是 MCP Another key concept is MCP. [17:03] Model Context Protocol Model Context Protocol. [17:05] 傳統做法是你要替每一個 API 單獨寫串接邏輯 The traditional approach is that you have to write integration logic for each API individually. [17:08] 教 LLM 這個 API 怎麼用 Teach the LLM how to use this API, [17:10] 那個 API 要傳什麼參數 what parameters to pass to that API. [17:11] MCP 的做法是在中間放一個協議層 The MCP approach is to place a protocol layer in the middle. [17:15] agent 不需要認識每一個 API The agent does not need to know every API. [17:17] 它只需要跟 MCP server 溝通 It only needs to communicate with the MCP server. [17:19] MCP 負責幫它跟後面的服務打交道 MCP is responsible for dealing with the backend services on its behalf. [17:22] 你可以把 MCP 想成一個通用插頭 You can think of MCP as a universal plug. [17:24] 以前每個國家的插座規格不一樣 In the past, each country had different socket specifications. [17:26] 你要帶一堆轉接頭 You had to bring a bunch of adapters. [17:28] 有了 MCP，插一個就全通了 With MCP, plugging in one makes everything work. [17:31] Stanford 教授還有提到對 MCP 一個更大的想像 The Stanford professor also mentioned a bigger vision for MCP. [17:34] 那就是 agent-to-agent communication That is agent-to-agent communication. [17:36] 你可以把別人做好的 agent 當作一種工具 You can treat someone else's agent as a tool. [17:39] 讓你自己的 agent 去呼叫它 Have your own agent call it. [17:40] 就像現在 agent 呼叫 API 一樣 Just like how agents call APIs now. [17:43] 這是 multi-agent 系統的基礎 This is the foundation of a multi-agent system. [17:44] 我們等一下會講到 We will talk about it later. [17:46] 不管是單一 agent 還是 multi-agent Whether it's a single agent or multi-agent, [17:47] 上線之前都要先回答同一個問題 before going live, you must first answer the same question. [17:49] 怎麼知道它真的有用？答案是 eval How do you know it really works? The answer is eval. [17:52] 也是 production agentic 系統的命脈 It is also the lifeline of production agentic systems. [17:54] 教授在課堂上給了一個完整的 eval 框架 The professor gave a complete eval framework in class. [17:56] 是三個維度交叉 It is a cross of three dimensions. [17:58] 第一個維度是 End-to-End 還有 Component-based The first dimension is End-to-End and Component-based. [18:02] End-to-end 是看整體 End-to-end looks at the whole. [18:04] 使用者用完給幾分，滿不滿意 After using it, the user gives a score, whether they are satisfied or not. [18:08] Component-based 是拆開每一步看 Component-based looks at each step separately. [18:09] 比方說 For example, [18:10] 這個 tool 老是忘記更新 email this tool always forgets to update the email. [18:11] 送 email 那一步格式不對 The format of the email sending step is wrong. [18:13] 光看整體你知道哪裡壞，但不知道為什麼壞 Looking only at the whole, you know where it's broken, but not why it's broken. [18:16] 光看 component 你可能修了細節但整體體驗還是差 Looking only at components, you might fix details but the overall experience is still poor. [18:20] 實務上兩個都要做 In practice, you need to do both. [18:22] 第二個維度是 Objective 還有 Subjective The second dimension is Objective and Subjective. [18:25] 也就是客觀和主觀的評價方式 That is, the objective and subjective evaluation methods. [18:28] Objective 是可以自動驗證的 Objective can be automatically verified. [18:30] 使用者說 order ID 是 X The user says the order ID is X. [18:31] LLM 寫進 DB 變成 Y The LLM writes it into the database as Y. [18:33] 這種東西你可以寫個 Python script 自動對齊 For this kind of thing, you can write a Python script to automatically align them. [18:36] 因為對就是對，錯就是錯 Because right is right, and wrong is wrong. [18:39] Subjective 是沒有標準答案的 Subjective has no standard answer. [18:41] 語氣好不好，回答夠不夠有同理心 Whether the tone is good, whether the response is empathetic enough. [18:43] 這種要靠人工評分 This kind of thing relies on human scoring. [18:44] 或是用另一個額外的 LLM 當作評審 Or use another additional LLM as a judge. [18:47] 這個等等我會細講 I will elaborate on this later. [18:49] 第三個維度是 Quantitative 還有 Qualitative The third dimension is Quantitative and Qualitative. [18:52] Quantitative 是數字 Quantitative is numbers. [18:54] 改地址成功率幾趴，每個環節的延遲多久 What is the success rate of address changes, and how long is the delay in each step. [18:58] Qualitative 是感覺 Qualitative is feeling. [18:59] 在哪裡幻覺，語氣哪裡不對 Where the hallucination occurs, where the tone is off. [19:02] 使用者哪一步卡住 Which step the user gets stuck on. [19:03] 這個要人工一筆一筆看，沒有捷徑 This requires manual review item by item; there is no shortcut. [19:06] 剛剛提到的用 LLM 當作評審 The previously mentioned using an LLM as a judge. [19:08] 又稱 LLM-as-Judge Also known as LLM-as-Judge. [19:10] 這是做 subjective eval 常用的做法 This is a common approach for doing subjective evaluation. [19:13] 設計上有四種主流玩法 There are four mainstream approaches in design. [19:15] 第一種是 Pair-wise comparison The first is pair-wise comparison. [19:17] 給 judge 兩個答案，問它哪個比較好 Give the judge two answers and ask which one is better. [19:20] 第二種是 Single-answer grading The second is single-answer grading. [19:22] 直接打一到五分 Directly rate from one to five. [19:23] 第三種是 Reference-guided pair-wise The third is reference-guided pair-wise. [19:25] 多給一個標準答案做對比，讓評分更有依據。 Provide a standard answer for comparison to make scoring more grounded. [19:29] 第四種是 Rubric-based。 The fourth type is rubric-based. [19:30] 你自己定義評分標準。 You define the scoring criteria yourself. [19:32] 比方說五分等於一百字以內，含三個重點。 For example, five points equals within one hundred words and contains three key points. [19:35] 第一句是 overview。 The first sentence is an overview. [19:36] 零分等於答非所問，冗長失焦。 Zero points equals an irrelevant answer, verbose and off-topic. [19:38] 這四種可以混用。 These four types can be mixed. [19:40] 比方說 rubric-based 加上 few-shot examples。 For example, rubric-based plus few-shot examples. [19:42] 讓評審更知道你要什麼。 To let the evaluator better know what you want. [19:45] 怎麼實際跑一個 subjective eval？ How to actually run a subjective eval? [19:47] 課堂上教授用 travel agent 的禮貌度評估當例子。 In class, the professor used the politeness evaluation of a travel agent as an example. [19:50] 共拆成四個步驟。 It is broken down into four steps. [19:52] 第一步是 error analysis。 The first step is error analysis. [19:54] 從一千個使用者裡抽二十個對話，人工讀。 Sample twenty conversations from a thousand users and read them manually. [19:58] 你可能就會發現 LLM 講話超短。 You might find that the LLM speaks very briefly. [20:00] 有點機車，沒同理心。 A bit annoying, lacking empathy. [20:02] 這步不能省，你要先知道問題長什麼樣。 This step cannot be skipped; you need to first know what the problem looks like. [20:05] 才能設計出對的 eval。 Only then can you design the right eval. [20:07] 第二步是設計 eval。 The second step is to design the eval. [20:08] 用 LLM-as-Judge 加上自己寫的禮貌度 rubric。 Use LLM-as-Judge plus your own politeness rubric. [20:11] 把你在第一步發現的問題翻譯成評分標準。 Translate the issues you found in the first step into scoring criteria. [20:15] 第三步是 A/B test 模型。 The third step is to A/B test the models. [20:17] 固定 prompt，把底層模型從 GPT 換成 Opus 或者其他模型。 Fix the prompt, swap the underlying model from GPT to Opus or other models. [20:21] 然後跑同一批對話，judge 評分。 Then run the same set of conversations and have the judge score them. [20:24] 看哪個模型禮貌度最高。 See which model has the highest politeness. [20:26] 第四步是 A-B test prompt [20:28] 固定模型 [20:29] 把 act like a travel agent 改成 [20:31] act like a helpful travel agent [20:33] 看一個詞的差距影響有多大 [20:36] 核心原則只有一個 [20:37] 先人工掃出問題，再設計自動化 eval [20:41] 而且模型跟 prompt 這兩個變因一次只動一個 [20:43] 不然不知道是哪個改動造成了差異 [20:46] 講完 eval，教授給了一個完整的 case study [20:48] 把前面所有東西串起來 [20:50] 題目很簡單 [20:51] 我們要做一個客服 AI agent [20:53] 使用者可能會說 [20:54] 『我要改 A127 訂單的地址，因為我搬家到建國南路了』 [20:59] 這樣的 AI 客服 agent 你會怎麼做？ [21:02] 這題目有兩層 [21:03] 表面看是怎麼改地址 [21:04] 本質是怎麼做一個會自己處理客服請求的 agent [21:08] 課堂上一個學生回答說 [21:10] 我會先去客服旁邊坐一到兩天 [21:12] 看他們實際怎麼處理這種請求 [21:15] 他們日常的工作流是什麼 [21:17] 教授很喜歡這個答案 [21:18] 因為你要先理解人怎麼做這件事 [21:21] 才知道怎麼讓 AI 做 [21:22] 而這就是做 agentic workflow 的第一步 [21:24] 叫做 task decomposition [21:26] 也就是把大任務拆解成小任務 [21:29] 讓 LLM 可以逐個擊破 [21:31] 觀察之後，你會發現客服處理一個改地址請求 [21:35] 其實走了五步 [21:36] 第一步，抽出關鍵資訊 [21:38] 包含客戶的 intent 是什麼 [21:41] order ID 是哪個，新地址是什麼 [21:43] 第二步，去資料庫查客戶紀錄 [21:46] 第三步，查公司政策 [21:48] 像是這筆訂單能不能改地址？是不是已經出貨了？ [21:52] 第四步，根據前面收集到的資訊起草回信 [21:55] 第五步，送出 email [21:57] 每一步都明確，這就是 task decomposition [22:00] 有了拆解，下一步是決定每一步用什麼工具 [22:03] 第一步抽資訊 [22:05] 單純用 LLM 的一次 API 呼叫通常就能解決 [22:08] 第二步查和改 Database [22:10] 要 custom tool 或 MCP server [22:12] 第三步政策檢查，用 RAG [22:14] 因為政策文件會更新，也需要有效率的檢索 [22:17] 不能讓客戶等太久 [22:19] 第四步是根據前幾步搜集到的資訊 [22:21] 開始撰寫 email [22:23] 第五步送 email [22:24] 用 agent 能夠使用的 email 寄送工具 [22:28] 做這些判斷的方式很簡單 [22:30] 先問自己哪些步驟是 fuzzy [22:32] 哪些是 deterministic [22:33] 再決定每一步是用 LLM one-shot [22:36] RAG，tool，還是其他工具 [22:38] 這就是 AI Builder 真實在做的工作 [22:40] 知道每個工具，每種技術的能力和限制 [22:43] 然後炒出一盤你需要的菜 [22:45] 最後是 build evals [22:47] 把前面講的三個維度全部用上 [22:49] End-to-end 看最終回覆正確性 [22:52] 語氣，使用者滿意度 [22:54] Component-based 拆開看每一步 [22:56] 抽資訊的準度，API 錯誤率，政策遵守率 [22:59] Objective 的部分可以自動驗證 [23:01] 抽出來的 order ID 是否正確 [23:03] 有沒有違反退費政策 [23:05] Subjective 的部分要靠人工加 LLM-as-Judge [23:08] 比方說回信是否有禮貌，有沒有同理心 [23:11] Quantitative 看改地址成功率 [23:14] latency，退費正確率 [23:17] Qualitative 看哪裡有幻覺 [23:18] 哪裡語氣不一致 [23:20] 使用者在哪一步感到困惑 [23:22] 到這邊，一個完整的 AI 客服 agent [23:24] 從零到上線的工程流程就出來了 [23:26] 其實總共就是三個步驟 [23:28] 先把大任務拆解成小任務 [23:30] 再設計工作流程 [23:31] 最後建立評估系統，確保產出穩定 [23:34] 這堂課的最後一個主題是 Multi-Agent [23:36] 前面一個 agent 已經能拆步驟 [23:38] call tool，做 RAG，寫信 [23:40] 看起來夠用了 [23:42] 但 multi-agent 還是有它存在的理由 [23:44] 主要原因是平行處理 [23:46] 有些事沒理由排隊跑 [23:48] 訂機票的時候，找航班，找飯店，查天氣 [23:52] 這三件事完全可以同時進行 [23:54] 如果只有一個 agent，就只能一件一件做 [23:57] Multi-agent 就是把這幾件可以平行的事 [23:59] 拆給三個 specialized agent 同時跑 [24:02] 速度直接拉起來 [24:04] 次要原因是 reusability，也就是可復用性 [24:07] 公司裡的 design agent 可以給行銷團隊用 [24:10] 也可以給產品團隊用 [24:12] 不過，multi-agent 聽起來很厲害 [24:15] 但做產品的時候要先問自己真的有需要嗎？ [24:18] 如果一個 agent 就能解決的任務 [24:20] 硬上 multi-agent 反而增加複雜度 [24:23] 所以還是老話一句，工程設計能簡單就簡單 [24:26] 不要 over design [24:27] 教授用智慧家庭當 brainstorm 範例 [24:30] 一個完整的智慧家庭 multi-agent 系統可能有溫度控制 [24:33] 燈光，保全，娛樂，通知 [24:36] 還有能源管理這些 agent [24:37] 加上一個 Orchestrator 統籌 [24:39] 互動模式有兩種 [24:41] 第一種是 Hierarchical [24:42] 使用者只跟 orchestrator 講話 [24:43] 由它派工給下面的 agent，指揮鏈清楚 [24:47] 第二種是 Flat [24:48] agent 之間直接互通，沒有中間人 [24:51] 教授建議智慧家庭以 hierarchical 為主 [24:54] 因為從使用者的角度 [24:55] 你不想同時跟五個 agent 講話 [24:57] 你只想跟一個 assistant 說我要出門了 [25:00] 它就會自己去協調燈光，保全，溫控 [25:03] 但在後台，某些 agent 之間可以有彼此之間水平溝通的連線方式 [25:09] 比方說溫度控制的 agent 跟能源管理的 agent 直接互通 [25:13] 省掉每次都要過 orchestrator 的溝通成本 [25:16] 當你讓 agent 之間互相溝通 [25:18] 本質上就是 MCP protocol [25:20] 你把 agent 當作 tool [25:21] 就跟把 API 當作 tool 一樣 [25:24] 這個心態一旦想通，設計就清楚了 [25:27] 每個 agent 對外暴露一組 tool-like 介面 [25:28] 其他 agent 像呼叫工具一樣呼叫這個 agent [25:31] 這樣子做結構乾淨，容易 debug [25:33] 也支援平行處理 [25:35] Multi-agent 聽起來複雜 [25:36] 但其實只是把你已經懂的東西，多疊一層 [25:39] 好，那我們最後整理一下這堂課的重點 [25:42] 這支影片我們從基礎模型的限制開始講起 [25:45] 並帶到能夠加強模型表現的技巧和技術 [25:47] 首先是 Prompt Engineering [25:49] 強化 LLM 輸出，成本最低 [25:51] 重點放在 chaining 跟 testing [25:53] 第二層是 Fine-Tuning [25:54] 除非你在處理法律，科學那種需要重複高精度的 domain [25:58] 或者興趣使然 [25:59] 否則別沒事找事去調模型 [26:02] 第三層是 RAG [26:03] 幫模型補足知識的標準解法 [26:06] 做 AI 產品基本上一定會碰到 [26:08] 最後是 Agentic Workflow [26:10] 從強化單一 LLM 進到系統設計 [26:13] 重點是思維上的心態轉變 [26:15] 從過往的 deterministic engineering [26:17] 到現在的 fuzzy engineering [26:19] 以及應對 fuzzy engineering 的 evaluation 系統 [26:22] 還有在實戰上要如何從拆解任務開始 [26:25] 打造自己的 agentic workflow [26:27] 第五層是 Multi-Agent [26:28] 把每個 agent 都當成工具 [26:30] 彼此協作來提升處理速度跟復用性 [26:33] 教授在兩個小時內跑完產業上常見的實作技巧 [26:36] 每個技術的存在理由跟適用情境都交代清楚 [26:39] 有了這樣的認知，你就不會盲目跟風 [26:42] 看到別人在做 multi-agent 就跟著做 [26:44] 看到別人講 fine-tuning 覺得很帥就去玩自己的模型 [26:47] 所以你的下一步是什麼？ [26:48] 我給你的建議是，從實作中學習 [26:51] 看看你的生活上或者工作上有什麼痛點 [26:54] 從痛點出發去思考解決方法 [26:56] 在這個過程中你就會發現自己需要學會哪些技術 [27:00] 就能夠很有效率的去規劃自己的學習路線 [27:03] 我把這支影片整理成了一篇深度文章 [27:05] 附帶 4 組 prompts [27:06] 幫你想清楚自己的 workflow 怎麼拆 [27:08] 現有架構有沒有 over-engineering [27:10] 幫你建立 eval 系統 [27:12] 以及該不該上 multi-agent [27:13] 有興趣的朋友可以在資訊欄找到連結 [27:15] 那今天這支影片差不多就講到這邊 [27:17] 如果你喜歡這樣的內容 [27:19] 想請你幫我點個讚 [27:20] 也在底下留言告訴我你們想看什麼樣的內容 [27:22] 那我們下次見