# Writing & Thinking: Claude Code for Economists with Paul Goldsmith-Pinkham | Markus Academy | 162-5

https://www.youtube.com/watch?v=BxfSiB3Moyo

[00:08] So, welcome back everybody to another see Claude code for economist for applied economist a video from the mini video series with Paul Goldsmith Pinkham.
[00:17] Hi Paul.
[00:19] Hi Marcus, how are you doing?
[00:22] Today we have the fourth mini video.
[00:23] We talk about writing and reporting.
[00:27] In particular, we will talk the indowing Claude with your personal voice as a first part.
[00:32] The second part will be about refereeing and in particular strategically revising your paper once you get the paper back from the editor with referee reports.
[00:41] And the third component today will be thinking through ideas and note taking with LLMs.
[00:49] Paul, the floor is yours.
[00:51] So, what we're going to do is we're going to kind of talk about three things.
[00:55] Um
[00:56] This is should say one, two, and three, but it's one, three, and five.
[00:58] Um we're going to say uh you know, first we're going to just talk about a writing guide.
[01:04] So, I want to kind of show you what that would look like.
[01:06] Then I'm going to show you two
[01:08] skills.
[01:11] Um one well, one is building your own skill, how you might do it, and
[01:13] the other is using a skill that I happen to find um quite powerful for thinking about referee responses.
[01:20] And then lastly, I want to kind of just talk about how you can use it for thinking about um writing and editing, and that would be a kind of very brief.
[01:32] So, let me start by talking about a style guide.
[01:34] So, we're here we're going to just pause here.
[01:36] I just want to say like this is a pretty straightforward exercise.
[01:40] I'm going to What we're going to do is we're going to build a skill or a style guide based on our own writing.
[01:46] So, I'm going to do it for mine, but Marcus, you could just as easily do this for yours.
[01:49] Um
[01:51] and you'll kind of see in the end what this going to do is it's going to build like a basically a list of rules of what it thinks about how to tweak the LLM's behavior.
[02:00] It's not necessarily going to capture my voice perfectly, but it will it's almost like a what happens when you take a complicated multifaceted person and try and project them on the wall.
[02:09] Well, this like it's going to capture some aspect of it, and the question is is like is that going to do it correctly?
[02:17] Take a lot a lot of my writings to Claude and say, "Please extract my voice."
[02:21] yeah, so here's so I'm going to do it with just my papers, which of course is not a perfect example cuz I didn't write every paper.
[02:28] I co you know, I co-authored all of them.
[02:30] But, what we're going to do is we're going to take samples of uh writing that I'm happy with, and I'll ask it to analyze it.
[02:36] So, what I'm going to do is the following.
[02:37] So, I'll say
[02:38] So, um we're going to do that right now.
[02:41] So, I'm going to pause stop sharing here, Marcus.
[02:45] And what we'll do is I'm going to set up and we're going to actually use um Claude co-work to do exactly this if that works.
[02:52] So, if you remember from the beginning, co-work writes these three different things.
[02:55] It's a chat, there's co-work, and then um code, which is like a version of doing coding here.
[03:02] Here co-work is like the this is kind of the application version of doing um Claude code on the command line.
[03:08] And what's nice here is so, what I'm going to do is I'm going to tell it to work in
[03:10] a particular folder.
[03:10] This is my website.
[03:13] So, what the way that I'm going to do this is I'm going to say and you can art I you know, there's not a perfect articulation of this, but what I'm going to say is I would like you to take the papers in the papers folder and the blog posts in my on my folder and the blog posts in my on my blog and create a um writing sta uh writing style guide for for Claude um as a skill so that I can um get Claude to write in my voice when I want to uh make it sound more like me.
[04:00] So, this is a pretty bad prompt and not exactly articulating it well, but Opus is going to know what I mean.
[04:06] I'm going to say, "Let's go."
[04:08] And so, it's going to get started.
[04:10] It's
[04:11] saying, "Okay, here's some instructions."
[04:15] And it's thinking.
[04:17] It's going to ask me a few questions, I think.
[04:19] It's going to It's sort of looking here.
[04:19] It's looking what's there.
[04:22] It's going to find the blog posts.
[04:25] Um it's going to update its to-do list.
[04:28] So, it's going to read this.
[04:30] And so, "Let me read through your blog posts, and I'll I'll start I'll start by reading a good sample of blog posts."
[04:35] So, it's it's looking in here.
[04:37] Of course, the downside here is I can't actually see what it's doing in this version, but it's running a sub agent to kind of read through the blog posts that I have in here.
[04:44] I have just a bunch of things that I've written in my voice.
[04:48] What you know, I actually have two style guides.
[04:50] Um I have one for like social media style things, which I use my blog posts, and then of course I have my papers.
[04:55] But, this is fine.
[04:56] So, it's now read a bunch of things.
[04:59] So, it found here's some blog posts.
[05:01] This is my Bardic paper.
[05:02] Here's a financial event studies.
[05:05] You know, various papers here.
[05:07] These are all the different blog posts.
[05:12] What I you know, there's there are ways you can hone this and then obviously have it do more.
[05:17] So, um you know, one aspect of this that is sort of important is that I may want it to really iterate over a number of papers.
[05:24] This is only a few papers that it's looking at so far.
[05:29] Um part of the reason for that is it doesn't want to blow through its context window.
[05:31] And so, there are ways of saying, "All right.
[05:35] Write it It's written this down.
[05:38] Um It's going to basically now it's going to you know, update this.
[05:43] Now, let me create the skill.
[05:45] I'll set up the directory structure and write the skill.md.
[05:46] So, it's done.
[05:48] It looked at this.
[05:50] And I want you to just kind of be clear that like so, Claude actually knows a lot about itself.
[05:53] So, one of the things to kind of keep in mind is if you're interested in kind of writing skills and other things to repeatable tasks that you're going to do, if you ask Claude on how to do things, say, "I want to like be able to do X, Y, or Z many, many times."
[06:09] It will know more or
[06:12] less how to do it often.
[06:13] You can even ask it without doing it, and it will sort of say, "What you need to do is say X, Y, or Z, and this will how it be how it gets set up."
[06:21] So, it's a the skill.
[06:23] So, it creates a skill.
[06:25] So, where does this is it saves it in this folder.
[06:27] So, there's a folder in that repository that's called .claude skills paulvoice skill.md.
[06:32] So, it makes a file there.
[06:34] It's just a markdown file.
[06:36] So, a markdown file means a text file.
[06:38] A text file that looks like this.
[06:41] So, it just uses hashes to do sections.
[06:43] It makes lists.
[06:44] It's very It's very much like text.
[06:46] And this is a skill.
[06:48] It says, "Write in Paul Goldsmith Pinkham voice and style.
[06:50] Use this guide whenever it asks to do write in my voice like I could write it.
[06:52] Um you know, blah, blah, blah.
[06:55] When it says to write like the way I would use this skill.
[06:58] So, what's going to happen if you use this skill is it this just gets dumped into the context.
[07:01] So, I I want you to sort of realize that this is kind of just like having someone try to synthesize how you write.
[07:11] It sounds very positive.
[07:11] You're
[07:12] intellectually honest.
[07:15] Yeah, I mean you I know.
[07:16] I think one thing about this is that this is relatively sycophantic.
[07:22] Um don't you know, every person, even the world's worst researcher would get a relatively positive take.
[07:29] It's just like having um you know how they say that feedback from a a supervisor should always have two good things around the bad thing.
[07:37] There's no bad thing that gets stuck in there with this one.
[07:39] So, you have to push Claude pretty hard to say negative things.
[07:40] But, is is it wise to push Claude on this to be also negative?
[07:48] You know, I think you I don't know.
[07:48] I mean it sort of depends.
[07:50] Like if you have writing ticks or things that you think are important reflections.
[07:53] So, there's a really great I don't have it here, but there's a great comment by I think it's Nabokov um writing to the New Yorker about editing.
[08:04] And there's a a great comment that he has.
[08:07] It's something like, "I'm happy to be edited, but I have I'm paraphrasing,
[08:12] but basically along the lines of
[08:14] I have very long run-on sentences that I don't want to be edited.
[08:19] This is obviously, you know, you could think of that as a bad writing style, but this is reflects kind of what I want for my writing.
[08:25] If that makes sense.
[08:25] Yes.
[08:28] And and I think we would all agree like sometimes kind of quirks in writing are reflective of kind of what we want our voice to sound like.
[08:37] And so, that's sort of an interesting thing of like I don't know quite how it would do this.
[08:39] I did What I kind of want to make clear here is that you know, this is basically just a bunch of text saying, "Here's how I would Here's how Paul writes."
[08:49] So, you know, So, what's about Is it the only in English?
[08:51] Can you also do the same thing in German or French?
[08:54] Yes, yes, yeah, 100%.
[08:54] Yeah, there's I mean Claude is quite multilingual.
[08:56] So, I do a lot of I often will write in French.
[08:58] You know, I speak French, and I would say the writing in French is quite good.
[09:04] Um I don't So, you know, I don't know if that's true for every language.
[09:08] There's certainly I don't You know, we'd have to ask my Chinese colleagues exactly how
[09:13] good it is, but certainly it would do reasonably well there.
[09:17] Um So, you know, what it's doing though is that this is just a list.
[09:19] You know, uh tone calibration based on context.
[09:24] In blog posts, it's conversational.
[09:26] Blah, blah, blah.
[09:26] In academic papers, first-person plural.
[09:28] So, I always you know, I tend to write uh with we even if I'm solo authoring.
[09:36] I don't use the passive voice.
[09:37] Um There's sentence level structure.
[09:39] Um distinct verbal habits.
[09:44] Um You know, so it's really giving examples like of how the writing is.
[09:48] Okay.
[09:51] Um handling technical context.
[09:51] First, I would do this.
[09:52] Here's what not to do.
[09:56] Um I would say that this is not true.
[09:57] I think I can be tend to be, you know, on my social media I I do sometimes get breathlessly enthusiastic.
[10:04] So, um I try not to hedge.
[10:09] So, this is kind of I would say these are true.
[10:10] Um
[10:13] but you know, so I I guess I just want to kind of clarify is like I would say this is useful.
[10:19] So, one thing that we could do that I think would be a useful exercise is let's do the following task which we can say the following.
[10:26] So, let's say, "Please write um markdown um Oh, this isn't working right now.
[10:34] Please try Let's see if this works.
[10:36] Please write a write markdown document explaining um difference in differences to a PSC level audience.
[10:46] So, let's hopefully it will not use this skill.
[10:50] So, we're going to do this and then we're going to say um we'll ask it to do the same um using your voice.
[10:57] My voice, exactly.
[11:01] Um So, it's doing this. It's kind of thinking about
[11:05] So, but could you do this in chat as well or it has to be in Claude work?
[11:08] No, no, absolutely you could use in chat.
[11:10] In fact, the only reason I did it in Claude work is so that it would be easy to access the skill
[11:15] um but I'm pretty confident that in chat, so what I would need to do to do that there's a couple of things that I could do.
[11:20] So, the easiest thing to do is I would add the skill here.
[11:22] I would need to go and find it.
[11:25] I think I would have to upload it or do something else.
[11:26] Like so, um I don't often do this.
[11:28] But, you know, the easiest thing to do of course is with the skill since I told you it was just text is I can just copy and paste that skill that we that we did there, right?
[11:39] So, we have this one here.
[11:44] So, I can take that skill and I could just copy this text.
[11:46] I could say, "Okay, you know, take this text, copy it um and go to the chat and say explain it, you know, using this."
[11:54] So, let's see what it said for difference in difference.
[11:55] It's currently working.
[11:57] Still working.
[11:58] It's thinking very Maybe this
[12:01] So, can you show me how you put it into chat?
[12:03] So, then you put in the preference
[12:04] Yeah, yeah, so here, right?
[12:06] So, let me run two chats.
[12:08] So, first let's do one where I just say, "Please explain the difference in difference um
[12:15] methodology to a PSC level audience.
[12:20] Um it seemed like Claude work it was having some trouble.
[12:23] So, maybe this Let's do this.
[12:25] Um this seems to be broken.
[12:27] I'm not sure what's going Oh, here we go.
[12:29] Um
[12:30] So, here it's this and so then we could do it.
[12:32] So, let's let this one run and then in here I can say, "Please explain the difference in difference methodology to a PSC level audience.
[12:44] Use this writing skill.
[12:49] And so, I just pasted it.
[12:51] So, this is just a paste thing of exactly this.
[12:55] Let me fix all these typos even though Claude doesn't care about that.
[12:59] So, I did this and so now let's see what it says.
[13:03] And so, let's compare this.
[13:05] So, this is what my voice.
[13:05] Let me lay out the how difference in difference works where in different blah blah blah.
[13:10] Here's the practical problem.
[13:13] Parallel trends assumptions.
[13:17] So, I would say reading through this of course, you know, sounds like you.
[13:22] This is kind of a it sounds like me. I would say it's a little bit of a caricature of how I write which I think often is how people feel about some of these things is that they see people will make a skill that looks like it.
[13:34] That being said, this is pretty accurate.
[13:38] Whereas if we compare it to this one, um, this is quite similar but this sounds more kind of like the default AI voice.
[13:46] So, this is kind of directly to this. There's less of a setup. It kind of gets to the method part. Here's what the assumption is.
[13:54] I I think this is also quite good. Like what's amazing about Opus is that it can kind of write up these things. But, it this does sound closer to how I would do it.
[14:01] I would start with the practical problem and kind of set it up. Do the two by two example before I get into any kind of regression analysis.
[14:09] You know, how I know that it kind of does it exactly right, that's kind of a big if.
[14:15] Um, so, I think what I want to just emphasize is like when people talk about
[14:18] there being skills and other things that are done, you know, it's really important to just know that like it's just a laundry list often of doing things like this.
[14:28] Um That being said, it can get more kind of There can be more.
[14:31] So, I'll just kind of um I'll I can post an example of what my my writing style thing that I've done is actually made it go through many more papers and do much more work there.
[14:45] Um here's an example.
[14:45] you also say, "Oh, I would like to have the writing style of Asimov or something?"
[14:49] Yeah, yeah, exactly.
[14:51] No, you can.
[14:53] I think um Claude might complain a little bit about some of the um You can ask it to do in the style but if you ask it to read through Asimov's work and then summarize it, it might complain about it it would be a little bit It might complain about some copyright stuff but you can absolutely say like um you know, you know, uh now please summarize DID in the style of, you know, let's let's pick your favorite um Dostoevsky.
[15:18] Yeah, let's say Asimov.
[15:20] mean, Isaac Asimov I I don't really know how Isaac Asimov would inherently um.
[15:25] Yeah, Dostoevsky Dostoevsky. I can't even spell Dos- Dos- Dostoevsky.
[15:29] Yes, it won't know It will know who I mean.
[15:34] My wife is going to kill me.
[15:37] Right. So, this is kind of a caricature of how Dostoevsky would write.
[15:42] I'm mad.
[15:43] This is like This is, you know, this is kind of Crime and Punishment but added to different things.
[15:49] So, this is kind of This is a little bit of like a It's a caricature to some extent.
[15:53] Overdone. Yeah. Yeah. So, So, that's kind of I just that you can do more.
[15:56] You can do more things here.
[15:58] In fact, let me just quickly in here Marcus while we're here.
[16:02] Let me just quickly show you an example of what mine looks like in a kind of a broader version.
[16:07] Just so you can get a sense of what does it look like with a broader one.
[16:15] You know, here I have a style guide here that is based
[16:20] off of a lot more work published in this.
[16:22] size?
[16:22] Sorry.
[16:22] Yes, yes.
[16:26] So, here I have kind of I'm a Let me sign one second.
[16:29] Let me close this and this.
[16:32] And then this over.
[16:34] So, I have kind of more writing style patterns, sentence structure, kind of similar in some ways but it's much more specific.
[16:41] Use subordinate clauses.
[16:43] I do a lot of m dashes.
[16:46] Um I have What do I do?
[16:48] I kind of define thing This is much more specific to my This is kind of a much more detail of the structure of what mine looks like in part because I made it focus on a lot more papers and iterate over.
[16:59] Kind of it has
[17:00] But, this was also Claude generated, no?
[17:02] Claude generated I used Claude code to do this but yeah.
[17:04] So, I had it So, like the only thing I want to emphasize is like you know, there's more things in there like decomposition, policy and economic significance, make sure to translate to dollar terms.
[17:13] Okay.
[17:15] Pivot, you know, position.
[17:17] So, that you can add a lot This is just a much more of a laundry list than what we just looked at, right?
[17:20] I avoid AI generated
[17:22] writing pattern.
[17:24] I'm not allowed to use these words.
[17:25] Um you know, hollow I try really not to have hollow intensifiers.
[17:30] Um do you make this the thing available on your Substack?
[17:36] I will Yeah, why don't I link it so people can see it.
[17:37] So, I have, you know, kind of my opening style.
[17:39] Here's some examples from my papers of kind of how I might do it in in these papers.
[17:47] Um so, you know, that this is the same thing.
[17:48] It's just more detailed and it's not obvious it's perfect but it is useful um for me to have here and I have different
[17:54] This is for my for writing papers whereas for my social media guide it's kind of you know, it's it's a little bit more painfully sycophantic.
[18:04] Um so, you know, there's just like a whole it's very painful.
[18:16] Let me kind of um pivot here a little bit cuz I want to show practically speaking I think what it could be very helpful for folks is to see um
[18:25] Um, how to use, how you might.
[18:27] So, there's two things that I want to kind of simultaneously point to.
[18:30] One is going to be very straightforward.
[18:32] This could be done in Claude work or in um in Claude code.
[18:37] Would be to just go in Claude work, make a new task and I have a folder where I keep all my referee reports.
[18:47] So, um, you know, it's in my Dropbox folder.
[18:49] Be essentially for anything data related to use Claude code but anything which is more writing you use Claude Claude code work.
[18:56] Yeah, I mean, I don't use Claude work.
[18:58] I'm kind of using this more I think I don't want people to feel like everything has to always be done at the command line to take advantage of these tools.
[19:04] So, I want you to be a kind of aware that you could easily use this.
[19:08] I would I'm very comfortable in the command line and I find it to be to work just as well.
[19:14] Um, and often, you know, I'm happy to have it be able to reference many different things.
[19:19] But, you could I think for writing stuff certainly as a first page Claude work works great.
[19:24] And so, I'm that's part of why we're doing it here.
[19:25] and it's obviously it it's nicely formatted and so forth.
[19:28] You know, what I wanted to kind of emphasize here is let's not put this one in here is you can look at my ref report referee reports.
[19:36] and what I basically you can say is, "Please read through all the referee reports um in this folder that I wrote.
[19:42] I want you to construct a skill to generate reports that look like these focus on the types of issues and concerns I flagged.
[19:58] And so this is another way of writing kind of a similar skill or or set of things to focus on.
[20:03] Um So this is very similar to what we just did.
[20:07] And what it will generate it.
[20:10] So I am not encouraging you to inherently use Claude to write referee reports.
[20:15] Mhm. Um what I am encouraging one reason that I use this is I often write a referee report on myself for my own papers.
[20:22] Mhm. Or if I have a paper I want to look at to give feedback for
[20:26] other people that's not when I'm being a
[20:28] formal referee. I will have it summarize
[20:31] a paper to make sure like kind of I
[20:33] often am not able to spend nearly as
[20:35] much time on a referee report as I would
[20:37] uh and on somebody emailing me versus on
[20:39] a referee report. So I will would look
[20:41] at that. Um How does it compare to
[20:44] refine.ink and collapse?
[20:46] >> Collapse.
[20:46] So refine.ink has much less Yeah, yeah.
[20:49] So refine.ink So I use refine.ink often
[20:52] when I'm getting ready to submit a
[20:53] paper. Refine.ink is often doing much
[20:56] more to make sure to look that things
[20:58] are being um
[21:00] inherently consistent. I would say that
[21:02] like refine.ink I think I would have to
[21:05] ask Ben if this is true is doing less to
[21:07] identify whether or not it is being um
[21:12] It doesn't have a It doesn't try to
[21:14] establish as much taste or preferences
[21:16] about what a paper looks like. It is
[21:17] more trying to make sure that the paper
[21:19] is mutually consistent and making claims
[21:21] that are consistent with the what's
[21:22] going on and in is in is true.
[21:25] Um
[21:27] You know, this referee much like referee
[21:28] reports often when we write referee
[21:30] reports, right? It's some combination of
[21:32] identifying truth versus
[21:34] I additional things that need to be done
[21:36] or methodological approaches that need
[21:38] to be done correctly, right? Or like
[21:41] Yeah. So some of the things will be
[21:42] similar, some will not. I would say that
[21:44] refine.ink is far better than this. This
[21:46] is, you know, incredibly cheap to run
[21:48] though is one reason to do it and
[21:50] you it also you have certain things that
[21:53] you care about more than others and so
[21:54] that can be a helpful thing to I kind of
[21:56] just want to highlight that this is a
[21:57] thing that can be can just be helpful.
[21:59] So why don't we let this run in the
[22:01] background so I can show in a second.
[22:02] The flip version that I want to really
[22:04] make sure that people see that I think
[22:06] is really kind of a great
[22:08] um really great um
[22:11] skill is
[22:13] there is a skill from
[22:15] um
[22:16] I want to find This is a little bit
[22:19] tough because I want to make sure that
[22:20] we do do the I want to do this without
[22:22] having to necessarily go through a
[22:23] revision of my own paper. Um This is
[22:26] going all on the internet. So here's
[22:28] what we're going to do
[22:29] is I
[22:31] am going to use
[22:35] um
[22:36] I'm going to do the following. So what
[22:38] we're going to do is we are going to
[22:41] look at a paper that I have posted. Um
[22:44] and I'm going to ask it to write a
[22:45] referee report.
[22:47] Um
[22:49] And then we're going to do this task. So
[22:51] um
[22:52] let me just quickly show what that thing
[22:54] looks like. So there is a skill um
[22:57] called strategic revision.
[23:01] So this is the So the the faculty member
[23:03] who who made this is is Jukka
[23:05] Savolainen.
[23:06] Um
[23:07] He has a number of skills that he's
[23:09] done. The one that I really like is I
[23:11] think he's developed um
[23:14] Let me share
[23:15] this here is that he has I'll just share
[23:18] his LinkedIn page.
[23:20] So he has developed this skill that I'll
[23:21] show you in just a second which is
[23:22] called strategic revision. And what I
[23:24] like about it
[23:26] is um
[23:29] is that he
[23:31] what it does is it takes a paper that
[23:34] you have submitted
[23:36] and does the
[23:39] You have done
[23:40] You submitted a paper. You have gotten a
[23:42] set of referee reports back and a
[23:44] an editor's letter. Mhm. Given that, how
[23:47] should you create a revision plan?
[23:50] So what this does is it says give me the
[23:52] manuscript and give me the reviews and
[23:56] give me a plan back on how to do this.
[23:58] And so this you can do inside Claude
[24:00] code and I'm going to show you kind of
[24:02] what that would look like um now. So
[24:05] All right. So here's what we're going to
[24:06] do first is we are going to
[24:11] write a referee report about a paper
[24:12] that I have just so that no one We could
[24:15] do this about Marcus's paper, but I
[24:16] don't know if I have
[24:18] strong benefits. Um
[24:21] Who knows what my value add is on
[24:22] Marcus's papers. Let's see. So I what
[24:25] I'm going to do is I'm going to put a um
[24:27] a
[24:29] paper that is already published so that
[24:31] we don't I don't have to feel
[24:33] stressed about it and we'll say
[24:37] um I'm going to put the the published
[24:39] version um on this paper. So I'm going
[24:41] to ask it to write a referee report on
[24:43] this paper. So I've given it the skill.
[24:44] You would actually typically would be a
[24:46] command, but it should know how to do
[24:48] this.
[24:49] Um
[24:50] It's going to read this. It's going to
[24:52] read this paper.
[24:54] It's going to try and write a referee
[24:55] report and then we're going to save take
[24:57] that referee report and the draft and
[25:00] we're going to use strategic revision to
[25:01] do that.
[25:05] It's trying to read this. It's
[25:07] struggling cuz it's Claude and it
[25:08] doesn't want to
[25:10] It is it has trouble converting PDFs.
[25:14] It's doing this. It's writing an editor
[25:16] letter and the detailed referee report.
[25:19] And then we'll do after this, what
[25:21] journal is this for? This is for the
[25:24] RFS, let's say.
[25:27] So my skill that I have is Mhm. actually
[25:30] writes an editor letter as well.
[25:33] And
[25:35] That's what the editors will do in the
[25:37] future.
[25:38] Like I I don't know. I've talked to some
[25:40] editors about this. You know, I think
[25:43] I think it's um I think probably the
[25:45] biggest value on some of this and this
[25:47] comes back to this idea of writing and
[25:49] so forth is less of an idea of doing it
[25:52] for you, but giving you a very good
[25:54] first path.
[25:57] Okay, great. So it took a little while
[25:58] cuz we're having Claude was having some
[26:00] issues.
[26:01] Marcus was saying earlier this is like a
[26:02] systemic risk. This is a challenge of
[26:04] using these um things if Anthropic goes
[26:06] down and it wrote a report and we're,
[26:08] you know, I I clearly didn't like my own
[26:11] my own work.
[26:13] Here.
[26:14] Did it check Well, it's a good paper on
[26:15] an important topic. Um
[26:18] So it's here. So you as I I you know, we
[26:20] don't we can read through it, but
[26:23] uh
[26:24] I just want to kind of briefly highlight
[26:25] what it looks like. This is a paper
[26:27] about climate change. You know, this is
[26:28] It looks like a referee report. Mhm. Um
[26:33] So what I want to do now is I want to
[26:36] take this. I'm going to download it.
[26:38] Actually it it already saved it in a
[26:39] folder here.
[26:41] Uh it already saved it in a folder. So
[26:43] what I'm going to do now is remember in
[26:45] chat I'm going to say I'm going to use
[26:46] this strategic revision skill.
[26:49] I'm going to say
[26:51] I'm going to
[26:51] >> you move to chat, you can still do it in
[26:52] co-work.
[26:53] >> could do it in co-work. Good question.
[26:54] So I haven't done Let's see. So
[26:56] strategic Oh yes, I could just do it in
[26:58] co-work. So here um I'm going to say I'm
[27:01] going to add the two files. So I'm going
[27:02] to say
[27:03] uh it needs to be
[27:06] So I'm adding the report
[27:09] and I am going to add the the paper. So
[27:12] the strategic revision you had to upload
[27:14] or connect to us. Yes, exactly. So I had
[27:17] I downloaded and connected it. Um and
[27:19] that's a straightforward task. So let me
[27:21] show you how you would do that actually
[27:22] before we get into it. Good good
[27:23] question. And is it if it's connected
[27:25] for co-work, it's connected for chat or
[27:27] is it separate?
[27:27] >> Yes, yes. So the the app here as So if
[27:30] you add a skill Okay.
[27:32] >> the way that I would do it is um
[27:34] you know actually I think I had to I
[27:37] wasn't even sure how to do it when I did
[27:39] it and so what I said was
[27:41] said man, so there's a skill here. Yes.
[27:43] So what I did you go to manage skills.
[27:45] So we're in chat. You go to manage
[27:47] skills.
[27:49] And then
[27:50] once you're there, you add a skill. So
[27:52] here are there's these different ones.
[27:54] Um and then I
[27:56] uploaded a skill. So this was a file
[27:58] that I brought
[28:00] that is on that website that I showed
[28:01] you the strategic revision. Um Okay.
[28:05] And this is kind of the command and it
[28:06] has all these things in here of what to
[28:08] do.
[28:09] And so if we go back um to co-work
[28:12] we could say
[28:14] what I'll just give it the paper. Often
[28:16] of course you would have an editor
[28:17] letter too and other things, but we're
[28:19] just trying to It's show proof of
[28:20] concept.
[28:21] Um and I'm going to show it give it the
[28:23] draft and the report and we say let's
[28:24] go.
[28:28] And so I'll tell it to do a strategic
[28:30] revision. Strategic revision, all it
[28:32] does, remember there's a skill and
[28:35] it just has a bunch of text from that
[28:36] skill that it's dumping into the the
[28:39] context.
[28:42] So
[28:44] it's now running the skill.
[28:47] So this is what it does. So just so that
[28:49] you can see what it says. It says create
[28:51] a rigorous dependency map revision
[28:53] master plan from peer review reports
[28:55] using a computationally DAG validate
[28:57] using with computational DAG validation
[28:59] using NetworkX. So what that means what
[29:02] I've liked about this is that it creates
[29:04] a set of tasks that you need to do based
[29:06] off the referee report knowing kind of
[29:09] what needs to be done first. Like what
[29:10] depends on what.
[29:12] And I find this to be very helpful
[29:13] because even though you read the referee
[29:15] report, sometimes it can be a little
[29:16] overwhelming especially if you have four
[29:18] reports and you need to decide what to
[29:19] do or multiple reports. And so it
[29:22] organizes into a master document. What
[29:24] what does DAG stand for? Sorry. Oh, DAG
[29:26] stands for directed acyclic graph. Okay.
[29:29] >> So um
[29:31] it's a way that people use in computer
[29:33] science to um it's it's used for a lot
[29:37] of things. It gets used also in causal
[29:38] inference um but it's a way of saying
[29:40] that you know, X affects Y or like X is
[29:44] Y is dependent on X like X maps to Y.
[29:46] And so because you make it as a graph
[29:48] that's directed, um
[29:50] you can use computer science to sort of
[29:54] computer science. You can use network
[29:55] theory to know if there are cycles,
[29:58] right? So, cycles would mean that you've
[30:00] got the wrong dependent structure.
[30:02] You know, here it says, "Adopt the
[30:03] perspective of a distinguished professor
[30:05] and a former associate editor with 25
[30:06] years of editorial experiences." You
[30:08] know, someone like you, Marcus.
[30:10] Uh be methodologically obsessed with
[30:12] efficiency, logical consistency, and
[30:13] computational rigor. That's done like
[30:15] that.
[30:16] Yeah, exactly. So, what it does is it
[30:18] sort of constructs things. So, it's
[30:19] giving a a workflow plan on how you
[30:21] should do it. So, it now is doing these
[30:24] things. It's running something in the
[30:25] back. It actually has to use Python to
[30:27] do this. Mhm. Um because it's going to
[30:29] run a validator. Um
[30:32] I think actually the package itself,
[30:34] like it's supposed to have this thing in
[30:35] here, but it doesn't. So, then it writes
[30:37] it itself. Um but it's going to do many
[30:39] things. So, it takes a little while. Um
[30:43] and then at the end, it should give us
[30:44] it will kind of give an end goal, uh an
[30:47] end master plan.
[30:49] And so, it will just give a file. So,
[30:50] this is what it's going to do, and I
[30:53] found this really helpful. I think this
[30:54] is a really cool sort of structure here.
[30:57] So, it's going to try and validate So,
[30:58] what it's done is it's written a bunch
[31:00] of things. Um it is now trying to
[31:02] validate the DAG structure.
[31:05] Uh it says that it's worked. Um assign
[31:07] execution blocks and run the full phase
[31:08] six analysis. Let me update the JSON.
[31:14] So, it's going to be a bunch of text in
[31:15] here.
[31:17] And really what it's doing is it's
[31:18] reading the report and then it's reading
[31:20] the paper, meaning like, "Okay, what do
[31:21] you need to do?"
[31:23] Does it do the revision, too?
[31:26] Well, you could then take this if you
[31:27] want to use an LLM to help you with this
[31:29] is what it's done. You know, remember we
[31:31] talked about being concrete and specific
[31:32] is really helpful for LLMs. Having a
[31:35] really specified set of tasks is the
[31:37] same way of working with an RA. That
[31:39] certainly makes it a lot easier to
[31:41] approach the problem. Now, what I'll
[31:42] tell you is that you still have to have
[31:44] a lot of discretion in how you
[31:46] do these things. Um and so, that's going
[31:48] to be important.
[31:49] So, now actually we're almost done. So,
[31:52] um it's fitting it's finished writing
[31:54] this whole plan. It's going to be quite
[31:56] long. It's doing a little bit of
[31:58] analysis, and here is the final plan.
[32:00] So, 26 tasks extracted from the referee
[32:03] report, organized into five blocks.
[32:05] Um the critical paths the critical path
[32:08] is this.
[32:09] Um the key bottlenecks are here. And you
[32:13] know, many of these can be parallelized.
[32:15] So, let's just look at what this looks
[32:16] like just so you can have a sense.
[32:19] Um
[32:21] So, oh.
[32:21] So, what it's going to do, is this okay
[32:23] size-wise? Yes.
[32:25] So, it's going to it has a revision
[32:27] master plan. This is a single referee
[32:28] report. It was generated here. Here are
[32:30] the tasks. What it does is it takes
[32:32] every task and makes it into a single
[32:36] um single thing that you need to do. So,
[32:38] this is every comment that my referee
[32:40] report did. Blah blah blah. Blah blah
[32:42] blah. So, many many comments, very
[32:44] annoying. Minor, non-minor.
[32:47] Then it classifies them. Some of them
[32:49] are argumentative.
[32:50] >> Mhm. Just, you know, writing text. Some
[32:52] are empirical robustness things I need
[32:54] to do.
[32:55] Um
[32:57] Most of the and then one two
[32:58] clarification points.
[33:00] Um and then an editorial decision.
[33:03] So, it's heavily empirical.
[33:07] Um and here are the blockers. So, this
[33:09] is trying to figure out what needs
[33:10] what's relying on what.
[33:12] And sort of collateral risks in terms of
[33:14] what is affecting what. So, let me just
[33:15] kind of skip to the what it's going to
[33:17] end up doing is it's going going to
[33:19] create um
[33:21] blocks of It actually one thing that it
[33:24] will do, by the way, is that because
[33:26] if you have multiple referees, Marcus,
[33:28] presumably you've had this, is there
[33:29] will be referees who disagree about what
[33:31] you should do. So, it will actually
[33:33] identify when those are present, and it
[33:36] will say, "Look, if you do X, then
[33:38] person Y might be annoyed."
[33:40] Um and so, what this will do is let me
[33:44] see if
[33:45] >> I can. It's an enormous document
[33:46] already, no? Yeah, I mean, it has it has
[33:48] a lot of stuff in here. Now, whether or
[33:50] not it all needs to be this exhaustive
[33:52] is unclear. Um
[33:54] It even proposes how the co-authors
[33:56] should divide the tasks.
[33:58] Um Oh, I see.
[34:00] Which is very funny.
[34:02] Um and so, then it gets So, it kind of
[34:04] gives the tasks and says how you should
[34:05] be doing them in what order. It's a
[34:07] little hard to see here, but let me show
[34:08] you the picture Um very quickly. I like
[34:10] this picture that it does, which is this
[34:12] one. See if we can So, it's going to
[34:15] look like this. So, just so you can see,
[34:17] you know, here's the block of these
[34:19] tasks, block A, block B,
[34:22] block C.
[34:23] Oh, I guess it's a little the formatting
[34:26] is not perfect, but so, then you have
[34:27] these, and then this is batch two. The
[34:29] next thing needs to be done in here is
[34:30] batch three, batch four, batch five.
[34:33] So, you know, this you basically can
[34:35] copy these over and say, "All right, now
[34:36] we need to do these tasks." and sort of
[34:38] structure them into issues and things
[34:39] you want to do. So, I really encourage
[34:41] This obviously is not the end-all
[34:42] be-all. There's ways you could improve
[34:44] on this scale. I just happen to really
[34:45] like this one that this person created.
[34:48] Um What is the strategic revision that's
[34:51] your favorite one, but there are
[34:52] competitors?
[34:54] You know, I don't know if there are
[34:55] other people. What I like about this one
[34:56] is that I sort of I have not like worked
[34:59] super I think I've done two papers where
[35:00] I've used this as a structure to do
[35:02] stuff with. I have not looked that much
[35:05] into different versions. I think what I
[35:06] like most about this one is it is
[35:09] concrete in trying to make the tasks
[35:11] very
[35:12] um
[35:13] actionable, categorize them, and also
[35:16] see what's dependent on what, which
[35:18] makes it really much easier to then work
[35:20] with LLMs on subsequent problems as
[35:22] well. But would you say it's
[35:24] as good as for empirical work than it is
[35:26] for theoretical work?
[35:28] Yes. So, I've used it for both,
[35:30] actually. And I thought it was good for
[35:31] both. In fact, I would say it was
[35:32] actually in some ways better. Like,
[35:34] empirical work is so um
[35:37] in some ways empirical work is so much
[35:39] taste involved in it that you kind of
[35:41] have to revisit. Whereas with
[35:43] theoretical, it's kind of you know which
[35:44] proofs have to are relying on what.
[35:51] So, let me in the last um couple minutes
[35:53] just talk about
[35:55] you know, we went through a few things
[35:56] on here, and I want to kind of just
[35:58] We've talked about a style guide. We've
[35:59] talked about referee report skills. Um I
[36:01] just want to kind of quickly talk about
[36:03] ways in using LLMs
[36:06] for editing.
[36:08] Um
[36:09] Here, I think the one thing that I want
[36:11] to kind of flag, there's a number of
[36:12] ways you know,
[36:14] we
[36:15] it's very easy if you have a few vague
[36:17] ideas to hand them off to an LLM and try
[36:19] and let the model do intellectual work
[36:21] for you. I guess what I want to
[36:22] encourage is kind of similar to how we
[36:24] talked about the context window is that
[36:26] you want to do kind of
[36:28] intentional
[36:30] compaction and note-taking to kind of
[36:32] iterate on ideas. And so,
[36:35] you know,
[36:36] better for you to kind of really iterate
[36:38] out the ideas
[36:41] as much as you can before you talk to
[36:43] the LLM because the specificity is going
[36:45] to be enormously valuable. The less
[36:47] specificity you have, you know, if I say
[36:49] something like, "Hey, I'd really like to
[36:50] think about a life cycle model that
[36:51] thinks about savings."
[36:53] That's you're not going to get
[36:55] particularly groundbreaking ideas there.
[36:58] Um
[36:59] So, the more specific you can be, the
[37:00] better. One reason that it's very
[37:02] valuable to use something like Claude
[37:04] Code or Co-work is to add more more
[37:07] files and more context into the LLM when
[37:11] it's doing it. You can say, "Look, I'm
[37:12] in this folder. Here's a bunch of papers
[37:15] that are related to the thing I'm
[37:16] interested in. Here's some notes that we
[37:17] did. Here's some slide decks. Here's
[37:19] some websites that are relevant to this
[37:21] policy. Here's what we're thinking
[37:23] about." Um you know, work from there.
[37:27] I That's like kind of one key thing of
[37:28] just like that's kind of a useful thing
[37:30] when you're iterating. The more
[37:31] information you can give, the more
[37:33] tools.
[37:34] Um
[37:35] I think one really kind of
[37:37] key thing that a lot of people are
[37:39] really uncomfortable on kind of thinking
[37:40] about the LLM doing the writing itself.
[37:42] So, one useful thing that I have found
[37:45] is to actually have the LLM behave more
[37:48] as an editor in the sense in the
[37:50] following sense. So, one thing that you
[37:51] can ask is to say,
[37:54] "I don't want you to edit my writing."
[37:57] So, a good example of this would be I
[37:59] would open up a paper,
[38:02] Mhm.
[38:02] >> and I would say, "Please read through
[38:04] this and give me comments on
[38:08] um
[38:09] ways in which I can improve or tighten
[38:11] my writing on this." So, I'll give you
[38:12] I'm going to sort of show you what that
[38:13] would look like. Um I'll show you in
[38:16] >> And and you do this by talking to the
[38:18] LLM or you type?
[38:20] It could be either way, but I mean, so
[38:21] what I you know, I'm going to type, but
[38:23] you could also do this with yours. So,
[38:25] one thing I could do, so, I have these
[38:27] blog posts, and so I could say, um
[38:30] "Please take a look at Let's pick a
[38:33] We're going to pick a
[38:35] blog post." Just So, we're doing blog
[38:36] posts just to kind of make it smaller so
[38:38] that it's not complicated, but this
[38:39] would have worked just as well with um
[38:42] with a research paper.
[38:44] Um
[38:46] And so, I have this
[38:48] um
[38:49] I have this blog post that I did um
[38:53] about um
[38:56] uh a friend of mine's recent monograph
[38:59] they put out. And I want to say,
[39:02] um at
[39:04] I'm going to say Oh, not this one. I'm
[39:06] going to say, "Please take a look at
[39:07] this blog post." Let's see. One second.
[39:09] I'm going to name it. So, add this.
[39:12] So, it's called
[39:14] Right. I read it. So, it's not Claude
[39:16] Code. So, I just didn't say this. In my
[39:18] blog post,
[39:20] um
[39:21] "I would like to get edits in the style
[39:26] of a New York Times
[39:29] editor for writing
[39:32] and clarity.
[39:34] However, do not
[39:36] directly edit my writing. Instead,
[39:40] please um
[39:42] create inline
[39:45] comments around places where my argument
[39:49] is poor or unclear and where I can
[39:53] improve.
[39:56] writing.
[39:57] So, let's see if this we can do this.
[39:59] So, I just kind of want to highlight
[40:00] this cuz one thing that can be useful is
[40:02] to say um
[40:05] you know, if you do this is that then
[40:09] what it will do is it will just write
[40:11] thing. It won't change the things that
[40:12] you're inherently doing, but it will um
[40:16] it will potentially give comments, you
[40:19] know, which is which is helpful. So,
[40:20] it's going to read this. It's reading
[40:21] through it. I'm going to read through
[40:22] it.
[40:22] >> this in co-work, you could also do it in
[40:24] chat. Equally, you know. No, you
[40:26] couldn't do it in chat because chat
[40:28] would not directly edit the files in
[40:29] your computer. This is a co-work system
[40:31] task. So, I would use Claude code the
[40:32] command line, which would also work, but
[40:34] it needs to be a co-work thing
[40:36] specifically. Um you could, of course,
[40:38] upload the document to chat, tell it to
[40:41] do it, and then do that. And that would
[40:42] work. So, that's true that you could do
[40:44] that. I would say that it um it would
[40:47] work much better in co-work because it
[40:49] will use um
[40:52] kind of skills local on the computer to
[40:54] do it.
[40:55] >> Mhm. Um I think it's a better as a
[40:57] co-work exercise cuz what it's doing in
[40:59] chat is it's
[41:00] creating kind of a uh computing
[41:02] environment up there to do the same
[41:04] task. And the more distance that you're
[41:05] kind of creating between one versus the
[41:07] other is going to create problems.
[41:10] Okay, great. So, now it's done. It took
[41:11] a little while. So,
[41:13] this is the main context that it did
[41:15] structure and framing, throat clearing,
[41:16] you know, these are editorial comments
[41:19] more than anything. What I can do is
[41:21] just to kind of I will sort of show you
[41:23] an example of this of what it does in
[41:26] the writing. Um let me quickly do that
[41:29] is
[41:30] um share this, and I'll just show you
[41:32] what that looks like. Um
[41:35] is here.
[41:37] So, this is what the original post
[41:39] looked like. And now it's written
[41:40] another file in here, which is the
[41:42] edited one. And it will will do is it
[41:44] will say
[41:45] you know, have comments here, which are
[41:47] these are these are HTML These are
[41:49] actually markdown HTML comments, and so
[41:52] they wouldn't show up. They didn't
[41:53] haven't edited it.
[41:54] But this is like getting a little note,
[41:56] right? This is like what you might do
[41:57] for your graduate student. Um
[41:59] same thing here.
[42:01] You leave the section with this.
[42:03] Um the 10-shape is jargon only bond
[42:06] literature will know, etc.
[42:08] Simple and pretty neat is filled with
[42:10] throat clearing. I agree. Um
[42:14] is vague praise that doesn't advance the
[42:16] argument. So, lots of I think good
[42:17] editorial comments
[42:19] about exactly what's going on.
[42:22] Um
[42:23] I find this to be helpful because this
[42:24] is often if you're trying to preserve
[42:27] your own voice, the kind of editorial
[42:29] style you would want, right? Like if
[42:31] somebody kind of presents you and marks
[42:33] up and tells you exactly how to write
[42:34] things, your graduate student sometimes
[42:35] you're very happy with it, but it's
[42:37] harder to preserve your own voice in the
[42:39] writing.
[42:40] Um
[42:42] So, that's kind of one really I think
[42:45] it's often that
[42:46] >> you could also have said or at the end
[42:47] also propose an alternative. Yes,
[42:50] exactly. And then it would have it in
[42:51] there. And sometimes it will do that
[42:52] when I do this, but that's that's
[42:53] exactly that's exactly right. Um you
[42:57] know, of course, the danger with those
[42:58] is always that Yes. then it's the you
[43:00] know, it's the I totally agree. It's
[43:02] sometimes I will what I'll do is I'll
[43:03] copy that piece of the text and say
[43:06] all right, what should I do here? Like
[43:07] what is you know, what's a good idea?
[43:09] Depends you know, we don't always have
[43:10] infinite amounts of energy to think
[43:12] about our writing. Yes. Um so, that's
[43:14] one I think it's just one very simple
[43:16] thing, but often I think
[43:18] rather than thinking about this as a
[43:20] thing that can only write is what you
[43:22] might think of it as is actually as a
[43:24] relatively good engine for thinking, an
[43:27] analytics engine, right? It's it's
[43:28] trying to do um reasoning. And the
[43:31] reasoning is to help you with the
[43:33] structure of your argument. It doesn't
[43:34] have to rewrite your writing, but it
[43:35] could give you explanations for why it
[43:38] thinks this is poor reasoning or or poor
[43:40] writing.
[43:41] Um so, let me kind of wrap up then in
[43:43] here. I know we sort of is that we
[43:45] touched on a kind of a bunch of
[43:46] different things. We could talk about
[43:47] this for ages, I know.
[43:50] Um
[43:50] but I kind of want to just emphasize
[43:53] that
[43:54] kind of the overarching thing that I
[43:55] wanted to emphasize is just like A, this
[43:57] idea that many of these skills and
[43:58] things are just putting more text at the
[44:00] beginning, summarizing them. I think
[44:02] it's a useful thing to see that when you
[44:04] do a skill, like if we did a Marcus
[44:06] skill or a Paul skill, in the end it's a
[44:08] summary that is a caricature in some
[44:10] ways of what you are, but is
[44:13] useful.
[44:15] Um it's a sounding board that kind of
[44:16] reflects it's like a puppet. Um
[44:20] I think editing and writing I I have
[44:22] found this to be a useful way kind of in
[44:24] a subgame way to prevent myself from
[44:27] writing too much that sounds like an LLM
[44:29] in cases where I really want to kind of
[44:31] maintain my voice or use an LLM. But
[44:33] kind of the key graph figure I wanted to
[44:35] show or picture I wanted to show was was
[44:37] kind of this. This is from an IBM This
[44:39] is this famous on the internet this IBM
[44:41] guide from 1977, I think, about
[44:45] you know, this is this a worker training
[44:47] manual is, you know, a computer can
[44:48] never be held accountable. Therefore, a
[44:50] computer must never make a The original
[44:52] is management decision, but in your case
[44:54] is that when a computer, you know, when
[44:55] we use these
[44:57] in the end, remember this is our
[44:58] writing. We're using these tools, and so
[45:02] you really are responsible for what's
[45:04] going on there.
[45:05] Um
[45:06] So, I can The key takeaways here is I
[45:08] think like the writing itself is
[45:10] thinking. You should be deliberate about
[45:11] what you offload and you do yourself.
[45:13] The style guides are useful and
[45:15] interesting, but aren't going to be
[45:16] perfect. Um you should curate it, and
[45:18] you can iterate, and you can do more,
[45:20] but it's never going to be perfect. I
[45:22] think boilerplate, you know, first pass
[45:24] writing on on banal writing is
[45:26] um really helpful for me personally.
[45:29] And then in terms of referee reports, I
[45:31] do think the strategic revision thing is
[45:32] quite lovely, and there are other ones,
[45:34] too, that people should explore. Um and
[45:36] yeah, and so I think this is like is a
[45:38] very contentious topic, but I do think
[45:40] that there's a lot of value here,
[45:41] especially, you know, when you want to
[45:43] get stuff moving and get stuff off the
[45:44] ground. So, that's pretty much that. And
[45:47] I think you know, next time we're going
[45:48] to do kind of a lot more customization.
[45:50] We'll talk about skills and containers
[45:51] and how to do stuff um in what's called
[45:53] Yolo mode, which is letting Claude run
[45:56] without any confirmation from us, which
[45:58] is what happens in a lot of the the
[46:00] co-work space.
[46:03] Great. Very good. Thanks a lot, Paul.
[46:05] So, we're looking forward to
[46:07] learning all of this what you told us
[46:09] today, but also what you bring next time
[46:11] to the table.
[46:12] And um I thanks to all of you for
[46:14] watching it and hopefully adopting a lot
[46:16] of the insights provided here.
[46:19] Bye-bye.
[46:20] Bye. Bye.