Spec-driven development: The AI engineering workflow at Notion | Ryan Nystrom

[00:00] One line that I've been putting in my prompts lately is, I literally don't know what I'm doing here. You gotta explain it like I'm a five-year-old. I didn't start with writing code. I didn't start with anything. I just started with an empty markdown document. I actually just opened up Whisper and just started yapping about how this feature should work. I gave the yap session to Codex and was like, write a spec. I then opened up Codex again, pointed it at this spec file, and I said, build it. [00:25] and basically one-shotted this. I've been in software engineering for 20 plus years. We were writing these documents. [00:31] And we were sitting in meetings with other engineers debating the merits of one implementation versus another. And now no more waiting for the meeting. No more waiting for review. I'm not a CI expert, but I kind of know what I want. And so other folks were kind of like, can you just bring some of your like puppy dog energy to like CI and just see what we can do? Your AI, your agent is never going to complain when you ask it to do this five minutes before the meeting starts. [00:58] It is more relaxing and it's more fun. And I feel like I'm getting more done. It's weird to have this like win, win, win. They do the triangle and they're like, pick two. And you're like, no, I'm going to pick all three. Give me the whole triangle. Give me the whole triangle.

[01:13] Welcome back to How I AI. I'm Claire Vo, product leader and AI obsessive here on a mission to help you build better with these new tools. Today, we have Ryan Nystrom from Notion, and he's going to show us as an engineering manager, how you can [01:26] never prep for a stand-up again. [01:29] We're also going to see how you can get a background agent to write code for a fix your friend texted you and how SPECT driven development really works in a code base at scale. [01:38] Let's get to it. This episode is brought to you by WorkOS. AI has already changed how we work. Tools are helping teams write better code, analyze customer data, and even handle support tickets automatically. But there's a catch. [01:52] These tools only work well when they have deep access to company systems. Your copilot needs to see your entire code base. Your chatbot needs to search across internal docs. And for enterprise buyers, that raises serious security concerns. [02:07] That's why these apps face intense IT scrutiny from day one. To pass, they need secure authentication, access controls, audit logs, the whole suite of enterprise features. [02:18] Building all that from scratch? It's a massive lift. That's where WorkOS comes in. WorkOS gives you drop-in APIs for enterprise features, so your app can become enterprise-ready and scale up market faster. Think of it like Stripe for enterprise features.

[02:48] today. [02:49] Ryan, welcome to How I AI. [02:53] I am really excited because you're going to show us, I think... [02:57] Start. [02:59] Medium. [03:00] advanced mode on some AI coding stuff. And so before we jump in, [03:06] How has AI just changed how you live your life at work? [03:11] I mean, it has... [03:13] completely upended. [03:14] the way I work. Like I've been doing this, I did a lot of like mobile iOS work in my past and I worked the same way. [03:23] every day for like 12 plus years. [03:26] And then the last year I have changed [03:29] IDEs, terminals, [03:32] tools, like whatever, like [03:35] 10 plus times. [03:38] So it's like really weird and scary to be like changing this stuff so much. But it's also I feel that like. [03:44] I've been doing this for a while and I'm feeling like so much like joy and freshness and like newness in this that like I wake up every day super excited to like tinker and build things. [03:54] And I'm also like working faster and harder than I feel like I ever have, but like in a good way. I know people are like, [04:02] kind of freaked out about like everything's changing the pace is up but like it's really energizing for me. [04:07] Well, you are not alone. I think every How I AI guest has come on and said, I'm having more fun. [04:13] I'm working faster and everything is different. And what I love about what you're going to show us is it's not just the set of tools, which I think has changed or how we write code has changed.

[04:24] How you, like, run a team has changed. So... [04:28] And I think for the better. [04:29] you know, so I would love to see how you're using Notion AI to actually run teams differently. Yeah, for so for context, I manage a team of like, [04:41] six, seven people. Um, I'm like an engineering manager or, uh, tech, [04:46] technical engineering tech lead manager or whatever we call it i manage people [04:51] Interact code is like my role, which I love. [04:54] And I've run a bunch of different projects here at Notion. The one I'm going to show today, we've nicknamed Afterburner. [05:03] Kind of a quick backstory is... [05:06] I've been kind of vocal about our like, [05:09] DevEx CI for a while. I've just [05:12] I worked at places where it's really slow. I've worked at places where it's really fast. And I came here and I was like, we're kind of like in between. [05:18] But I feel like we're slower [05:20] than we need to be. And eventually this caught up [05:23] to me and somebody was like [05:26] Could you just... [05:27] come fix it and like come work on it. And I'm not like a, [05:30] I'm not a CI expert. [05:33] But I kind of know [05:35] What I want? [05:36] And I think also most importantly, the group that I manage and the org that I work in [05:42] were a little notorious for being like, [05:45] really fast and like very, very AI pilled. And so other folks were kind of like, can you just bring [05:51] Some of your like [05:52] puppy dog energy to like CI and just like see what we can do. And that's what we've, that's what we've done. So we had this really aggressive goal to cut our CI into like a quarter of what it is.

[06:03] we're on the path to doing that but [06:06] So what I want to show you is a little bit about how [06:09] we run projects in Notion and how I'm using AI to kind of streamline those projects. So [06:17] What we're looking at here is like our project hub called Afterburner. [06:22] And so in here, I've got all this documentation. I've got databases. We'll look at some meetings. I have like a automation that looks for [06:30] Any sort of like little wins if we knock off seconds from different jobs or whatever, and we keep track of them in here. [06:37] But what I want to show you is basically how we run our meetings, our [06:42] Small group, we'd run a stand-up every single day, [06:46] And [06:47] doing stand-ups where everyone just like [06:50] is kind of like dead-eyed and going around being like, [06:53] I did this, I shipped this change, or, you know, no updates for me, thanks, is like, [07:00] painful [07:02] And in my opinion, like a huge waste of time. I want to like get to the meat. [07:06] So, [07:07] We have this kind of like automated meeting template that shows up every single day that we run our meetings, which is basically every day. [07:14] And it starts [07:15] Blank. [07:16] And then what I have set up behind the scenes is a custom agent. So we chipped this Notion AI custom agent stuff a little bit ago. [07:26] And this runs [07:28] right after the meeting template gets generated. [07:31] And it looks through all of our like Slack conversation in the last 24 hours.

[07:36] Any tasks in Notion that we close? [07:40] Any... [07:41] pull requests that we've like merged just like all sorts of contexts. Oh, and it looks at like yesterday's meeting transcript as well. [07:49] And then it compiles it basically like a pre-read. [07:52] And this is one from a week or so ago. [07:56] And [07:57] it yeah it shows it pulls metrics it can show us like what our latest ci time is [08:03] It shows some of the things we've decided [08:06] shows like progress on different like projects or like different things that we're trying to like make faster, [08:12] bugs. [08:13] feedback. [08:14] open questions, anything that's of concern. [08:18] And like, [08:19] I... [08:21] Can basically work up until like the minute. [08:25] of our meeting. [08:27] without having done a bunch of like prep. And then we all get on a video call and we look at the screen and we're like, [08:35] Okay, here's what we need to talk about. [08:37] And we'll hit each bullet. [08:39] We have like meeting notes that we run down here within Notion. [08:45] And like all the context is basically captured. All of the like agenda is like [08:49] set up for me. So we spend the entire time talking about like, [08:52] problems, decisions, wins, findings, like what are we going to work on next? And it's less the like, [08:59] Oh, I did this thing. [09:00] Yeah, what I want to call out for folks that are maybe listening and not watching is this is a very detailed.

[09:07] meeting, kind of like pre-read/status update. [09:11] Which if you're a TLM or an EM, [09:15] Running good meetings, unfortunately, is part of the job. And... [09:19] There's a really big difference between a good stand-up and a bad stand-up. And I think the ones you described, these like wrote, like, "I wrote this PR." [09:28] Today I'm going to start on this. [09:29] And then basically like a notes document that's a very high level because some human is putting it together. [09:37] It stinks and... [09:39] You start to like do the other thing that I found, and maybe I'm curious if this has impacted how you work, [09:45] or just made it easier, which is like, you start to have those meetings less because the updates aren't super rich. [09:52] People don't feel like it's a good use of their time. And I think you lose something by reducing the frequency. But if you can have high bandwidth, [09:59] high quality meetings [10:02] with high frequency without the overhead. [10:05] I think you can get [10:07] better, more detailed work done collaboratively, especially if you're running like a remote [10:13] team or not everybody's in the same room. [10:16] So I'm curious. [10:17] Do you feel like just being able to get to this level of detail in your stand up has improved how you can actually build this product and do these do this work? Basically, I've been in way too many meetings where I can tell like everybody's eyes are glazed over. Nobody's paying attention. [10:32] And I like... [10:32] I have... [10:34] run enough [10:35] projects at this point where like

[10:37] To me, that's pretty dangerous because everybody in that room has ideas. [10:43] They have insight. [10:44] But because we've made it so... [10:47] nonproductive and like not engaging that like we're not [10:50] We're just like [10:51] We need the point of the meeting is like exchange ideas and information. And if we're like not doing that, like literally what's the point? The whole like this could have been an email like that's what the meme is from. Um. [11:01] And so I found that this [11:03] is [11:05] basically a good conversation starter. [11:08] And [11:10] A key thing to me that tells me that this is working is when I'm like, [11:16] I'm running these meetings and kind of going through like bullet by bullet. And then I see [11:20] Like, oh, somebody like... [11:22] fixed our mock server environment in our our jest tests and we're seeing a test improvement by like up to 13 percent. [11:29] I'm like, oh, [11:30] I'd miss that, you know? That's super cool. Let's talk about it. And then from that, maybe... [11:37] oh, there's additional headroom that we can make on this. And so all of a sudden, like, let's drill into it. [11:42] And so just like. [11:44] I also think it's sort of [11:45] democratizes that like sharing so that [11:48] You've got some people that, um, [11:51] I could talk for an entire 30 meeting without shutting up. And then other engineers are like, you know, kept keeping to themselves, but like, [11:58] brilliant and super talented. And I think it's great [12:01] this kind of like [12:02] raises everything up to the top and makes it very visible. [12:06] I was thinking last night, truly, this is a little bit of a sidebar, but I was thinking last night how people don't know this. It's very, very confusing, as Claire Vo, the podcast host.

[12:15] I am such an introvert. I'm a crazy introvert. I will avoid humans with every fiber of my being. [12:23] And I find like AI as a proxy for me to get over my own anxiety communicating with the other side world. It's like so insane. But I really feel it. Whenever I like message my open claw and I'm like, hey, Polly, can you email so and so blah, blah, blah on my behalf? [12:41] my anxiety is gone. But when I sit in front of my Gmail and start typing an email, my introvert starts showing. I'm like, I can't, I can't do it. So I think this is a real point, which is [12:51] Like, you got to pull information out of everybody. [12:54] And there's like spikes and valleys of [12:56] who is going to come and show off their work, how detailed they're going to be. It does not mean the work is not good. It does not mean they are not talented. [13:04] they just have different skills in different environments. And so I love this call out of like, [13:09] We're going to pull equal amounts of information out of everybody. [13:13] because the playing field is equal. [13:16] I also want to go back to something you said, which makes me think about burnout. [13:20] But she said, I can like work up until this meeting starts. [13:23] And I know so many people who feel like they're in... [13:27] meeting after meeting after meeting, [13:30] One, because when they're not in meetings, they're preparing for meetings. So you're getting rid of that. Yeah. [13:35] And then two, [13:37] I think managers being able to, for example, [13:41] Code? [13:42] up until the standup is such a burnout like protection mechanism.

[13:47] which is you would much rather your managers hands on the code, hands on doing work, filling a creative impulse, [13:55] them prepping for meetings. Sure, they're great at running the meetings, but let's do that in like this just in time. [14:00] mechanism where they're supported. [14:03] through these automations. And so [14:06] I just imagine I used to feel so stressed as a leader and as a manager being like, [14:11] If I'm not in a meeting, I'm prepping for a meeting and then I'm in a meeting, [14:14] And if I can carve back more of my time to just do real things, [14:18] that feels so much, so much, so much better. So I'm guessing you're having more fun just through the balance of the kind of work you're trying to do. [14:27] I'm having so much more fun working this way. I have done the like, [14:33] run a big engineering group where, yes, you're spending half of your time just [14:38] compiling information, synthesizing it, writing reports. [14:42] like doing all this stuff and, uh, [14:44] I hated it. Like, it's so draining. Yeah. [14:48] And now I feel like I'm in a sweet spot where I can support and work with a team of like very... [14:55] talented individuals, [14:57] but also not have to, yeah, be like doing like paperwork the entire time. Like I don't want to do like the tedium. I get a lot of joy out of, [15:05] working with people. It's like why I like managing. I like talking with them, having fun. I love like [15:10] start solving hard problems together. [15:13] And then I like... [15:15] building stuff and--

[15:17] Yeah, I think that we're maybe at... [15:20] an inflection point where [15:22] Like... [15:24] Maybe this is controversial or not, but... [15:26] Like if you're like a line manager, [15:29] like [15:30] Write code. [15:31] You know, get in there and like stay, stay close. Maybe don't do the, the, the P zero hero projects, but like help your team fix bugs, like make optimizations, like whatever, you know, I, it's just. [15:42] It's so easy now. [15:44] I mean, I'm going to pull that thread all the way up, which is like directors of engineering, VPs of engineering, like CTOs. Yeah. [15:52] CPOs, write some code. Now is the time. And I say this all the time on this podcast. This is the era of the hard skill. This is not... [16:02] "How do I get better at my soft skills and managing stakeholders?" This is literally like, [16:06] How do you write code? How do you write automations? How do you learn these new tools? How do you understand what models do do what for your own skills? [16:13] That eating is super important. [16:15] Okay, we could go on forever on this topic. I do want to show folks how you [16:19] built this. Yes. Because we haven't seen [16:22] actually haven't seen we've had a couple notion folks on the podcast we haven't seen notion AI [16:27] in action and I just want to see kind of your thought process on how you build something like this out. [16:32] So I'm going to flip over to this or a custom agent. Don't ask me why we got this like potato theme for like the entire project. I think it was kind of something about like CI is just this like cobbled together like mess. And so we're going to like make the potato like a rocket ship.

[16:47] I don't know. I don't even know if that makes any sense, but now we're having fun with it and we have... [16:52] reactions and like agents and it has spun off into its own thing which is fun [16:58] But this is our hot potato agent. So you can see I have this set up to run at 9 a.m. every single day. [17:05] It's also set up for chat. It's set up if the agent's mentioned, but we never really use any of that. And probably the important part... [17:12] is the actual instructions. [17:14] So in this instruction page, [17:17] giving context on like what the purpose of this agent is [17:21] I am telling it to run. Yeah, look back at [17:24] 24 hours so basically telling it your job is to run every single day and i only want you to look back for the last like 24 hours [17:31] of activity. [17:33] I'm explicitly telling it to use sub-agents, which is... [17:35] kind of a sleeper feature in Notion AI. Like this exists, but we don't [17:40] I don't really push it to use it very often yet because it's one, it's very expensive. And two, it can be kind of finicky sometimes. But. [17:48] Um [17:49] I helped build it, so I know how this works. [17:53] And then I ask you to kind of like fan out and do like a map reduce where I'm saying, go use the Honeycomb MCP to figure out what the latest metric is. [18:01] look in our project channel and like find updates, feedback, questions. [18:07] I tell it where the task [18:09] databases and how to look for tasks within this project. [18:13] and then how to find yesterday's meeting. [18:15] And then [18:17] I give it a template in the instructions or I'm like, this is your format. I care about CI speed, decisions, progress, changes, bugs, questions, risks.

[18:27] A little bit of guidance on writing. [18:30] And then when it's done, I have it post to Slack. [18:33] And I like emphasize this like, I want it to be brief and fun. And sometimes it's really corny. And then sometimes it's like, [18:39] really good uh and it'll be very quirky and just post this like link in our slack channel and it's like hey here's your your pre-read some little quibble about like whatever you know like hey you guys are not making enough progress [18:53] Uh, [18:54] And then that's it. And then we have our meeting note. It's updated. [18:59] What I like the most about this one, I'm going to show you some of our, like, [19:03] internal settings so this is like i give it access to all these things i'm like you can only view all of this stuff because i don't want it going in like [19:11] Modifying our task database, our project database, everybody at Notion uses those. [19:16] But this meetings database in particular, I'm like, you know, you can edit content because this is the one you're going to like write and update the page. [19:24] It can... [19:26] read from our slack channels respond to our project one [19:31] And then this was new to me, actually, when I set this agent up, is our MCP. [19:36] So we've had MCP and we have this other thing. [19:41] called workers which is like kind of like writing code [19:44] I haven't used them very much within custom agents, but in this one in particular, I'm like, I know exactly where this metric is. [19:50] It's in Honeycomb. [19:52] And so I like just... [19:53] configured the MCP in Notion. By the way, I like used...

[19:58] The agent set itself up. [20:00] I was like, here's the query. [20:02] I literally gave it a screenshot of the Hidingcomb query. I was like, I don't know how this works. [20:06] Can you just like update your instructions? I love that you screenshot it. You didn't even copy and paste it. You're like, please OCR this screenshot. Exactly. Too lazy. I'm like, here it is. Just take it. [20:18] figured out. And it, [20:19] It kind of, it got it mostly, like most of the way there. I had to fiddle with it a little bit, but yeah. [20:26] What I appreciate about this, and again, for anybody trying to just brainstorm... [20:31] workflows where AI can actually have a huge impact on your productivity at work or life [20:36] I just feel like write down what you would do if you had time. [20:39] If you had time every morning at nine, you would sit down and you with your eyebrows would go through Slack. You would go through Honeycomb. You would ask people what's going on. You would look at GitHub. [20:50] And then you would compile it. And then you would try to be very fun in Slack. Like, it's just a description of what you do. [20:58] would do and it doesn't have to be that complicated. You can iterate so quickly on it. What I appreciate about this, [21:06] you know, versus [21:07] old era of more deterministic like workflow style builders is the updates are so easy to make. Just change the natural language, redo the order, change the trigger, give it more [21:19] access to data and then [21:22] It's ready to go. [21:24] Yeah, and you know, I think [21:26] The other thing I've gotten hung up on when trying to think about like these automations and I've seen others do is that like,

[21:33] You get this, like... [21:35] you start gigabraining it and you're like, well, how am I going to save like, [21:39] five hours of work a day and I, [21:42] What you just said, [21:43] made me realize too that like the tedium that this removes for me is not like [21:50] world changing. [21:52] but it's like 20 minutes a day. [21:56] And that's like 20 minutes I can spend doing other stuff. And that like... [21:59] It's not even just about like saving that 20 minutes, but it's like, [22:03] protecting my brain from like having to context shift about all this stuff and like ingest it and instead [22:09] Yeah, it's just I know that the information will be there when I'm ready to like read it and I'm ready to like shift gears to this project. [22:15] rather than... [22:17] Bye. [22:18] I hate doing the like... [22:20] read this update copy all this information like put it into it like an update board like [22:24] It's soul sucking. I mean, you and I have been doing this for a while. That's like that was I feel like 70 percent of my job at some point. 70 percent of my job was just like, what's going on? How do I massage it into a format appropriate for the audience at hand? And it's always the same information. It's just like, what's the executive version of it? And what's the team version of it? And what's the full team version of it? And like. [22:52] Like my shoulders dropped out of my ears when I realized we don't have like we just don't have to do it anymore. And the other thing that I think people maybe under appreciate about AI and this like just in time delivery.

[23:03] is incredible. [23:04] Your AI, your agent is never going to complain when you ask it to do this five minutes before the meeting starts. [23:10] I know. It's so great. It's so great. It's just like when you have it, drop it. [23:16] get it done and out of your brain, I think is again, I go back to like burnout and enjoying your work. [23:24] and reducing toil, and it just feels like a more... [23:28] relaxed way to work. [23:31] we're happy it's so funny because it is more relaxing and it's more fun and I feel like I'm getting more done it's weird to have like this like win win win [23:41] Yeah. [23:42] You know, they do the triangle and they're like, pick two. And you're like, no, I want to pick all three. Yeah. And I'm like, give me the whole triangle. Give me the whole triangle. Stamp it for the YouTube thumbnail. Give me the whole triangle. All right. [23:53] This episode is brought to you by Orcus, the company behind Open Source Conductor, which powers complex workflows and process orchestration for modern enterprise apps in agentic workflows. Legacy business process automation tools are breaking down. [24:08] Siloed low-code platforms, outdated process management systems, and disconnected API management tools weren't built for today's AI-powered world. Orcus changes that. With Orcus Conductor, you get a modern orchestration layer that scales with high reliability, [24:22] It brings humans, AI, and systems together in real time. [24:26] It's not just about tasks. It's about orchestrating everything. [24:30] APIs, microservices, data pipelines, human-in-the-loop actions, and even autonomous agents. So build, test, and debug complex workflows with ease, all while maintaining enterprise-grade security, compliance, and observability.

[24:44] Orcus. [24:45] Orchestrate the future of work. Learn more and start building at orcas.io. [24:51] Let's talk about, so we're talking about how meetings happen. Love this. You do write code though. [24:57] I do write code. Sometimes with your fingers and sometimes with [25:00] You're my favorite. [25:02] harness of the moment. So let's go to how you get [25:06] Coded. [25:07] Yeah, I want to show the little bit of a new workflow that we have going on in Notion. [25:15] I honestly don't think this is necessarily a big feature that we're going to ship or we might ship some version of this feature. So this is all basically internal only at this point. [25:25] But... [25:26] Prior to this, [25:27] the way one, [25:29] I love Codex. I've been a Codex stan for like [25:33] I don't know, six, seven months now. [25:36] And we started building this codex integration into Notion. Prior to this, it was like, [25:43] I mean, obviously you're using the CLI to like write your prompt and then they created the Codex app, which is, which is nice, but I'm still like writing my prompt in this thing. So I started actually writing prompts in Notion pages where I can be a little bit more like freeform and structured and [25:59] You know, I'm in like the CLI. I don't have to worry about like hitting enter and then like, oh shit, like I sent my prompt. I can actually like... [26:06] write a document in Notion. [26:10] But then of course I'm like cop or like highlighting all of the texts, copying it, going into my terminal hitting pace and like letting it go. It's like fine.

[26:17] Um... [26:18] There's like MCPs and other stuff that I could be using, but I'm a very simple person. It was like too much, too much stuff. [26:25] So we built this [26:27] thing that we're kind of like calling it [26:31] I think we're calling it both Software Factory, but I like its internal project name is Boxee. [26:36] um because it's like all these little vms that we install codex and cloud code on it's our little boxes [26:42] Where now we can actually invoke them from... [26:46] like tasks within Notion. And so... [26:49] Literally, [26:50] happened this morning a friend of mine um who's a notion fan texts me he's like hey i like [26:56] the tab block that you built [26:58] which is this thing. [27:00] He's like, but I really wish I could like, I can like copy, I'll show you now. [27:04] If I click on the dot, dot, dot thing next to a block, there's like... [27:09] copy link to block he was like man i really wish i could just like copy link to a tab [27:14] and then like send it to somebody. [27:16] And I was like, oh yeah, [27:17] That sounds really easy. So [27:19] I opened up this [27:20] task and i just took some notes and i dropped in this screenshot showing them uh where it could live you know this is like on the the tab block if i right click on it and get this little flyover menu [27:33] And I just described the task. [27:35] I was like, yeah, let's put a copy link button here. I also noticed that hovering over the delete button didn't change to red. So I was like, yeah, we should fix that. [27:43] And I was like, I know one edge case is like, [27:46] You're gonna have to like,

[27:48] If you land on a URL, [27:50] where... [27:51] We're like linking to this block. [27:53] and this tab like it's got to like switch to the tab on like fresh refreshes so i like [27:58] kind of outlined all the cases, but [28:00] You can see here, this is one, two... [28:03] three paragraphs for like four sentences and a screenshot was like not a lot [28:09] And then [28:10] this new thing that we built, I can actually mention [28:13] codecs from within our comments. [28:16] And this triggers are like boxy. Uh, this is all our like internal dev tooling stuff. [28:21] And then it got to work. [28:23] And [28:24] I was looking at the timestamps earlier and I think 10:40, 10:51. [28:29] started the implementation, [28:31] And then another 10 minutes later, [28:34] It replies with a pull request link. [28:37] And a preview URL because we do the preview environment stuff. [28:42] And [28:43] It built the entire thing. And if I switch actually over to... [28:49] Here's the pull requests. [28:50] It built like the entire thing. [28:52] It was like, here's how I tested it. [28:54] And this was actually the coolest part to me, is it actually uploaded screenshots of it doing its own UI verification. [29:01] And [29:02] There was like a CI failure in it, so I was like, hey, [29:06] I replied down here like, [29:08] Oh, yeah, this part of this code change, I was like, I don't know what is going on here. This doesn't make sense. [29:14] and there's some type check things, and it was like, cool. [29:17] Here's why we did this change.

[29:20] And I fixed your types. [29:22] And then there was a merge conflict. [29:24] Let's also talk about just a world new world. [29:27] the emotions around code review. Like, I love that you're like, [29:34] I don't get this. [29:35] Not I'm going to try to get it. [29:37] Or I've done my best to investigate and I'm pretty sure this is wrong. Like literally just I don't get it. [29:42] I don't get it. [29:44] I mean, I didn't like sending that to somebody. [29:47] One of your human teammates. They're like, "No, look it up." So that's one of my code review mechanisms. [29:55] as well as i'm just like i don't think this is right [29:59] Yeah, one line that I've been putting in my prompts lately is I'm like, [30:04] I literally don't know what I'm doing here. [30:07] You need to explain this to me, especially doing all the CI stuff. I'm like, I'm in over my head. Like, you got to you got to, like, explain it like I'm a five year old. And sometimes it's like literally like caveman style. Like, here are the things. And I'm like, I needed this. Well, what I appreciate about this and we'll talk about it a little bit later, I think, is. [30:27] We're not getting any of these Claude Code warm fuzzies from Codex. I love it. [30:32] But when it talks to me, I'm like, I feel real dumb when you're talking to me, Codex. Can you? [30:37] - Give me chills. [30:39] Okay, dum-dum, here's what's going on. [30:42] So I think that sort of like, I don't understand what I'm doing, please explain it to me, is... I love it, but I think it is a codec-specific experience that you get. Claude Code shows up and is like, hey, buddy, guess what I made for you?

[30:59] Okay, so stepping back before we go too deep on the personalities of all these coding models, you have built just... [31:07] At mention, and I think people miss this, but it's important for particularly our engineering team and engineering leader listeners to pay attention to, which is, [31:15] Everybody that knows it's smart. [31:17] It has some background agent hooked up to a virtual machine that they kick off. [31:22] so that you're not spinning up your local environment and doing all this stuff on your machine. [31:28] And it just, I think the, we'll talk about this, but the velocity plus your DevEx plus your CI, I think is, [31:35] a huge piece of AI adoption in engineering. [31:39] If you are not, if you don't have a VM strategy and background agent strategy in your large engineering org, [31:46] Time to get one. Time to do it. And we'll just ask Codex to build it. I don't know what I'm talking about, but I think Claire said this is a good idea. Please build. Make no mistake. [31:59] Okay, last one. We have a last use case you were gonna walk us through. [32:06] Yeah, so we recently rebuilt our... [32:11] entire agent harness. [32:12] The quick TLDR is just like everybody else. We reached like this point of tool and destruction fatigue where you had this bloated system prompts. We're like, okay, we need to dumb this down and, [32:22] We borrowed the kind of like skills and progressive disclosure concept from coding agents and brought that to Notion AI. [32:29] And when we were doing this big rewrite, we were asking ourselves, like,

[32:33] You know, it had only been like six months since we shipped. [32:36] the last rewrite. [32:38] And this time we were like, what would we do differently? Like seeing the state of the art with coding agents. And someone had this really great idea that, [32:47] to like, let's not start with code. [32:50] Like, let's just start with specs. [32:53] And. [32:54] What we've ended up building is we have this like in our checked into our code base. You see this. We have this. We're looking at. This is a notion repo. But we have this agent specs subfolder. [33:05] And within this subfolder we have [33:07] all of these markdown documents. This is one that I worked on. We have this thing in our AI called Ask Mode where we basically [33:16] ban all the like mutating tools so it can only just like read and answer questions. [33:21] And so when I was building this, I didn't start with writing code. I didn't start with anything. I just started with an empty Markdown document. [33:29] And, uh, [33:30] I mean, I actually just opened up like Whisper. [33:33] and just started yapping about how this feature should work. [33:37] And at the end of it, I gave that [33:40] I gave the yap session to Codex. [33:43] And it was like, here's our other like spec library, learn the format. [33:47] Take my information. [33:49] Write us back. [33:50] And then it spiked the first version. I did a couple revisions on it. [33:56] And I ended up with this Markdown document. [33:58] The Mark Dunn document is like... [34:00] It's nice, but what we did with it next, in my opinion, is I kind of think that this is like the future of software engineering.

[34:07] where I then opened up Codex again, [34:11] pointed it at this spec file, [34:13] And I said, [34:15] Build it. [34:16] And [34:17] It basically one-shotted this because the entire spec file is so comprehensive with code pointers, with [34:23] Down at the bottom, we have verification. It was like, here is how you're going to verify all of this stuff works. [34:31] And we've even built our own CLI tools so that you can run Notion AI from the CLI. [34:37] And it could, once it's done seeing that all the tests pass, it can actually just spin up Notion AI [34:43] itself. Send it queries, send it questions, enable ask mode, disable ask mode, [34:48] and then see the transcripts and see what actually happens. [34:52] And I think the first shot of this took a couple hours, but I came back to whatever, a couple thousand lines. [34:58] Did some code review. [34:59] played with it myself, and I was like, [35:01] It's right. [35:02] It's like done. [35:03] and [35:04] You know, since then we've made, I can, the other beauty of this is like, this is inversion control. [35:10] So I can go to the past changes of this spec file, and I can see how the spec has evolved. [35:17] And it. [35:18] I could go look through all of the code changes, which also have their own like history. [35:22] But this is now the sort of like source of truth for how this part of Notion AI works. And it's just in plain English that can then be verified and implemented by agents. Well, I think the other thing that people don't appreciate is, you know, taking this outside of the engineering flow approach.

[35:39] is [35:40] This plain English can be ingested by other parts of the business that need this information. [35:45] So let's say you need to then release this feature. [35:49] via some sort of marketing. [35:51] This is actually like a pretty good asset. [35:54] that explains how it works. [35:56] that can be translated into another another thing in a way that like code itself [36:01] is still a little intractable. And so, [36:04] this idea of spectrum and development but what i like about what you said i don't want people to miss is [36:09] The way you make these updates is you update the spec and go... [36:12] Go look. [36:13] make the update, change the thing. [36:15] And so like the spec is the source of truth. The spec as the, the, the change log, I think is a really interesting point. [36:23] model and [36:25] For people that aren't watching, [36:27] It's very detailed. It's very technical. So it's not like there is not code. [36:32] in the track. [36:33] There's just not all the code in the spec. And I think that's a really kind of good hybrid model for the [36:42] experienced engineers to start to bridge into what would it look like to have an agent [36:48] Do more of your coding work while you still do architecture work, while you still do design work, while you still make sure that the thing is going to scale. [36:56] Yeah, exactly. [36:58] I view... [36:59] our job as like engineers evolving into like systems thinkers and architects [37:06] And not even just necessarily writing the spec and thinking about the behaviors, but

[37:11] most importantly is like the verification loop like [37:15] Is it a-- how should it verify correctness of this feature? [37:19] or this change, [37:21] And honestly, it's like if it can't or if like the verification is a little hazy, it's like that's the first thing you actually should be going and doing is like. [37:29] Do you have a tool to let the agent [37:31] run itself. That's like one of the first things we did with this project was like we should actually build a CLI [37:36] so that I can tell codecs, send this prompt and see what happens. [37:40] And now that we have that, then we can take these specs and actually just like go deeper and deeper and deeper. [37:46] And so it's like we're still doing engineering, but I'm not doing the plumbing work of wiring up this Ask Mode feature. [37:53] Well, and what I think is really funny is I've been in software engineering for [37:58] 20 plus years like we were writing these documents. [38:01] Anyways, we were writing technical documents anyway. And we were sitting in meetings with other engineers debating the merits of one implementation versus another. [38:13] and then we still had to go write the code yep it's like really it hasn't added work to to go into this model [38:20] It's maybe like shifted the emphasis of where the human attention goes. [38:25] But these were documents that at least I was in every org that I've ever been in writing before. And the other thing... [38:32] that I think has changed so much is those docks [38:35] then waited for review. [38:37] and they waited for a meeting, [38:39] And now, [38:41] No more waiting for the meeting. No more waiting for review. Ship it.

[38:45] have a verification loop, [38:47] debated on the merits of it being live and working versus the theoretical merits [38:52] of it sitting in a document waiting for everybody's calendar to open up. [38:56] for... [38:57] you know, a live argument. [38:59] Yep. [38:59] Yeah. [39:00] Couldn't agree more. [39:01] Let's let's do it. This is the mic that I just you and I could talk all day about this. Just to recap for everybody because I know I got to get you out of here. [39:09] Three use cases. One, never prep for a meeting again. Hook it up not only to your Slack, but to your meeting notes, your GitHub, your telemetry, and build the best stand-up meeting so no one has to stand there glassy-eyed, being boring, giving updates. [39:23] Second use case, background agents, at mention from wherever you work, Notion being a great place to do that, kicking off virtual machines, getting PRs done, just... [39:33] saying yes when your friend texts you, can you ship this feature? [39:36] And then the last one, [39:38] putting all your specs in your repo, using them as a source of truth for a more autonomous coding agent like Codex, [39:46] let it cook for a couple hours, [39:48] Review the code. And then when you update, update the specs. Don't update the code. Did I get it right? Nailed it. [39:53] Okay, let's do a couple lightning round questions and I'll get you out of here. [39:57] You and I love Codex. [39:59] why why you love codex i'll tell you why i love codex but you go first [40:02] Okay, well, so I first fell in love [40:07] Oh my God, did I just say that? You did. I first fell in love with Codex. [40:12] It's because when I was evaluating both Cloud Code and Codex,

[40:18] I found... [40:20] any time Cloud Code, like, [40:21] filled up its context window window it would just kind of like lose the plot really quickly [40:26] And Codex, I don't-- [40:28] I don't know exactly what it's doing, if it's the model, if it's the compaction, if it's like both. [40:34] It can grind for like hours. And with the way that I work, both the systems and things that I work on, [40:43] And just like, I like to be able to fire off a bunch of them, like at the same time, and then like go to a meeting or go do something else or like... [40:52] spend my time kind of like round robin managing like all of these agents like I don't necessarily want to I'm not the person that is like sitting with the browser open and like some agent next to it and like iterate look at the browser iterate look at the browser I'm like [41:07] the more the closer i can get to one shotting solutions the better because that frees me up to do other stuff [41:13] So I found that Codex was pretty good about that. [41:17] I also just feel it's like, [41:19] Pretty simple. [41:21] Like it's there's not a lot of bells and whistles. It's not like too fancy necessarily. I'm happy with [41:29] The addition of like MCP and skills and some like other stuff coming out. I also really love GPT-5-4. I think it's a great model. So I'm like, all of those things together, it's just... [41:42] It really like matches my working style and type of work a lot. [41:46] Yeah, I'll tell you why I love Codex and I sent this to somebody I said,

[41:51] work trees everywhere, [41:53] ports 3,000 through 3009 spoken for. Like we're just, we're just going across, across the board. I do think it's fit at long running tasks. I like its concept of projects because I run a lot of different projects. I think it's just like a very helpful. [42:08] mental model and then it's great at code review. [42:12] Honestly, it's just like a really good code reviewer. It's a really good security reviewer. [42:18] tireless, uncomplaining, with the most tedious of things. And so I find it's [42:26] one of my daily drivers i just i really like it it's good okay second question because you and i also agree on this [42:33] give people the reason this isn't even a question this is a demand for you to share my point of view which is [42:39] Why bang on developer experience and CI speed right now? [42:44] A couple things. To me, see, I was like, [42:47] super important prior to agents. [42:51] Because in my opinion... [42:53] I mean, it's not even an opinion. It's like fact. The faster your CI completes, the faster you get signal. [43:00] The faster you're like, the more comfortable your engineers will feel with like making changes and doing things because they know. [43:08] Like I can make a change and get it pushed into dev or into production really quick because I know my CI is fast. [43:14] If it's slower, then you're building up these like [43:17] monster changes and you're going to be even slower about like judiciously like reviewing every little

[43:23] like tiny thing [43:25] when I'm like a learn through like doing sort of person and if [43:30] I can close the iteration loop even tighter. [43:34] then I'm going to be putting out a change. People are going to be using it. I will take that feedback. I will make another change. And I don't have to worry about... [43:41] Um... [43:42] you know, it taking another day before like the deploy train is ready to go. I just want to like, I want to crank really, really fast. [43:48] And that was all pre-agent. And now that we're like in the agent land, [43:52] It's it's that but like on steroids because agents don't get tired. [43:57] they can work on a vm they can work like while i'm sleeping [44:02] And if I've got a CI loop that takes an hour to run, [44:08] Your agent's just going to sit there and spin for like an hour waiting for results to like do something. If it takes three minutes to run. [44:14] Like, holy crap, how much more stuff are you as a human and then especially as your little swarm of agents going to be able to get done? [44:22] Like so, so much more. And so I, [44:25] Super important. I agree. And we just had Steve from Stripe on, and they're doing like 1,300 agent PRs a week. [44:33] You cannot do it. [44:35] Yeah, you like cannot do that if your CI is slow. It's just you might as well be throwing all those PRs in the trash. [44:42] And so I do think there is just a true mathematical limit [44:47] on your capacity to ship code to production. [44:50] that is a reflection of how fast your CI pipeline is. And so,

[44:54] If you are, again, engineering leaders, if you are not spending time on making that fast, [44:59] and you are tolerating something slow there, [45:01] you're not going to get the benefits of AI that you could. And honestly, like good for AI, good for humans. [45:06] No engineer wants to sit waiting for... [45:09] their stuff to hit production. It's miserable on the engineer side, lots of downsides. [45:14] So allocate time. [45:16] to your pipeline, please. [45:19] - Yep. - Ryan and Claire say so. All right, last question. [45:23] When AI is not listening, [45:27] It's writing bad specs. [45:29] It's not being funny in Slack. [45:31] What is your prompting strategy? Do you yell? [45:35] Yeah, I can tell. I can be a little bit of a diva sometimes if it like really goes off the rails. [45:42] The... [45:43] The other prompting strategy that has saved my ass working on the CI stuff lately [45:49] is because even the best models, I feel like, can sometimes be a little sycophantic. [45:54] I'll be like, [45:56] I will just be like, you're wrong. You need to defend your argument. [46:01] Because I want it to defend it in the way that I want, but I'm like, I just need... [46:07] I just need to see the evidence that if I push counter to what it has done, that it can like back up with like good. [46:13] pointed reasons rather than just be like, [46:15] You know, [46:17] "Are you sure this change looks okay?" It's like, "Oh boy, it's totally fine." [46:21] I mean, no, no, no, I need like [46:23] the sighted

[46:24] hard argument against it because like in a lot of times like with the CI stuff, I'm like, I, I, [46:30] don't know what I'm doing. [46:32] I know generally what I'm doing. [46:34] But like the specifics are a lot more nuanced and I like I need to get this right. And that's been super helpful. And then when it goes off the rails, I'm like, I could be such a diva to it. [46:45] escape all caps no that's what i do interrupt and no steer the conversation little arrow button has never gotten so much work ryan this was [46:57] So much fun. [46:59] Where can we find you and how can we be helpful? [47:02] You can find me on X. I'm out there shilling for Notion right now a lot. But that's basically where I spend my time. Yeah, my DMs are open. So if you have Notion problems, I like to try and fix them. I'll send it to Boxy and we'll get it done. [47:19] Perfect. Love it. Well, thanks for joining the podcast. [47:22] Thanks for having me, Claire. [47:33] You can also find this podcast on Apple Podcasts, Spotify, or your favorite podcast app. Please consider leaving us a rating and review, which will help others find the show. You can see all our episodes and learn more about the show at howiaipod.com. See you next time.