# I was laid off by Atlassian

https://www.youtube.com/watch?v=55pTFVoclvE

[00:00] I was recently affected by the layoffs made by Atlassian and I wanted to take some time out to reflect on the time that I spent working for Atlassian.
[00:11] I worked there for about eight years.
[00:13] During that time I built a lot of things and I wanted to talk about what I built, mainly the things that I personally found interesting or that I'm proud of.
[00:21] I hope that this video will be useful or helpful to someone who perhaps is or was in the same situation as me and maybe it'll give them some inspiration in terms of how they can tackle the same things that I did or something similar and perhaps avoid some of the mistakes that I've made.
[00:39] I also might talk about non-technical parts of my experience at Atlassian, although most of it will be technical and this video will be split into chapters so that you can skip to sections that are more interesting to you rather than rather than watching the video from start to finish.
[00:56] So I suppose to start with I'll talk about when I was.
[01:00] First hired and even though it was eight years ago, I still remember the interview process, which is different nowadays, and the reason why I was hired or at least from my perspective the reason why I was hired and the things that I started working on during the start.
[01:15] So yeah, let's just start at the interview process.
[01:18] So I was interviewed by some people that I now consider friends and I remember having the impression while being interviewed that these individuals were quite intelligent and that was something that was exciting for me.
[01:32] The interview process consisted of a coding quiz on HackerRank, which I aced with full marks.
[01:36] Then the first technical interview was with two interviewers and they gave me a white paper and asked me to read it while they sat out of the room for about 10 minutes.
[01:48] They came back in and then asked me questions about the white paper, asked me to basically articulate what was in in white paper and the white paper was actually about custom domains, and the white paper was by Cloudflare.
[02:01] They then asked me a few questions about things like microservices and architectural things like that, um containers and and whatnot.
[02:07] And they were happy enough.
[02:09] I don't remember the rest, but they were happy enough with me during that stage, so I continued to the uh second technical interview, which was a troubleshooting exercise where I was asked to essentially prompt the interviewer for information in order to troubleshoot a real incident that occurred in Atlassian.
[02:28] And it was a it was an application problem that lead led to a denial of service.
[02:34] Uh so that was fun.
[02:36] And then I think I was asked something about how um latency-based DNS works, and my answer was not accurate, but perhaps acceptable.
[02:45] I I I thought about it from first principles, and I thought that that's uh for example, I thought that Route 53 did a triangulation based on the actual latency of the client, but it is more like that they use a uh they.
[03:01] Probably use a geolocation database in order to do latency-based routing of DNS requests of the DNS answers, sorry.
[03:10] Then after that was a values uh interview.
[03:12] And to be honest, I don't really remember most of the questions for the values portion, but I do remember one thing, which was when I asked the question of I asked I asked the interviewers to think about 12 months from now and to look back retrospectively, what is the thing that I would have had to achieve in order to for for you to say it was a good decision hiring this person.
[03:36] And then they told me about an application that they needed to be built for the platform within Atlassian, and the application would facilitate self-service load balancers.
[03:47] Sort of similar to if you were using Amazon application load balances or the equivalent in any cloud provider.
[03:55] But for the internal developers of Atlassian.
[03:58] And it was essentially just a framework that I personally was not.
[04:03] Familiar with.
[04:05] And I said I could build it because I had confidence in building web apps with Python at that time.
[04:10] And they accepted my level of confidence and decided to hire me.
[04:13] So that's the the interview portion completed.
[04:17] So I joined Atlassian and they have this classic saying or impression that when you join Atlassian that you are drinking from the fire hose because there's so much information that you have to absorb in the first few weeks and months in order to just sort of get going.
[04:35] My first my very first task that at least task that I gave myself was to build the application that they had told me that they wanted.
[04:43] Let me just open a browser and we'll take a little looky at what I mean exactly.
[04:48] Let me just uh move my face a little bit.
[04:51] There's more real estate.
[04:52] Now obviously it's scalar draw is not what I care about.
[04:55] So they wanted me to build an open service broker.
[04:59] This is a a web app with an API which facilitates the provisioning of.
[05:04] Resources for a platform essentially.
[05:07] So you can see here it's sort of built to operate in a Kubernetes world where you're submitting these provisioning requests as things come up and down.
[05:17] And it's going to bind a resource to your pod or your cloud instance or whatever it is as you can see here.
[05:25] And it sort of sits in between these real resources.
[05:28] So you might provision something like a database and then you'll get MySQL.
[05:34] So you'll get something that's SQL compatible but that's abstracted away for your internal developers.
[05:40] Anyhow, the spec is here on GitHub.
[05:42] You can take a look at it.
[05:46] It goes into, like, for example, the catalog endpoint.
[05:51] And the catalog lists all of the services and plans that are available on the OSB, and just metadata about them.
[06:00] And you might say query the service broker and then.
[06:06] Display some of the metadata in your.
[06:08] Maybe you've got a console, like, the Amazon console, but maybe you've got something like that internally.
[06:14] Where developers can click and provision things.
[06:16] In the Atlassian case, it was all through configuration files that were committed to, um, version control, and then those would be uploaded, uh, during uploaded from a build server to deploy a service.
[06:30] Um, but, yeah.
[06:30] So, you might have, you know, other APIs like, uh, provisioning here.
[06:35] So, put and patch for updating and deletes and blah blah blah.
[06:40] So, you would just basically go ahead and implement this.
[06:41] Or I mean, if you wanted to build your own, and that is essentially what I did.
[06:47] You can see also there's an open API document here that has the endpoints.
[06:52] So, I chose to build this in Python using Flask.
[06:54] Oh, no.
[06:54] In fact, what I built it with first is with a library called Connection.
[07:02] This is a Python library which takes an open API document and then creates the API.
[07:09] Handlers for the paths for the API for the API routes that are in that document.
[07:13] Which is cool, but then we eventually I eventually migrated that to just pure Flask.
[07:18] And then eventually migrated that to, uh, Fast API, which I believe is what it still is at the moment.
[07:25] Um, okay.
[07:29] So, it's the first 2 weeks, and my primary focus is to build sort of what I promised in the interview, which is this web app that's going to be a broker for the platform, and is going to allow self-service provisioning of load balancing in Elastic.
[07:44] So, like I mentioned earlier, I started with this library called connection, which took an open API document, turned that into routes.
[07:49] But, I'm going to just go with what it ended up as, which is a fast API app.
[07:56] Let's just say we've got fast API here, and then we've got a worker, and then we have a database, which was DynamoDB.
[08:06] Oh, that's annoying.
[08:09] And we would have a client making.
[08:11] Requests.
[08:12] That's why that's a fast API.
[08:14] The client would say, "Hey, please provision something for me."
[08:19] And the web worker wouldn't do it itself.
[08:21] It would actually send that over SQS.
[08:25] It would drop the task details into SQS.
[08:28] And the worker would then handle that.
[08:31] So, what does a provisioning task actually look like?
[08:35] It's something like creating DNS records somewhere, maybe creating a CloudFront distribution, maybe creating some API calls.
[08:51] And this would be the provisioning task that the worker would do asynchronously, while the web and client would wait for it to be completed essentially.
[08:59] So, the client's polling continuously to say, "Is it ready? Is it ready?"
[09:06] And when it is completed, the worker writes it to the database, the web server checks the status, and then responds saying, "Yes, it's finished."
[09:12] Or it'll say that something went wrong and there was an error.
[09:16] So, then we can sort of encapsulate this as the open service broker that I built.
[09:23] Pretty straightforward.
[09:25] Um to be honest, there's not much more to this, but we're going to go and talk about some of the more complicated bits in just a second, and I will directly link to this as well.
[09:35] So, we got this client requesting uh let's say "Please provision a load balancing."
[09:41] And that is essentially what they were asking for was some kind of load balancing somewhere in the edge infrastructure of Atlassian to allow traffic to go to their service.
[09:53] So, that's a good uh demarcation point to start talking about the next thing that I sort of built.
[10:00] And I built it through necessity of Essentially, I began to understand and unravel the requirements more as I went along.
[10:09] One of the architects had this idea to replace the load balancers.
[10:15] At Atlassian, which were enterprise load balancers that had licensing costs, with a open-source cloud-native sort of commodity proxy.
[10:24] And the tech that we chose for that was Envoy proxy.
[10:26] You may be familiar with Envoy proxy.
[10:29] If you're not, then it's very similar to something like Nginx, but perhaps more modern than Nginx.
[10:37] Um you can take a look at its, you know, uh what's what's great about it if you want.
[10:42] You can just read through like why why choose Envoy, blah blah blah.
[10:46] But essentially, we wanted to replace the enterprise load balancers we had, make them self-service, so that devs effectively didn't have to talk to us to go set up their load balancing.
[10:57] So, Envoy has an API that allows you to configure it dynamically.
[11:02] Being able to reload the configuration at run time means that you can deploy a whole bunch of proxies and have them sit there running all the time.
[11:12] And then when someone needs different configuration for their particular uh
[11:16] Service, then they can push out a change through the provisioning task detailed here.
[11:21] And those changes should flow to the proxy somehow.
[11:25] And so now, that's a good time to talk about the Envoy management server that I built, which we called the Envoy control plane.
[11:35] And this was it's essentially quite similar uh to this.
[11:41] Yet again, we used a Fast API app.
[11:44] But this was slightly different actually.
[11:46] Let's go into a little bit of detail here.
[11:47] I'm just going to wing this because I should be able to wing it because I know it quite well.
[11:50] Uh this is actually a I open sourced this this software and I called it Sovereign.
[11:56] You can actually go find that on Bitbucket.
[11:58] It's it's a public repo at least for now.
[12:01] I don't know if that's going to be the case always.
[12:04] But essentially Sovereign runs a Fast API app.
[12:06] And some of the things that it takes in as uh configuration are templates and context.
[12:15] And so the app uh polls.
[12:20] These.
[12:20] Uh, it's obviously got like, uh, say, let's just say this is the configuration.
[12:25] Okay, let's stick.
[12:26] Now, so the templates might be particular resource types.
[12:32] And in Envoy, you've got stuff like clusters, routes, listeners.
[12:39] And let's just leave it at that for the moment.
[12:40] You'd have these kind of templates.
[12:43] And so when this, when this management server loads up, it'll read in these templates in the context and make these available as APIs for the uh proxies.
[12:55] So then you can imagine, let's just say we've got uh an Envoy here.
[13:02] It is going to request these things and Sovereign is going to respond by taking the context, putting it into the templates, and rendering out different content uh as the context changes.
[13:15] Now, where does the context come from?
[13:17] Well, this is part of this management server that is dynamic.
[13:19] So, well, let's just uh.
[13:22] Let's just do a bit of flip around here.
[13:26] Put the context.
[13:27] The context actually comes from this database, but we are requesting it from the broker.
[13:34] So, we're requesting data from the broker and other sources, in fact.
[13:38] Let's just add another source here.
[13:40] Let's just say we have a little S3 bucket with some data, and maybe that data is changing over time.
[13:46] So, we take that data, it's dynamic, we feed that into the templates.
[13:49] The templates have logic that spits out particular Envoy configuration, and then the proxy changes over time.
[13:57] So, what happens is we've got a client that's making a provisioning requests to our broker.
[14:02] The worker is doing some provisioning tasks, and then writing the new data to the database.
[14:10] Then the management server, let's say.
[14:13] Stop this.
[14:15] Let's encapsulate this a little bit.
[14:16] The management server is then polling that data from various places and generating new configuration.
[14:21] That configuration.
[14:23] Hits the proxy, and then it starts doing different stuff.
[14:26] That is essentially the second part of what I built.
[14:28] So, we've got a broker, we've got a management server, we've got the client, we've got the proxy.
[14:33] Uh why did this detach?
[14:35] Anyway, so now we've sort of figured this stuff out.
[14:39] This is all at a very high level.
[14:41] [clears throat]
[14:41] So, we've got this created.
[14:44] Now we can sort of think more about more infrastructure type things.
[14:47] We've got this proxy, but how do we end up with this proxy?
[14:51] How does that actually get provisioned?
[14:53] What is it?
[14:55] Where does it live?
[14:58] Well, let's start with one thing, which is that these proxies, there's many many many of them, as you would expect, and they are provisioned by um they are provisioned by a CloudFormation template.
[15:11] This is an AWS thing that allows you to essentially do infrastructure as code, and it allows you to create resources in in AWS that you would normally create via the console if you were just uh uh let's say.
[15:26] Uh, basic user.
[15:29] So, what kind of stuff do we create in here?
[15:31] Well, if we were to do this stuff from scratch, we'd probably have like a VPC and then we'd have, uh you know, a subnet inside that VPC and maybe we'd have an internet gateway, maybe we'd have uh we'd all security group, maybe we'd have a key pair, maybe an IAM role.
[15:54] Um, oh, of course we need to have the auto scaling group.
[15:58] Of course, that's what's going to be creating these EC2 instances.
[16:05] And well, the auto scaling group needs an AMI, doesn't it?
[16:10] Well, indeed it does need an AMI.
[16:12] IAM role has to be attached to uh must be attached to all these.
[16:18] The key pair goes on on the these.
[16:21] Security group is attached to to the this.
[16:24] Well, it's probably attached to the auto scaling group to be fair.
[16:26] Well, the.
[16:28] EC2 instances would inherit it from the from the ASG, blah blah blah.
[16:33] So, we've got all these like uh blocks of resources and stuff like that.
[16:35] Cool.
[16:37] Let's let's put these up put these up together, blah blah blah.
[16:38] Cool.
[16:41] So, yeah, we've kind of got like a little template going on here and it's creating these proxies in many different regions.
[16:48] Uh we might have we might have like uh an NLB in here, a layer four proxy.
[16:54] Maybe we'd have uh bit of maybe a bit of ACM.
[16:57] Of course, these acronyms might mean nothing to some people, but for people that have used AWS, they would know what these things are.
[17:03] And they know it's not really that complicated.
[17:07] It's like pretty basic building blocks and this is what we created uh say 2,000 proxies, uh something like 13 regions, blah blah blah.
[17:17] Um and we also had a little bit of route 53 records for other stuff.
[17:23] Now, the AMI, it's not really provisioned by the the template.
[17:26] It's more like it's referenced by the template, isn't it?
[17:30] So, that would bring us on to the next piece of this thing that I built, which is, well, we need to produce an AMI.
[17:37] We need to produce a standard image for these proxies, and it's going to include all the important stuff in there.
[17:43] So, how do we create this image for the proxy?
[17:45] Well, in this case, we had uh repository that was using HashiCorp Packer, and uh we had um a Salt Stack uh let's call it configuration.
[17:58] And so, we would use Packer to um let's say we'd have the EC Oh, we'd use the EC2 provisioner.
[18:05] And so, we would create an EC2 in like a dev account.
[18:08] We'd then upload all of our Salt Stack configuration.
[18:12] Salt Stack, by the way, is very similar to Puppet, Ansible, and Chef, in case you're not aware of what those are.
[18:18] It is configuration management tools, and that's a fancy way of saying that I want to run I want to install packages, put files, and run services on a machine in a particular.
[18:30] Way, in a particular order, and it automates that process, makes that process declarative for you.
[18:35] Well, not for you, but it helps you to to make it declarative.
[18:38] So, we created a little um created a little EC2 live running EC2 here.
[18:44] We dump the config on there, we do a provisioning step, and then we take the uh essentially turn this into an image, like shut it down, uh whatever, snapshot it, and turn it into an image.
[18:54] So, that essentially would just produce this uh AMI.
[18:58] Now, what was included in here?
[19:00] Let's let's say we can just uh we can include a few things here.
[19:03] Let's just say we had um we had states for for Envoy.
[19:07] So, like let's say install, configure, uh let's say just install and configure Envoy.
[19:13] Uh logging agents, security, let's say slash hardening, network tuning, containers, tracing.
[19:23] Oh, let's just say let's just say observability agent there.
[19:25] And that can cover I can cover logging, tracing.
[19:30] Metrics.
[19:32] So, that's essentially what's going on here.
[19:35] Produces the AMI, CloudFormation template takes this AMI provisions these EC2s EC2.
[19:40] And they're running with all this stuff.
[19:42] And then when they when they get provisioned, that's something that we forgot here.
[19:46] There's parameters.
[19:48] Parameters, bump that up, bump this up, make it neat.
[19:52] So, we've got the parameters and these at runtime would pass in secrets and keys and blah blah blah.
[20:00] And then these these proxies would grab the resources, be configured, and then they would be running and accepting traffic.
[20:07] Boom, that's it.
[20:09] Everything's done, working.
[20:11] This was essentially the first two years of working at Lyft.
[20:14] So, now when a developer says, "I want to run my service and I want it to be I want it to be accessible on the internet with all the fancy bells and whistles and routing and advanced stuff."
[20:25] We'd say, "Yes, no problem."
[20:28] Let me just get that provisioned for you.
[20:30] We.
[20:31] send off the provisioning task, we write
[20:33] something to the database, we tell them
[20:34] it's ready, then the management server,
[20:37] it's the
[20:39] broker says, "What's the current state
[20:42] of things?" Takes that data, plus other
[20:44] data, puts it into the templates,
[20:46] creates resources out of those
[20:48] templates, gives them to Envoy on when
[20:50] it requests them. And this was all
[20:53] pre-provisioned.
[20:55] That's long-lived infrastructure with
[20:57] CloudFormation. And the CloudFormation
[20:58] is relying on an AMI that it can use to
[21:01] provision those images, those machines.
[21:03] So, yeah, that is probably the first 24
[21:06] months. So, what was next after this?
[21:08] So, this was the foundation of our team,
[21:10] essentially the product that we were
[21:12] going forward with, um which is uh
[21:14] centralized load balancing managed by
[21:17] our team, and all of the features that
[21:20] we provided to our customers would live
[21:22] in logic defined in these templates.
[21:25] We've now laid the foundation for the
[21:27] team. We've got proxy infrastructure
[21:29] that's reacting dynamically to services
[21:32] that are being deployed with different
[21:34] configurations over time. What was next
[21:36] after that point? The big thing after
[21:38] that was taking some of the larger
[21:40] products and making it possible for them
[21:43] to use this platform component. That was
[21:46] one big part, and the second big part
[21:48] was migrating all of the microservices
[21:51] within Atlassian to use this. And that
[21:53] was relatively easier because we could
[21:56] enforce that through the platform.
[21:58] Essentially, what that means is that the
[22:00] platform was previously providing very
[22:03] basic load balancing to every service.
[22:05] And they forced a switch to where you
[22:08] could no longer expose your service
[22:10] publicly through their load balancer,
[22:12] which is too basic, and you had to go
[22:14] through our centralized load balancing
[22:16] infrastructure and to explicitly
[22:18] configure it as a way of signaling your
[22:21] intention for that service to be
[22:23] publicly accessible. Whereas previously,
[22:26] it could have just been maybe accidental
[22:28] that your service was public and not
[22:30] very well protected. So, that was the
[22:32] big major push. We got products like
[22:35] Jira, Confluence, Bitbucket, Status
[22:38] Page, and many others behind this edge
[22:41] infrastructure. And then, what was after
[22:43] that? Well, now we can sort of talk more
[22:45] about Let's say we can talk more about
[22:48] the uh the Envoy-based product that we
[22:51] had here. So, this particular thing,
[22:54] we've got this groundwork of being able
[22:57] to take basic inputs from a a developer
[23:00] and to turn that into templated
[23:03] configuration. Now, Envoy has a lot of
[23:06] configuration. It has a lot of stuff you
[23:09] can configure. Let's just look at the
[23:11] routes, for example. Let's look at the
[23:13] virtual host, for example. You can
[23:15] configure what domains to accept traffic
[23:17] on. Pretty basic. You can do routing.
[23:19] Sort of basic, but once you delve into
[23:21] how you can do this, it gets pretty
[23:23] complicated pretty quickly. You can
[23:25] match on different things. You can route
[23:27] it in different ways. You can do direct
[23:29] responses. do redirects, blah, blah,
[23:31] blah. You can add and remove headers. I
[23:34] guess I guess you could say that's
[23:35] pretty standard, but you can also
[23:37] choose, for example, when you're
[23:39] configuring a route action, you can also
[23:40] choose to send to any cluster that's on
[23:44] the proxy. So, then if I have a thousand
[23:47] devs, or a thousand services, and they
[23:50] each have their own cluster, and any
[23:52] route can send to any cluster, well, it
[23:54] sort of brings up this point of well,
[23:56] this
[23:57] data here needs to be validating that
[24:01] and abstracting that, and so on and so
[24:05] forth. So, there was definitely a
[24:07] concentration of a lot of the
[24:09] development work around this logic here,
[24:11] making sure it was validated here in
[24:13] terms of the parameters were validated
[24:16] such that when those parameters were run
[24:18] through the logic in these in these
[24:21] resources, that it would produce valid
[24:23] resources. Pretty standard, I suppose
[24:25] you could say. I don't know. Maybe I do
[24:27] feel like I have the curse of knowledge.
[24:29] Um and that this stuff seems easier to
[24:32] me now because I I've done so much with
[24:34] it. Uh but there's a lot. There's a lot
[24:36] in here. And if we go into, uh for
[24:38] example, extensions, there's a lot of
[24:40] extensions that can be applied to a
[24:43] listener or a or a cluster. For example,
[24:46] you might have, uh where is it? You've
[24:48] got, uh network filters here. You've got
[24:51] all kinds of network filters. And a big
[24:53] one that we obviously used was a HTTP
[24:56] connection manager, where you could
[24:58] configure routing and how to handle
[25:01] proxies and web sockets and all this
[25:04] stuff. And then, if we go a little bit
[25:06] before that, there's also things like
[25:08] external processing and external
[25:11] authorization. And this sort of brings
[25:13] us to Oh, let's say something that
[25:16] happened next. So, I did briefly mention
[25:18] that some of the big parts after
[25:19] building this was to migrate big
[25:21] products onto Let's assume that's all
[25:23] finished. It took It took some time. It
[25:25] took a couple years because there were
[25:27] many features that needed to be built
[25:29] out here and and wherever else in order
[25:32] to support the larger products and their
[25:34] special cases to work on
[25:37] what's effectively a generic
[25:39] multi-tenanted platform. So, let's just
[25:41] assume that they're all migrated. Then,
[25:42] we have more features that we want to
[25:45] add. I did sort of allude to like we
[25:47] have this We have this groundwork. We
[25:49] have this dynamic configuration. What
[25:52] I'm trying to say is that we we created
[25:54] opportunity. We created opportunity to
[25:57] centralize logic and to handle concerns
[26:01] early in the chain of requests. What I
[26:04] mean by that is a customer, let's just
[26:07] make a smiley customer. And customer is
[26:10] someone that's using our cloud products
[26:12] or Atlassian cloud products. They are
[26:14] hitting the
[26:16] Let's just say they're hitting an NLB
[26:17] first and that's then being proxied to
[26:19] these boys. Yes? If we can deal with the
[26:22] problems here before they reach a
[26:24] service,
[26:26] let's say and let's give it a square.
[26:28] Let's call it a back-end service, you
[26:30] know. So, the requests are flowing in
[26:31] from the customer to the proxies and to
[26:34] the Pretty standard stuff. If we can
[26:35] deal with certain concerns here before
[26:39] it reaches here, we save a lot of time,
[26:41] we save some money, which is and it
[26:44] saves the customer time. It's great for
[26:45] everyone, really. Um
[26:47] and one of those things Now, this is
[26:49] where the the diagram becomes
[26:51] complicated, so let's move off to the
[26:53] side. Let's just copy a few of these.
[26:55] Let's grab
[26:56] three things
[26:58] move over to the side. We've got the
[26:59] customer talking to the proxy and the
[27:01] proxy is talking to back end. Of course,
[27:04] the request comes back up and back out
[27:06] to the customer. Fine and dandy. Yes,
[27:08] this is a this is a proxy. Whatever,
[27:10] there's no surprises here. Now, without
[27:12] with with the products that Elysium
[27:15] runs, there's all kinds of stuff that
[27:17] needs to happen like authentication for
[27:19] example or authorization or
[27:23] DDoS protection or rate limiting or
[27:26] access logs. All this kinds of stuff
[27:28] that needs to happen and it's just turns
[27:31] out that we can deal with them here
[27:32] instead of on a bazillion bazillion back
[27:37] end services. Just imagine there are a
[27:40] bazillion bazillion of these. Just
[27:43] zillions upon zillions. See Daisy. Just
[27:47] zillions and zillions. They're like
[27:48] gazillion. Now, can you imagine if a
[27:50] thousand dev teams needed to deal with
[27:53] all this stuff plus more on their own
[27:56] service? It would be a tremendous waste
[27:59] of money for the company. It would slow
[28:01] down features. The customer wouldn't get
[28:03] their features when they need them and
[28:05] stuff is already hard enough to deliver
[28:07] as it is. Thus, the platform and
[28:10] centralized management of resources and
[28:12] centralized
[28:13] implementation of these features. So,
[28:16] how how were some of these things
[28:17] implemented? Well, DDoS protection was
[28:19] really provided by
[28:22] CloudFront. That was
[28:24] that was
[28:25] spearheaded by a colleague of mine who
[28:27] is very smart and conscientious. And
[28:30] essentially, let's make this a bit more
[28:33] accurate. Let's just say let's get rid
[28:34] of these. There's an NLB here. Oh, blah
[28:37] blah blah blah blah. And of course, it's
[28:39] two-way. So, that's one way that we can
[28:41] take care of that concern for these back
[28:43] end services. Great, we've solved solved
[28:45] the concern for that. Fantastic. These
[28:47] others, well, access logs, what we can
[28:50] do is something like we can use these
[28:53] network filters. Yes, we use the network
[28:55] filters. For example, in the HCM, we
[28:58] have
[29:03] Where are the access logs? Access log.
[29:05] Now, remember, all of this configuration
[29:08] is dynamic. It's all dynamic and it is
[29:11] created by templates which abstract away
[29:15] the resource configuration from the
[29:17] developer who wants to configure it.
[29:19] They provide simple parameters. Those
[29:21] parameters are then validated and then
[29:22] they are fed into the template as
[29:24] context so that we produce the correct
[29:27] template. That means that
[29:29] they send us a little bit of JSON and we
[29:32] set up this whole thing for them with
[29:34] all the access logging and blah blah
[29:36] blah, whatever. So, that is done, in
[29:38] fact, inside the proxy, natively.
[29:41] Fantastic. Some of these things,
[29:42] however, a little bit more complicated.
[29:44] These things, we need to use a sidecar
[29:47] model where Envoy is talking out the
[29:48] side and then these are their own
[29:51] services running locally on the on the
[29:53] proxy. So, these would be like
[29:56] containers, essentially. We've got this
[29:58] sidecar model and those sidecars, some
[30:01] of them were contributed by other teams
[30:03] and some of them were created by me and
[30:05] our team. The authentication and the
[30:07] authentication sidecar was created by
[30:09] me, written, of course, in the Lord's
[30:12] language, Rust. Authorization was done
[30:15] by another team and rate limiting was
[30:16] done by another team. And so, they were
[30:18] able to contribute these sidecars,
[30:20] which, by the way, were set up and
[30:23] were downloaded and configured onto the
[30:26] AMI by this provisioning AMI
[30:28] provisioning flow. Great. So, now we
[30:31] have a programmable proxy with sidecars
[30:35] that have their own separate logic from
[30:37] the proxy and they, too, can actually
[30:40] receive configuration, which is dynamic
[30:43] over the wire locally and
[30:46] and make it even more program. So, we're
[30:48] solving all these concerns before they
[30:50] hit these
[30:51] these back ends and in very very little
[30:54] time. So, that was
[30:55] essentially that is some of the stuff I
[30:58] worked on after migrations and blah blah
[31:00] blah. Yay. What I do after that? With
[31:03] this big blob rid of this mess. So, then
[31:05] we had some non-technical requirements
[31:08] come through. More compliance
[31:11] and things like that. And that effort
[31:13] was very tedious and boring for me
[31:15] personally. It didn't involve building
[31:17] new stuff. It involved taking all of
[31:19] this, making sure that it was compliance
[31:22] for in certain ways. Very bored boring
[31:26] checklist ticking work. Blah blah blah.
[31:29] So, I said earlier that I would also go
[31:30] over some of the non-technical things
[31:32] that I had to go through while working
[31:34] at Atlassian. Obviously, all of that
[31:35] stuff is sort of high-level technical
[31:37] stuff that I just showed. What was some
[31:40] of the other stuff that I went through
[31:42] during my eight-year slog at Atlassian?
[31:45] The first few things to come that come
[31:47] to mind is that I have grown
[31:50] tremendously in my diplomacy skills,
[31:54] conflict avoidance, probably conflict
[31:57] resolution as well. Being able to
[31:59] persuade, propose ideas, being able to
[32:02] teach, educate, and mentor. These are
[32:05] the non-technical things that you
[32:06] probably don't hear a lot about. But
[32:09] after Another thing is that the ability
[32:11] to maintain things, maintain software
[32:15] and systems, to see where the cracks
[32:17] show up and to build things so that
[32:21] those cracks don't show up as or at
[32:23] least to make them show up late as
[32:25] possible. That's definitely something
[32:26] that I picked up. Let's just talk about
[32:28] that maintenance for a sec. I noticed
[32:30] over the eight years that I was there,
[32:32] when I built these apps, these
[32:35] services, that there's obviously that at
[32:37] the very start there's the requirement
[32:40] to onboard people and write
[32:41] documentation and train people so that
[32:44] they understand how things work, know
[32:46] how to contribute to them, and debug
[32:48] them. So that when they become when they
[32:50] go on call, they know where to look,
[32:52] what could go wrong, where do things
[32:54] break essentially. So, you know, that's
[32:57] whether that's knowing what kind of what
[33:00] particular log messages mean, what sort
[33:02] of metrics to check when something is
[33:05] going wrong and what those metrics could
[33:07] allude to, how to resolve those
[33:10] um you know, particular expected
[33:12] problems if they're not automated away.
[33:14] Um and this could be like, you know,
[33:16] Amazon could have an outage and the
[33:18] database isn't access for example. What
[33:20] do you do in that case? What if SQS
[33:22] stops working and you can't do any
[33:24] provisioning tasks? What how what impact
[33:26] does that have on the services that need
[33:28] to provision the resources? And how do
[33:29] you resolve um
[33:31] What happens if an if a proxy receives
[33:33] bad configuration? What if it receives
[33:35] configuration that's valid, but that
[33:37] destroys the traffic that's flowing
[33:39] through? How do you pick up on those?
[33:41] What do you check, etc. etc. So there's
[33:43] obviously a lot of that at the start
[33:45] when you build something. There's a lot
[33:46] of that at the start. But the thing
[33:48] that's more difficult is over time
[33:50] people come and go. People get hired.
[33:53] People get people leave for other jobs
[33:55] and whatnot. And so you get you have to
[33:57] do that onboarding again obviously. But
[33:59] you should have more people that are
[34:01] able to do that onboarding collectively.
[34:02] But then you sort of bring in new
[34:04] opinions. People look at an existing
[34:06] codebase and they want to change things.
[34:08] They want to make it better, and so on
[34:10] and so forth. And so they do that. And
[34:12] change ends There's I I suppose there's
[34:15] this concept of churn in the codebase.
[34:18] The area that churns, it's sort of it
[34:20] becomes predictable where all the churn
[34:22] is going to be
[34:24] at a certain stage. And once you notice
[34:26] that there is some churn, it's sort of a
[34:29] a smell. It is it's an indication that
[34:32] that part of the service or project is
[34:35] going to keep increasing in size or
[34:38] complexity. And something there needs to
[34:41] happen. Something needs to be done to
[34:43] avoid that mess. It's just just how
[34:45] software goes, I suppose. It'll be
[34:47] interesting with all these
[34:49] vibe coded apps and AI assisted apps to
[34:52] see how we handle that. When we have
[34:54] people that are not really familiar with
[34:56] what they've created, and the
[34:58] maintenance burdens appear. They don't
[35:00] appear at the beginning. There's just
[35:01] not enough going through. It hasn't been
[35:03] around for long enough. There hasn't
[35:05] been enough changes. Building something
[35:06] is easy. Changing it and making sure
[35:08] that it you can still change it over
[35:09] time is difficult. Because as you change
[35:12] things, it slowly becomes harder to
[35:14] change. Things start to get coupled, and
[35:16] all of a sudden when you change
[35:18] something in one area, it affects
[35:19] another, and you have to deal with the
[35:20] task of detangling something. And you
[35:22] might be able to find these areas quite
[35:24] quickly, get an LLM to perform the
[35:26] detangling for you. I think that's If we
[35:28] can do that, that's fantastic. But I
[35:30] don't want to be too optimistic just in
[35:32] case. So, there's that on That's my
[35:35] opinion on the maintenance side of
[35:36] things. The next thing I want to talk
[35:38] about is when I mentioned diplomacy,
[35:41] what I'm really trying to say is that I
[35:43] was exposed to different types of
[35:46] managers and colleagues over time. And
[35:48] everyone has different personalities and
[35:50] styles of working. And because I was
[35:52] exposed to so many different types, I
[35:54] experienced conflicts with certain
[35:56] people. And even though I had conflicts,
[35:59] there's still people that I respect.
[36:00] It's just something that happens when
[36:03] you when your personality doesn't mix
[36:05] with their personality. And that's just
[36:06] something that's a bit inevitable. And I
[36:08] think that the only thing you can really
[36:10] do in those situations is to try to have
[36:13] the self-awareness and the awareness of
[36:15] the other person and the, I suppose,
[36:18] understanding of psychology and and how
[36:20] people work to an extent, so that you
[36:23] can be responsible for that difference
[36:25] and the potential for conflict, and to
[36:28] handle it effective to to anticipate the
[36:31] conflict that's going to arise and and
[36:33] to do something to make the relationship
[36:36] work. And maybe it's impossible. I don't
[36:37] know. But that was definitely a source
[36:39] of great stress and at [clears throat]
[36:41] times it affected my performance. And so
[36:45] I do think that because it affected my
[36:48] performance that I took it quite
[36:50] seriously and I learned and changed as a
[36:53] result. So the next time that those
[36:55] situations come around, I do firmly
[36:58] believe that I'll be able to handle them
[37:00] quite a lot better. And then some of the
[37:02] other things, in fact one of the things
[37:04] that I found quite challenging was
[37:08] mentoring. And so I find it easy to help
[37:11] people to point out areas where they
[37:15] need understanding and to deliver that
[37:18] understanding to them, to break down
[37:20] complex things into simple terms so that
[37:23] they can build a mental model of the
[37:26] system that they're working on. I have
[37:27] that ability. I'm quite good at that.
[37:29] But mentoring is distinct from that. I
[37:31] had an intern in the last year and I
[37:34] will first say that the result of their
[37:36] internship was that they got the highest
[37:38] rating possible and it essentially
[37:41] guarantees
[37:42] an offer to work at a last year. The
[37:44] project that they worked on was very
[37:46] impressive and how they approached it
[37:49] and and built it was very impressive.
[37:51] And so that's why they got that
[37:52] excellent rating. What I found
[37:54] personally difficult was striking the
[37:57] balance between It was essentially
[38:00] striking the balance between how much
[38:02] time I give to the mentee and what that
[38:06] time would consist of, whether it's, you
[38:08] know, I didn't I don't want to give them
[38:09] answers to problems, but I don't want
[38:11] them to get so stuck that they become
[38:13] frustrated. I have no idea if I reached
[38:16] that balance, but I suppose the results
[38:18] speak for themselves. I I but I I don't
[38:20] I don't I'm not sure if I can attribute
[38:22] the results to me necessarily. The
[38:24] intern was helped by some of my other
[38:26] colleagues when in areas that I'm much
[38:28] weaker in. So, they effectively got
[38:31] subject matter experts in a few
[38:34] different areas to contribute to their
[38:36] success. But then they did the majority
[38:38] of the legwork to actually build the
[38:41] thing and to test it and to make design
[38:44] decisions and stuff like that. And it
[38:45] was very successful. But I still have
[38:46] this lingering impression of feeling
[38:48] that mentoring is difficult for me and
[38:51] that I I don't have um a good way of um
[38:55] figuring that out because I've never
[38:56] been mentored myself. So, I don't really
[38:58] know what to expect and what they do.
[39:01] But I want to emphasize that that's a
[39:03] very specific type of mentoring that I'm
[39:05] not too sure about. Whereas training my
[39:07] colleagues, getting them to understand,
[39:09] working, you know, working through
[39:11] problems with my colleagues, that was
[39:12] that was essentially my bread and butter
[39:14] during the last half of my employment.
[39:16] You know, jumping on uh uh call and
[39:19] going through stuff. Feedback that I got
[39:21] from my colleagues all the time was that
[39:22] I was always available to help and that
[39:24] I could boil down hard topics into
[39:26] something that was understandable, which
[39:29] I'm pretty proud of. And I've been
[39:31] yapping for a while. I think that covers
[39:32] a quite a lot. If I remember more, I'll
[39:34] probably just make a second video. Um
[39:36] maybe maybe if people are interested, I
[39:38] could actually go through and build some
[39:40] of these things. I could actually go
[39:41] through and build some of these things
[39:42] from scratch on stream or just a video
[39:45] uploaded to kind of show, I guess,
[39:47] essentially what I made and maybe
[39:48] recreate and sharpen my skills a little
[39:50] bit more. Um maybe. I've got a lot of
[39:52] stuff on my to-do list, so maybe maybe
[39:54] not. It depends on the demand. Anyway,
[39:55] I'm going to cut the video from here. If
[39:57] you listened all the way through or to
[39:59] portions, then thank you very much. I
[40:01] hope it was interesting and enlightening
[40:03] and whatever else. I'll catch you
[40:05] around.