Saturday, December 28, 2019

Reading Log: Righting (sic) Software. Part 2

Holy fetch. Leave it to Nate Cunningham to be lurking on my dead-for-several-years blog and comment within a few days. Truly impressive! Thanks Nate. You're the man.

Here comes part two of the reading log. I'm experimenting with writing these things a few days after I do the reading. This forces me to consult my notes and revisit some of my thoughts. Perhaps it will help with retention.

~~~

Juval is about to sell us his "Method", which is his specific method for planning and designing software projects. I'm naturally wary of anybody selling me a "method" to do anything, especially one that is "Completely unique to me and is guaranteed to bring you riches and success". Feels like something from my spam folder. That being said, everything I've read so far has been good.

He says that Method = System Design (the software...) + Project Design. This is actually sort of a big deal. Only that I've never considered project design to be anywhere near as important as system design. He asserts that we all suck at delivering working software on time because:

  • Project design is strictly harder than system design
  • We study and practice system design, and none of us have ever read a book about project design
I don't have data about his first assertion, but I can tell you that I'm a grown-man-who-thinks-he's-good-at-this-stuff and I've never even considered project design as being a concept I should try to be good at. I guess this just wasn't something that I thought belonged to my profession. It's like waking up after four years and being like "Oh man, I was supposed to file taxes every year?". 

 A few more nuggets:
  • We need to validate the high-level system design early. If we spend three months building something before we figure out that the architecture won't actually work, we're pretty boned. 
    • How do we do that? As part of the project design, you should make sure that work is done early that will reveal whether or not this architecture is going to fly. I'm sure he'll talk about this more, but this feels a lot like some of the Agile ideas -- Build a very thin vertical slice to prove it'll work.
  • "Designs" are really a filled out decision tree. Specifically, any design is a leaf on a decision tree. See https://en.wikipedia.org/wiki/Decision_tree for the details of that. I had never thought about it this way. This analogy works really well for computer science nerds. We use trees a lot. But we also spend a lot of time simplifying, or pruning, trees. Trees get big fast. But if we know certain decisions don't work, we can prune off the entire subtree that follows that decision. 
  • Designing a project or system should be a time-boxed endeavor. Juval basically says "You should design in three to five days. Any more and you'll just be adding unnecessary fluff". 
So, in conclusion: You're going to produce a decision tree for your system design in about four days. And then you're going to spend the next four days producing a decision tree for your project design

Thursday, December 26, 2019

Reading Log: Righting (sic) Software. Part 1

Hey all. I'm trying something new for a while. I'm going to be reading two books re: software engineering in the next few months. I'm going to write down my thoughts about each part as I read it. This is mostly an exercise to help me solidify understanding of what I'm reading.

As an aside, one of my dreams is to run an online book club about software engineering. It'd be a Twitch stream where we discuss a reading assignment once a week. We'd pick a book together and assign a few chapters. People who read / contribute get special twitch emotes or something.

For what it's worth, it's 4 AM and I'm awake holding my daughter. Who is probably awake because her teeth hurt. There's an outside chance she's just awake to be a jerk. We're watching LBB on Netflix. That stuff is baby crack.

Merry Christmas, btw.

K, part one of Righting Software. Just the intro so far.

~~~
First of all, this dude pulls no punches. He's like "Everybody sucks at software. Except for me. So I guess I'll teach you how to deliver working software on time, on budget, and defect free". This is in stark contrast to my usual philosophy. My current philosophy, informed mostly by Ron Jeffries "The Nature of Software Development", is that everybody sucks terribly at software. So, instead of like, trying to estimate costs better, or plan better, or whatever better, we should just deliver one feature at a time, minimizing complexity and maximizing quality.

Ron's approach reduces the need for up front planning. But it does ignore some realities: Particularly "How are we going to pay for this? How many devs do I need to budget for? When can I tell the boss we'll recoup some of this money?". The "Screw everything we're agile" plan works super well when you have a fixed budget and loose timeline. It has lots of merit. But Juval Lowy (except with two dots over that 'o') points out that reality frequently requires that stuff. So, uh, I guess we're going to plan stuff.

This comes at a good time for me, though. Google is a big company and they expect me to plan. My manager isn't really buying the "it will be done when it's done" approach to my code. He also keeps wanting estimates about when this stuff will be done. Seriously though. I've never successfully estimated anything in my life.

It occurs to me that I have never delivered any meaningful software on time or on budget. I've written some fantastic software. I honestly feel really good about myself as an engineer. My quality is generally really high. My code is legit. I don't feel really bad about missing our "deadlines". Like, they were arbitrary, right? And our great working software continues to win the market. But yeah, my project management has historically sucked.

Juval says that a software architect is responsible for designing two different things. One is the software. That's what you and I always do. This component and this module and that framework etc. This is what I think of when I think about "software architect."

The second thing an architect is responsible for designing is the plan to build the dang thing. Or rather, the project design. "If there's no reasonable way to build it, why design the software part of it?". So apparently projects require design. That's the planning, estimation of costs, etc. I have no idea what this looks like yet. This is a massive change from my current modus operandi.

Basically: I approach life with the base assumption that "planning is futile. We cannot possibly know how long this will take. So we'll completely finish the smallest, most important part you can define. We'll just keep doing that until we run out of budget. I hope that you have something really great at the end :)"

Juval says that's bullcrap and that we should be effectively planning projects. He posits that "applying sound engineering practices from other (real) engineering disciplines" makes this easy. Given that literally all of us suck terribly at this, I'm not yet convinced. But I'm excited to see what his process is.

I'm committed to trying it out. I'll keep writing about it as I read. I've got a new small project starting up at work, and I'm one hundred percent going to run this method on it. I'll practice all the steps and let you know how it goes.

~~~

Good news: My daughter finally decided it was time for bed. That's legit. I'm going to bed too. I'll see y'all soon.

Please drop me a comment if you read this thing. Like, I don't actually expect visitors. But let me know if you're here because that will definitely make me more likely to continue this project.


Monday, July 08, 2019

The Jist

Hey all, happy Sunday!

Tomorrow is my first day working at Google. This has been a dream of mine for a long time and I'm very excited about it. I'm grateful for the support and coaching of my wife. She has taught me how to be brave and pursue the things I want in life. I'm stoked to give this whole thing a shot.

Given that I start tomorrow, and that I have no idea what the intellectual property rules are going to be like, I wanted to vomit out the jist of the software dev book I've been thinking about writing.

I really don't expect it to be good. It's just a rehash of the things I've learned in the first five years of doing this professionally. My desire is that it can stand as a replacement of me to my old team at Xima. It's an opinionated approach to how you should write software. Ready? Let's do this thing.



You suck at programming. And it's not your fault, really. But you suck at it. And you need to understand how you suck at it so you can compensate for your inherent suckiness. One of the core values we're going to try to live by is "Don't Suck".

Why do you suck at programming? There are three reasons.

1. You suck at understanding the rest of your code. 

Your brain can only hold a finite amount of data before it starts losing things. You can think of it as a stack. You can pop stuff on there repeatedly, but you're going to start losing things off the back of it as soon as you pop too much. This is just how you're built. It's not your fault. There's a decent amount of research into this. You can only handle so much cognitive load before things start sucking. Your physical condition affects this too. Less sleep leads to less cognitive capacity. Some people will have less cognitive capacity (smaller stack size) because of a traumatic brain injury or a urinary tract infection. You might be amazing and have a relatively large stack size. But suffice it to say: There exists a program that is too large for you to understand all at once. You don't have enough working memory to keep it all available.

Essentially: You are limited in how much you can understand of your software at one point. You have a moving window of comprehension that precludes you from writing your software well. We're going to refer to this problem as "You suck at understanding the rest of your code". In this usage, "the rest of your code" means all the code that is not currently loaded into your moving window of comprehension.

2. You suck at knowing what the code you wrote does. 

Thought experiment. Indulge me on this one. You're in a high stakes programming situation. There's a beaver dam that will be blown to shreds in five minutes unless you can write a specific method correctly. You know what the input looks like, and you know what the output should look like. You have to read data in from a flat text tile, and write out a new file with the solution in it.

You write code for four minutes straight. This is in your wheelhouse. These are not things you've never done before.

At five minutes it's time to see if you saved the beavers or if they all died and became hats. How confident are you that your dam survived? Based on your normal work history, what is the probability that they aren't hats?

If you're anything like me it's pretty fetching low. Nobody doesn't have this problem. No matter how good we think we are, we always make mistakes. Or maybe we don't make mistakes but our environment is janky. We rarely understand the real details of what we're asking the computer to do for us. We're frequently surprised.

I hope that an inspection of your history will convince you that this is accurate. I have so little confidence in the code that I write. Fun fact: I have even less confidence in the code that you write. The process of translating what we want the computer to do into the language that it understands is incredibly leaky. Even though you wrote the code, you just don't have high certainty about what it does. Really.

3. You suck at knowing what your code should do.

You're going to write software and then it is going to execute. It's going to execute in foreign environments with user-generated input. There is virtually no way that you can predict or test for all of these variables.

That software, assuming it works at all, is going to give users some level of satisfaction. Based on that satisfaction they are going to award you some amount of money. In this scenario, both satisfaction and awarded money might be less than zero. Lol.

How do you maximize satisfaction? How do you minimize disasters based on environmental things you don't control? One solution is to never give your software to users. That provides a very solid lower-bound for satisfaction and requests for refunds. Unfortunately, it also provides a frustratingly immutable upper-bound as well.

Here's the real kicker: You have no idea what your users want. And you have no idea how your software is going to behave when it's in the real world. Will it crash given very specific input? Will it offend your customers? Will it be a smash hit? You're a programmer. Who fetching knows? Not you.

~~~~

So, you suck at writing software. My condolences.

Let's briefly talk about what humanity has learned about overcoming these problems. Please note that we can't completely negate them. They will always be with us. But we can use strategies to make them less painful for us. At the end of the day, you're never going to be a good programmer that doesn't have these weaknesses. You're just going to put practices in place that negate some of their suckiness.

Problem 1 was that you don't understand the rest of your code. There are a few solutions to this.

First: You minimize the cognitive cost of understanding any one unit of code. As you write a new {module, class, method}, you make sure you write it cleanly a la Uncle Bob. Except without the racism. You name things well. You refactor for clarity. In fact, you make clarity the most important freakin' thing about that code. That's all you actually care about. It does the job, and it's incredibly clear. By reducing the cognitive cost you allow yourself and others to fit more code in their brains at once. This leads to fewer window errors.

Second: You leverage abstraction appropriately. You know what abstraction is? Abstraction is "only caring about the parts of something that you really need to care about". Let me give you an example.

Suppose your code uses a Myles. Myles, in the real world, is a person with glasses, hair, and an incredibly diverse array of bacteria in his gut. But representing that in code would be very difficult. What your code cares about is Myles' ability to writeCode().

So your class DevTeam has a Set of Myles in it. But that's just stupid, because your DevTeam does not need to know about Myles' digestive system. So you make Myles implement an interface called CanFetchingWriteCode. And now your DevTeam has a Set of CanFetchingWriteCode. 

You've taken Myles, a very complicated idea, and abstracted it down into what you really care about. Ignore the other parts. They are neat, but not important to your DevTeam.

Third thing here: Write modular code. Reduce coupling. Specifically: Make sure your code in DevTeam knows as little as reasonably possible about your code over in CafeteriaLine. If understanding DevTeam requires understanding CafeteriaLine, your cognitive cost here goes through the roof. Do what you can to decouple them.

~~~~~

Problem 2 is that you don't really know what your code does until it executes. This one is stupidly simple to solve. You just execute your code all the dang time.

Write automated tests that execute the code you just wrote. That's it. That's the whole tweet. Just write tests. Are you thinking about committing some code to the repo? You should execute it first. And the only responsible way to execute it is to wrap a freakin' test around it.

"But Chris, the code I just wrote is resistant to testing!" Well dang, you just wrote code that sucks.

But in a more generous tone; Take whatever "change" that you just made to the existing code and isolate it into something that can be tested. And then test that!

The second solution to this problem is to run your full software frequently. This would likely involve releasing it to customers far more frequently that you're used to.

"But Chris, I can't release this software! It will break something!". You're absolutely right. Your job is to reduce the pain of failure. Here's something that you should really just accept: You're going to break things. I don't care how much QA you have. I don't care how carefully you test. If you're releasing software you are going to freaking break things.

Given that breakages are going to occur, let's just accept that they will happen and let's do what we can to make those breakages less costly. This is called harm reduction. It's the same principle behind needle exchanges, btw. We're going to release our new software to a small percentage of customers first. We're going to have a way to roll customers back to a safe version without a disruption of service. We're going to be able to toggle on and off new features at runtime to get things into a working state.

Can we talk about needle exchanges? The idea is that people are going to do drugs. If they don't have clean needles some percentage of them will use dirty needles and get Hep C or something and cost the state more money. In order to reduce the money the state spends on Hep C, the state sets up a needle exchange booth where people can trade dirty needles for new needles. This reduces the incidence of Hep C and saves money (and lives, maybe). It also creates a safe place for people using drugs to get in contact with people who could offer help. I'm a big proponent of needle exchanges.

There are a lot of less-progressive voices that think needle exchanges are stupid. They dislike the idea of helping anyone do drugs. Offering needles equates to assisting someone in doing drugs, and that feels wrong. We should not be promoting this behavior.

I'ma punch that straw man down, if you'll allow me. The drugs are going to happen either way. That is not something we can stop. Would that we could. But we just can't Nemo. We can accept this truth and try to reduce harm, or we can complain about it and let people get Hep C and then pay for their medical bills. Harm reduction is a cool idea.

We're going to get our software out to the real world as soon as possible and we're going to gather data about how it behaves. We are going to release bad software. But that was going to happen anyways! The key here is to release software that is carefully bad. Specifically: we're going to focus on reducing our mean time to recovery instead of reducing our number of incidents. Releasing 100 bugs and recovering from all of them in 10 seconds each (1000 seconds of downtime) is so much better than releasing exactly one bug with 5000 seconds of downtime.

We're going to stop trying to never release a bug. But we're going to make the cost of bugs much lower.

~~~~

Problem 3 is that you don't know what your code should do. Or, you don't know what the customer wants. Or, you don't know what the market is going to do in three weeks so you might be building the wrong thing anyways.

First solution here is to release frequently. Yeah, you know how frequently you're thinking of releasing? Consider releasing faster than that.

Release and get feedback. Keep your code in an always deployable state.

How do you keep your code in a deployable state? You use trunk based development (aka continuous integration) and you take measures to commit safe code. That means you put risky code behind run-time toggler (look up the article(s) about feature toggles on Martin Fowler's site already...) and you protect your butt with a nice deployment pipeline.

When you're tempted to put your code in a non-deployable state, ask yourself: "Am I willing to throw this code away if priority changes in one week?" If the answer is no, then take measures to maintain releasability.

~~~~

K, that was a fast push through the jist of it. I hope that doesn't suck. Good luck and have fun!

Please hit me up with feedback in the comments section. Thanks.