In this episode, I talk about a recent software project and how I've used many of the processes I have introduced in prior podcasts. I talk about how I applied them. And how they fit together to allow me to develop software with confidence - minimizing risk and maximizing the ability to adapt.
Or listen at:
Published: Wed, 09 Dec 2020 18:37:18 GMT
Links
The Project source code on GitHub
Transcript
[00:00:35] Hello and welcome back to the Better ROI from Software Development podcast. A podcast in which I try to provide advice on how to get the best from your software development.
[00:00:46] In this episode, I want to take a look at some of the practices I recently applied to a project I've been working on for myself - just as an illustration of tying together some of the things I've talked about in previous podcasts. And to give some sort of idea of how they all come together.
[00:01:05] This is going to be based on my software survey, I've talked about this previously. I've also written articles about it.
[00:01:12] It's a software product that I was using to try and establish UK executives attitudes towards custom software development.
[00:01:23] I used this as an excuse for me to build a new system using new technologies that I've not worked with previously. So two birds, one stone for me personally, both in terms of a learning experience, working with new technologies. And to try and get a feel for the general market in terms of those attitudes within the UK executives.
[00:01:46] Now, while, unfortunately, the survey itself didn't really amount to anything - I struggled to get any respondents (and that's obviously a completely separate subject. You can go back to Episode 61, where I talk about that a little bit).
[00:01:58] But I wanted to talk about what I did from a technological basis - and tie that back to subjects I've covered in past episodes.
[00:02:08] Even though this software product was effectively just for me and it wasn't being developed as part of a team, many of the modern practices that I've talked about through out the episodes are still key to getting the job done.
[00:02:21] Well, let's start having a look at which ones we start with.
[00:02:28] First off, before I do anything else, I want to make sure that I'm putting whatever software I write into source control.
[00:02:36] I described source control in Episode 18.
[00:02:37] Source control is a means of being able to keep track of versions of the software code as I write it. It's a way of making sure that should my machine die, should I lose my machine, I've got another source of it - in this case in the cloud.
[00:02:56] I've also got a history of the changes I make to it.
[00:02:59] So that I know that if I remove something accidentally yesterday, I can go back and I can look at that change, I can actually reverse those changes.
[00:03:08] Source control for me is one of the most critical things you should start with before anything else (followed on by a number of other subjects I'm going to come onto in a second). But source control - first things first, I create somewhere to put my project.
[00:03:23] In the case of this, I'm using a product called GitHub. GitHub is an online source control system. It's probably fairly well known to most of your technical staff. It's probably one of the most widely known online repositories. It holds probably the vast majority of open source projects, but it can also be used for private projects as well.
[00:03:47] In my case, I actually do host this publicly. So if you have wanted to look at the source code for it; you can go there and see it without the need to login or register or do anything else.
[00:03:58] I'm quite happy to share that code because, for me, it's a learning exercise. There's nothing inherently sellable about that product. I'll probably put a link to it in the show notes.
[00:04:12] The next thing I want to do is set up the ability to have Unit Tests. I introduced Unit Test back in Episode 14, along with another number of other testing methods.
[00:04:23] Unit Tests get started with as a local benefit - and I'll probably talk a bit more about the unit test further on - but being able to write test to check that my individual components, as I write them, actually work as I expect them to.
[00:04:39] Great place to start.
[00:04:42] Next, I want to make sure that I'm able to use Continuous Integration, something I introduced in Episode 19.
[00:04:49] The idea is that as I make changes to my code, I pushed that into my source control, in this case GitHub as I said just a moment ago, that then gets pulled down into a Continuous Integration system. Which takes the code, builds it, runs my unit tests and make sure that everything is correct.
[00:05:12] Now, it's only going to be as good as I've written it, but the beauty is that it's making sure that things that work on my machine work on another machine.
[00:05:21] Its very easy for software to get to a place where you may just miss something when you put stuff into source control. So while it may work on one developer's machine, the rest of the team can't actually use it. Continuous integration provides a safety net for that by saying: "Actually, there's a problem here", "This isn't quite working", "Maybe you've missed something".
[00:05:43] Maybe you've got something wrong when you've pushed up into source control.
[00:05:49] To get the Continuous Integration, I'm actually using a tool from Microsoft called Azure DevOps. It provides a number of services, but the key one for me here is an ability to automate that Continuous Integration, Delivery and Deployment so that it's all automated.
[00:06:10] There are a variety of products out there that can do the same thing, but I quite like Azure DevOps. Its probably rather rather overlooked, but its a Microsoft product and probably one of their mainstays that are going to push for more and more over the coming years.
[00:06:26] I also utilise that Azure DevOps system for Continuous Delivery and Continuous Deployment,
[00:06:33] Continuous Delivery, introduced in episode 20, is about making sure that I've got something that I could deploy with a press of a button.
[00:06:41] Continuous Deployment is that automatic deployment into a production system.
[00:06:48] So effectively going from Source Control, Unit Tests, Continuous Integration (builds, make sure everything works, runs those Unit Tests), Continuous Delivering and then Continuous Deployment actually then push any code change I've just made out onto a website that's available to the public.
[00:07:07] I don't have to be public. It could be a private website, but in this case, it goes to a public website somewhere that I could actually go and check what I've just changed has had the desired effect. So I can go from changing something here on my laptop to it being available to the worldwide public in a matter of minutes at most.
[00:07:29] And doing that, it's gone through all my unit tests, so it's got a level of validity and it's been deployed out onto Microsoft Azure - using that cloud infrastructure cloud, which I talked about in Episode nine (obviously a great way of being at a spin up stuff very quickly and get it available with very little overhead in terms of work).
[00:07:52] And these are all good things just to get started.
[00:07:54] Interestingly, a lot of this is actually about getting the pipeline work and actually done much more before you actually start writing any real code.
[00:08:04] Look at the things I've talked about;
[00:08:05] Source control, to make sure I don't lose it, make sure I've got history.
[00:08:08] Unit Test, to make sure I can test things and make sure they validated.
[00:08:12] Continuous Integration, to make sure that it's working when it's not on my machine.
[00:08:16] Continuous Delivery, Continuous Deployment, Cloud - to make sure that those changes are pushed up and made available very quickly for review.
[00:08:28] Now, at this point, we're just getting started with project, it may have very little to show for it, so I may go there and look at it and it's actually very much an empty template.
[00:08:38] It may be very much an empty website
[00:08:39] But because I've got those fundamentals in place, I can make sure I've got that safety net. Plus I've got a confidence that I can see things working. I can be confident, I can release and make changes quickly and efficiently. That's the benefits of doing all of that stuff early.
[00:09:00] Alongside that, not far after I'd actually started to write some real code, I started to develop an End to End test.
[00:09:07] I introduced End to End tests again in Episode 14, the same episode I introduced Unit Tests.
[00:09:12] Its very much a test to make sure that the end journey, shall we say, for the customer, works as you expect.
[00:09:21] And this instance, I basically built a very simple happy path test.
[00:09:27] The test started the survey field, filled in the survey questions page by page by page, and then made sure that all of that response data was stored correctly to the database.
[00:09:40] It's a happy path test.
[00:09:43] Certainly in a more production environment, I'd probably take some unhappy paths. So; errors, if a customer does something strange, if a user tries to do something different.
[00:09:54] But in this case, a nice, simple, happy path test (that could be built up as I built the system) was great because I was able to then link that into my Continuous Integration, Delivery, Deployment platform. So that it could test to make sure that I hadn't broken anything, make sure that I hadn't regressed and made sure it was doing what it needed to do.
[00:10:17] I even got to a point where I actually built an entire separate environment where I could actually push out to a staging environment, rather than the production environment, run all the Unit Tests and then run the happy part test. Then if it was correct and passed the tests, it could be automatically promoted to the production environment without me having to do anything. And this is actually quite good.
[00:10:42] It might scare people initially - the idea of software being deployed automatically without somebody being there to approve it.
[00:10:52] However, why does a manual approval prove that anything is working?
[00:10:58] Why does it prove that what you're doing is correct?
[00:11:02] Generally, it doesn't.
[00:11:04] Generally, we've added a manual process in there just effectively to say we've got something to point out.
[00:11:10] Whereas if we've got an automated process and we've built tests into that (be that Unit Test and ideally End to End automated tests), we've got proof that all the testing was done. We've got proof what that deployment went through to actually go into production use.
[00:11:32] You then come to the question; was that enough testing? Rather than; did it actually go through the agreed steps?
[00:11:42] This is great for having an audit trail that you can show to people to say these are the steps we have gone through to ensure that the software we have delivered into production is of a relevant quality and it is valid for what it's being asked to do.
[00:12:01] I was also able to use that end to end test for Synthetic Transactions.
[00:12:07] A Synthetic Transaction allows me to validate the system is working as expected.
[00:12:14] In the case of this site, while it was running during September, while I was waiting for respondants to come and actually provide responses, I had my Synthetic test run an end to end happy path every hour.
[00:12:32] This would highlight to me very quickly, if for some reason there was a problem in any part of the system.
[00:12:39] I've seen this being used to good effect before in transactional websites - so web shops where people are buying things in a basket.
[00:12:48] How would you be confident there isn't anything wrong with all the various things that go into actually taking that order from the customer online?
[00:12:56] You create a Synthetic Transaction.
[00:12:58] It goes onto the system as if it's a real person, orders a product, goes through to the ordering process, and it validates that it's worked.
[00:13:05] If any part of that fails, it raises alerts.
[00:13:09] So for me, having those End-To-End tests served multiple purposes, not only was it good for me to effectively document and validate the site works as it should do during my development, it's great during regression testing, during releases to make sure I haven't broken anything in additional changes. And I could then use it as a means of actively monitoring the site after it got life.
[00:13:35] This gave me more confidence in the system, this idea of this synthetic transaction running every hour because it's post live and I talked about this in monitoring in Episode 15 - those checks you can do after a system is live to make sure that you have confidence that is acting and behaving as it should.
[00:13:55] Pre-live, you have the stuff we talked about in Episode 14, with the testing - Unit Testing, Automated Testing, Manual Testing.
[00:14:04] Post live, you have that monitoring aspect which could include things like Synthetic Transactions as well as logging and metrics, both of which are built in at that stage there.
[00:14:19] I've talked about a number of things to get started - and some of those practises that helped me develop quickly.
[00:14:23] I've not talked about actually writing code.
[00:14:27] That's almost a, I don't want to say by-product because obviously that's what I'm trying to do, but you've got all these things I've put around it as a safety net - whether it's getting started, making sure I've got all of those tools that allow me to work quickly, all of those tests, all those validations, all that monitoring so that I can have a safety net.
[00:14:46] But these are all things that allow me to have a confidence in what I'm doing.
[00:14:50] It allows me to reduce the risk of any change that I'm making because I have that safety net. I have that automated set up. It allows me to work quickly. If I want to change how the front page looks, I can make that change in a matter of minutes, push it out and can we live within again minutes.
[00:15:13] It helps me to avoid mistakes.
[00:15:15] It helps me to avoid costly mistakes.
[00:15:18] And all of this is great for making an environment where I can react quickly to what I want to do in this project.
[00:15:24] Bear in mind, this is a project I'm doing on my own, something I have to pick up and put down as and when I get a chance to. It's not something I can focus on fully, but that's probably true of most projects these days. Very few projects are projects where you would start work on it and you'll be done within a couple of days and never touch it again.
[00:15:43] So having all these things in place makes it very easy to then pick up making sure, even if I pass this on to someone else, I have confidence that there are safety nets in place for them to be able to both make changes, because most of its automated, but also to catch any problems that they might accidentally had.
[00:16:03] I hope this episode has been useful to you - as a bit of an explanation of how I tie some of the various subjects that I've talked about into my own work, into my own process.
[00:16:14] It would be great to get any feedback from you.
[00:16:15] So feel free to let me know; reach out to me at Twitter @RedFolderMark.
[00:16:23] Thank you very much for listening. I look forward to speaking to you again next week.