Hi there. This is my attempt at writing a developer guidebook 📖. I originally did this for the STFC Hartree Centre where I still work as the Research Software Engineering Group leader. Feel free to raise issues if you disagree with anything, its just my opinion which is subject to change - I believe that strong opinions should be loosely held. There is nothing truly original here, but what I do hope to add is a balanced and informed opinion based on many years of hard won experience. Many of the published practices out there I agree with and have personally encountered or applied on projects, others I'm not so sure about. Happy reading. DaveM
Last Updated 19/12/25. Enjoy.
dv.view('toc')
--startContents
- Table Of Contents
- Why Software Dev Guidebook
- High Level Recommendations
- If it is not in Git it does not exist
- Adopt Semantic Versioning for tags and releases
- Review each others code and be supportive
- If the critical path has not been reviewed it should not go onto master branch
- Continuous Integration - If it does not have tests it does not work
- Continuous Delivery
- Continuous Deployment
- If it Does not have Documentation it is Not Usable
- Learning mindset
- Customer Bill of Rights - modified from Uncle Bob Martin Clean Agile
- Developer Bill of Rights - modified from Uncle Bob Martin Clean Agile
- Tooling
- Coding Recommendations and Best Practices
- High Quality Code is Easy to Understand and Change
- Quality is the best shortcut - Fowler Design Stamina Hypothesis
- Meaningful and Descriptive Names
- Testing has Three Main Purposes
- Keep Most Functions Small-ish
- Should I Limit the Number of Function Arguments
- Separation of Concerns for Locality of Behaviour
- Information Hiding - Deep Rather than Shallow Interfaces
- Code Should be Cohesive
- OOP Redefined
- SOLID
- Pervasive Polymorphism
- Inheritance is Solely for Strong 'Is A' Type Relationships to Maintain State Invariants and Post-Conditions
- Inheritance Should be Explicitly Designed For
- Avoid Paying too much Inheritance Tax - Use Parametric Polymorphism to Represent New Types
- Ad-Hoc Polymorphism via Type-Classes aka Externally Implemented Interfaces
- Simpler Extensions - Avoid Polluting Core Abstractions With Small Behaviour Customisations
- Structural Polymorphism and Duck Typing
- Sealing
- Composition / Delegation
- Parametric Polymorphism Language Comparison
- Data Orientated Programming vs OOP - Choose Two
- Dependency Rule and Dependency Inversion Principle
- Dependency Injection and Inversion of Control to Implement the Dependency Rule and DI
- Circular Dependencies via Setters and Lazy Initialisation
- Dynamic Late Binding vs Static Binding
- It Should Not be Possible to Create an Object in an Invalid State
- Know Some Design Patterns
- Information Hiding
- Keep it Simple Stupid KISS
- DRY Do not Repeat Yourself
- YAGNI You Are not Going to Need It
- Comment in line As You Go
- The Boy Scout Rule
- Principal of Least Knowledge and Train Wrecks - The Law of Demeter
- FP vs OOP - Choose Two
- Data Orientated Programming with Algebraic Data Types - ADTs
- Error Handling
- Error Handling - Four Types of Problems
- Error Handling - Fail Early
- Error Handling - Be defensive at application boundaries, not within your inner domain logic
- Error Handling - Beware Flaky Defensive If-Checks such as TOCTOU
- Error Handling - Model the Absence of Values Explicitly
- Error Handling - Exceptions vs Errors-as-Values
- Error Handling - Exceptions Should Not be Used for Flow Control - Exceptional Does Not Mean Conditional
- Error Handling - Only use Exceptions for Exceptional Situations Such As Coding Errors and Unexpected Errors
- Error Handling - Provide Relevant Exceptions for the Abstraction Layer
- Error Handling - Bubble Exceptions Upwards or Trap at Source
- Error Handling - Do not create custom exception types when the standard library exception types handle your cases
- Error Handling - Use assertions
- Error Handling - Catch Less and Throw More
- Error Handling - Model Exceptions as Values with Algebraic Data Types
- Error Handling in the Functional Way - Returning Smarter Wrapper Types eg the Either Monad
- What are Monads aka Higher-Kinded Types
- Basic Monad implementation
- Example Type Safe Functional Composition by Short-Circuiting on Errors
- Can I Combine Monads and ADTs to Model Multiple Success or Error states
- Other Error Monads such as Validation and Ior
- Inlining within a Computation Block to Avoid Nesting
- Effect Orientated Programming
- Concurrency and Parallelism
- Security Development Practices
- Agile Process Guide aka Feedback Driven Development
- Design Thinking Workshops and Scoping Document
- Epics and Work Package Span Multiple Sprints
- Define user stories with the INVEST Framework or Who-What-Why or the Connextra Card Template – all are good and you do not need to be too rigid
- Arrange core user stories into a Journey Map with a narrative flow or backbone of Big Activities moving from left to right
- Task Backlog
- Requirements Document and System Architecture Document
- 1 to 2-week Sprints
- Inline Testing
- Demo and Playbacks
- Acceptance with Sign Off and Cucumbers
- Iteration and Incrementalism
- Cup Cake Road Maps
- Appendix Recommended Texts --endContents
To help everyone in the Centre build great software, I've put together a collection of development guidelines to help you build scalable, maintainable, reliable, performant, and usable code. Like all guidelines, these are not strict rules, and knowing when and where to apply these guidelines largely comes down to practice and experience. This is not an exhaustive list. For more in-depth analysis, please see the list of recommended texts in the appendix. I'm not going to repeat all that good advice here, that’s what the books are for, but I have tried to distil a range of key recommendations.
Tip
Recognise that code is navigated and read far more than it is written, and that code is a form of expression designed for humans (machine code is for the machines).
_"Programs must be written for people to read, and only incidentally for machines to execute" Harold Abelson, the author of Structure and Interpretation of Computer Programs
This guide is intended for folks who read and write code, mostly for the full-stack application/infrastructure developers. It is not possible to produce a ‘one size fits all’ set of guidelines for everyone. If you predominantly use Python/R via Jupyter Notebooks for example, much of this advice might be overkill, and for that reason, there is separate section for notebooks in the original guidebook. Recognise there are ways to bring more good software practices into Notebooks, see https://nbdev.fast.ai/ which includes good stuff such as Git-friendly notebooks, built in support for CI/CD, support for tests as regular notebook cells and more.
Similarly, if you’re focussing on numerical computation in HPC using C/C++ or Fortran (formula translator), many of the guidelines are simply not appropriate; computational science is a very different domain to full-stack application development and enterprise systems, each optimise for different priorities. Please bear this in mind, these are not rules, interpret them judiciously for your scenario, and as ever, the real answer is always “it depends.” I'll keep evolving this document and welcome any comments.
- Use a GitLab/Github service - https://gitlab.stfc.ac.uk/
- Learn git concepts, not commands
- Branch early, commit little and often with ‘logically sensible commits’ multiple times a day.
- Use a dev branch for your main development and a main branch for your production releasable code.
- Use topic branches (aka feature branches) for your new developments.
Research by Forsgen & Humble (‘Accelerate’ book) shows that long-lived feature branches that remain open for prolonged periods of time hinder delivery and productivity. Team members are less likely to interact and merge conflicts are more likely. The general recommendation is to try and merge feature branches into dev every one or two days. However, the Accelerate book authors do acknowledge that longer lived feature branches are suitable for open-source development where committers are less likely to work full time on features, and so often need more long-lived feature branches.
https://nvie.com/posts/a-successful-git-branching-model/
Tutorials: https://github.com/davidmeredith/scdIntroToGit
[1] https://github.com/davidmeredith/scdIntroToGit/blob/master/introToGit.pdf
There are two strategies to incorporating upstream commits from other branches - merging and rebasing. Upstream commits are new commits that exist on another branch which need to be incorporated into your current branch to keep the branch up to date:
a) Periodically merge the changes in the target branch (dev) into your feature branch. This creates the ‘braided’ graph pattern show opposite (flow is from top to bottom). When you’re ready to merge dev into master, a new merge-commit is created on the tip of the master branch.
b) As shown below, rebasing basically ‘breaks off the feature branch from its root (yellow), and re-attaches it to the tip of the target branch (grey)’. During the rebase, the commits that exist on the feature branch are internally used to create a set of diffs in temporary files which are used by git to create new updated versions of your feature commits. Git needs to do this in order to incorporate any upstream changes that may have occurred on the target branch.
Note that rebasing does not delete the feature branch, the feature branch still exists, but it is now ahead of the target branch, by 2 commits in the diagram below. To bring the target branch up to date, a fast-forward merge is required on the target branch.
Whether to rebase or merge is generally down to preference:
- Use ‘rebase’ to produce a clean and linear commit history. You can optionally use ‘interactive-rebasing’ and ‘squashing’ to clean up your commits too. Also observe the golden rule of rebasing: Don’t use rebasing if your feature branch is shared amongst multiple developers – rebasing essentially ‘pulls the rug from the under the feet of the other developers’ working on the feature branch.
- Use merge if you need to preserve a full history, for detailed auditing purposes for example. If this is the case, you probably don’t want to delete the feature branch after merging it, but you can do this later if needed.
- Foster a friendly and supportive environment and politely shout-out vulnerabilities and apparent issues, don’t be shy. As a reviewer you’ll learn something. Code reviews and pair-programming really does improve code quality and exposure to different projects/codes/practices.
- Basic code quality relies on having at least two pairs of eyes on code, to catch errors, suggest improvements, build shared knowledge, and improve code style. Get into the habit of developing on branches or forks and using pull / merge requests to facilitate code review before merging.
- To be able to say something 'works' we judge it against some (implicit) criteria. Writing tests makes our success criteria explicit. Automation (Continuous Integration) prevents regressions. Eventually, this leads towards test-driven development (TDD) where we think clearly about specifying what 'working' looks like up-front by writing code from a caller’s perspective.
- Test as you go along to a level that’s feasible and pragmatic. Extensive testing with production-level coverage (70 to 80%) is not always achievable or useful. Given project budgets and timescales, focus on testing the application’s critical path as a minimum.
- Avoid gold-plating and focus on shipping code ASAP for customer review & feedback.
- Your Unit tests should be fast to complete – order of seconds.
- Do not just rely on unit tests - system and integration tests are also needed. Unit tests alone will instil a false sense of security.
- Integration and System tests can take longer to complete.
- For more details on best practices for testing, including the different types of testing from Unit, Integration to System tests, see: https://epubs.stfc.ac.uk/work/50305274
Means that the software should always be in a releasable-ready condition. This is a recommendation for your master and dev branches. If you’re run over by a bus (ROBAB), and someone must come along and pick up your code and they must fight with it from the outset, there’s a strong chance it will become shelfware. If you need to have prolonged branches for experimentation that aren’t release-ready, create a feature branch such as ‘feature:homersSandbox’ to isolate your experiments. https://epubs.stfc.ac.uk/work/47984368
Means that once the software is merged into master branch, it automatically gets pushed into production. The idea is to make the large and risky ‘big feature release’ a legacy practice. By continuously deploying to production with small and frequent updates, if something goes wrong, its quick and easy to rollback. This might be a stretch for Hartree because its more relevant for long-lived production software & products, not so appropriate for proof of concepts.
- Publish docs: Jira (task/sprints), Confluent (docs), GitLab, (code, merge requests, CI/CD), Bid register (PMO tracking tools)
- Use xDoc style code comments such as JavaDoc, PyDoc
- Document the intended purpose / intent of a function/class/package.
- If it’s tricky to document the intended purpose, then your class/function is likely too long and needs breaking down into smaller units.
- Always add a README.md.
- Always document inline - we rarely go back and document our code after its written, fact.
- End-users need to know how to use our software, so think about the right level of documentation for users. Consider separate user & developer docs.
- The C4 approach to technical diagrams is good. https://c4model.com/
- For more details on how to write good documentation for different users including ‘How To’ Guides, tools such as Markdown/AsciiDoctor, and reference guide formats, see: https://epubs.stfc.ac.uk/work/47984356
- Keep reading & learning and record all your training activities.
- Push beyond your comfort zone.
- Participate in RSE Skills & Learn sessions.
- Keep up the pursuit of software engineering craftsmanship, mastery and professionalism.
- Tinker - its really important to do hobby projects and dev stuff you enjoy.
Customers have the right to:
- An overall plan and to understand what can approximately be accomplished and at an estimated cost.
- Get the most possible value out of their projects.
- See progress in the development of a system.
- Change their mind, to substitute functionality, and to change priorities subject to agreement and re-scoping of the plan.
- Be informed of schedule and estimate changes, in time to choose how to reduce the scope to a meet a required date.
- Cancel at any time and be left with useful outputs reflecting their investment to date.
- Developers have the right to:
- Know what is needed with clear declarations of priority.
- To always produce high-quality work.
- Ask for and receive help from peers, managers, and customers.
- Make and update their estimates at any time.
- Challenge the task and the responsibilities instead of having them assigned – professionals accept work, they are not assigned work. A professional developer has every right to say no to a particular job for various reasons, from ethical to overloading.
- Knowing ‘accepting work’ comes with a cost – acceptance comes with responsibility.
As a centre, we should be using the right tools for the job, we all have our preferences, but there’s no need to be stubbornly loyal about a particular language. As software professionals, we should recognise the right tools for the job and for our clients.
Have the customer in mind. For example, Haskell and other Lisps are great (I’ve played with Clojure), but don’t be smart and use this as an opportunity to explore your favourite pet-programmer project, it’s not going to be much use to the customer. It’s hard to hire Haskell programmers.
For certain types of programming, e.g. numerical algorithm development, HPC, and when squeezing software into tight spaces such as embedded systems, a Garbage Collected (GC) language probably isn’t the best choice - a GC adds extra memory, disk, and CPU requirement. Therefore, venerable manual memory managed languages without a runtime such as C/C++/Fortran are still a good choice. Newer memory safe languages such as Zig, Rust, Mojo, Julia are interesting and relevant in this space, but I do not have enough hard-won experience in these languages to recommend them or not.
For most other types of software development such as full-stack, enterprise, web services, mobile, and general-purpose programming, I do recommend using a memory safe language and this implies having a GC for the majority of languages (apart from Rust). There is a recognised shift in industry away from memory unsafe languages as the vast majority Common Vulnerability Exploits (CVEs) stem from unsafe memory language exploits, causing organisations such as Google (for Android), NSA and Microsoft to urge the use of memory-safe languages, DARPA through its 'TRACTOR' programme (Translating All C to Rust), and the US Gov for all its governmental projects to start all new projects in a memory safe language.
Tip
“A human garbage collector is just wasted effort” (Eckle & Ward, Happy Path Programming).
At the time of writing, there doesn't seem to be a non-GC language that supports monads for functional composition coupled with a colourless async runtime eg green threads and coroutines. Note, I do not mean platform/kernel/OS threads, I mean virtual or 'green' threads that are implemented by the runtime eg Go's Goroutines, Java's Project Loom, Fibres PHP and Ruby, Kotlin coroutines. This appears to be a current open research topic in non-GC'd languages. According to this Rust maintainer, async is possible with coloured approaches such as async/await (eg C++, Rust), but colourless approaches such as green threads and coroutines, and advanced functional programming with monads currently requires a GC. The GC handles memory clean-up for several low level complexities eg async task cancellation, how to implement memory clean up without higher-level approaches such as RAII and destructors, and providing a debugger that is usable for complex async code. I'm currently monitoring what Zig does in future - here is an interesting summary, and apparently Zig's defer/errdefer becomes impractical in an async context.
Note
For me, the increased productivity, and availability of a colourless async runtime with the ability to implement monads for functional composition is (currently) worth the cost of using a GC language, especially for application programming.
TODO I don't have enough hard-won experience to confidently make this call, so I'll have to defer to the opinion of of others. According to Lars Bergstrom, Director of Engineering for Android at Google, coming soon.... top
Keep all project documents and the Decision Point Review templates in a single repo such as Confluence so the client has full visibility. Its best if the client creates the (free) Jira instance and invites us as admins on their Kanban/Jira/Confluence. This way, the client has ownership of the project after the project completes. Make sure you export and download a copy of the Jira archive file so we can restore it within our own Jira instances if needed in future or for follow on projects.
It is the developer’s responsibility to use an appropriate build tool that manages the dependencies of your project. It should be possible to clean the project, download dependencies, re-build a project, run unit and integration tests, and build a deployable package all from the command line. It is also the responsibility of developer to create the necessary environment configuration files (with specific version of modules or libraries) in a consistent state so that someone else can pick up your project easily.
Static code analysis helps bring consistency to your code. Within a project, adopt the same style guides and agree the linters up front e.g., Black for Python, Google style guide for Java are good examples. There’s no ruling here, pick one that suits the team and be consistent. There are plenty of static code analysis tools out there such as CheckStyle, FindBugs, SonarCube, Black, Google linter, IDE checks.
Containers have become the de-facto way to mitigate the common claim “well, it works on my machine”. OCI compliant containers (Open Container Initiative) are great for wrapping code with all their dependencies into shareable images that can be uploaded to image repositories such as STFC’s Harbor service (https://harbor.stfc.ac.uk/). Containers are a great way to share code with your clients, especially if they need to run your code on their runtime platform. Containers are ubiquitous and can run on the Desktop, in Kubernetes clusters, in cloud Functions such as AWS Lambda, on HPC such as Apptainer/Singularity, and more.
Here are some recommendations:
-
Only include runtime dependencies: Be mindful of what you’re including in your container - for production, you really don’t need to include compilers, package managers, and tools that are meant for use only at compile time (unless you’re containerising your dev environment of course). For example, Google’s Distroless containers and Alpine Linux are great for production use, providing cut down versions of Linux. Containers such as these are great because they have less memory-footprint, and by reducing the amount of unnecessary stuff in them, they reduce the vulnerability attack surface, so they’re safer. Here’s a great video that shows how to build super slim production containers – ignore the fact that it is for Java, the concepts and tools discussed are generic and apply for many languages.
-
Don’t statically link glibc – use the musl library instead. Glibc is notoriously unfriendly for containerisation and was not designed to be so.
-
Don’t run your code within the container using the root user unless you really must.
-
Use image layering to split up build time and run time dependencies. For example, having a separate 'build' and ‘run’ layers in your docker file allows you to copy only the built application code and dependencies into the container, leaving out all the unnecessary compile time dependencies. It also means you can re-run the container more quickly without having to re-build each time as build layers are built and cached locally.
-
Testcontainers is awesome, download and run containerised apps/dependencies such as databases, services, tools and use them in your integration-test suites. Comes highly recommended: https://www.testcontainers.org/
-
See this great guide from the RSE team for more info on how to run HPC Singularity (now Apptainer) and Conda images with worked examples.
Please refer to this separate document that characterises all the different types of workflows we use at Hartree, including Data Flow Engines that orchestrate containers using DAGs (Directed Acyclic Graphs): http://purl.org/net/epubs/work/50844906. Our ‘Demystifying Data Engineering’ Explain course provides more details into Data Flow runtimes and tooling.
This is not an exhaustive list of coding recommendations. For more in-depth explanations, please see the list of highly recommended texts in the appendix. I’m not going to repeat all that excellent advice here, that’s what the books are for, but please find below a collection of development best practices that we should consider when developing our software. Like all guidelines, they aren’t strict rules, knowing when and where to apply largely comes down to experience.
Code needs to achieve its purpose under certain parameters, but assuming that it does, what is high quality code? There are many definitions, but I like "High quality code is easy to understand and change." It implies the code is readable, it has good abstractions with well named constructs (classes, functions, variables, packages etc), and overall it is maintainable. Watch this.
- “Clean code does one thing well.” (Bjarne Stroustrup)
- “Clean code reads like well written prose.” (Grady Booch)
- “Clean code always looks like it was written by someone who cares.” (M. Feathers)
- “Getting code to work is only half your job, and it’s the least important part. The most important part is that you write code that other people can maintain/use. If you hand me code that works that I can’t understand, it is useless as soon as the requirements change.” (Uncle Bob Martin).
I'm not sure I agree with Uncle Bob on this one. Getting the code to 'work' first, especially for proofs of concept or blue-sky research code, is probably the higher prioirty.
Interestingly, The Primeagen seems to emphasise a slightly different point of view: "Not all code can be or needs to be made readable; it also depends on the skills of the programmer.” I believe this is partly true, I acknowledge his point when applied to deeply numerical code or reactive JS for example, these types of code-bases are tricky to refactor for readability, but I als notice the push-back in the first comment on his video:
- Think carefully about compromising the quality of your code for delivery speed; high quality code quickly becomes easier and faster to develop and overtakes hastily hacked together code. This is evidence based, see Martin Fowler's Design Stamina Hypothesis) which describes how the velocity of software development declines with time due to poor design. We have also experienced this before in actual projects – the hypothesis has played out in practice at Hartree.
Tip
- High quality code is easier to maintain.
- High quality code increases developer productivity (post design pay off line)
- High quality code can thus reduce cost (time is money)
- “There are only two hard things in computer science, invalidation and naming things” (P.Karlton), see: https://martinfowler.com/bliki/TwoHardThings.html
Tip
- Use intention revealing names for your abstractions, classes, functions & variables.
- For example,
process_modelis too generic, what does it mean?execute_nlp_training_modelis better, its more self-documenting.
- For example,
- Don't use single character variable names
- Single chars for implicit loops etc are OK, but for global/module/class members, please use sensible names.
- Class names should have noun phrases e.g., Customer, WikiPage, Account, AccountParser.
- Don't name abstractions after constituent parts.
- API comments should describe the intention, not the implementation or how (internal comments ok)
- Don’t write comments when you can use a well named function or variable.
The following two examples are taken from Uncle Bob's 'Clean Code' book:
Which is easier to read?
// Java
// check to see if the employee is elibible for full benefits
// This
if((employee.flags & HOURLY_FLAG) && employee.age > 65) { // 🤯
...
}
// Or this?
if(employee.isEligibleForFullPension()){
...
}I have mixed feelings about this piece of advice from Uncle Bob: "Every time you express yourself in code, you should pat yourself on the back. Every time you write a comment, you should grimace and feel the failure of your [lack] of ability of expression." Bob goes on to explain that you should use well-named variables instead of comments, as shown in the example below (real-estate limits apply - imagine a much larger example). I understand the sentiment here, but I think it can be easily misinterpreted - I believe we do still need comments, and public api-doc should focus on the intention, not the 'how' (reserve 'how' comments for internal-function comments).
// Java
// For our system, we need to get all the depndent systems
// and see if our sub-system is dependent on it.
if(this.getSystems().contains(this.subSystems.current())){
...
}
var allSubSystems = this.getSystems();
var ourSystem = this.subSystems.current();
if(allSubSystems.contains(ourSystem)){
...
}- Do not name abstractions after their constituent parts using a 'bottom up' approach. By definition, an abstraction should use a higher level vocabulary and shield you from the lower level details. Gregor Hohpe from 'Enterprise Application Integration' book fame provides a great example: If software engineers had named the automobile, it would have been something like:
PistonCrankshaftGearWheelAssemblyFactorySingleton- not a useful name (this reminds me of some Spring Framework classes). Another simple example from Gregor: Q. consider theGasPedalof the car, is that a good abstraction? A. no, the car might not use gas, a more suitable name is theAccelerator.
Abstractions give us freedom and protect us from more volatile concrete implementation changes. However, adding abstractions does make code more complex and it can be overdone - take a look at the FizzBuzz parody for an example.
Tip
You do not need to add abstractions to everything by default. Hide core classes behind interfaces. Use abstractions when crossing core boundaries in your application code.
Tip
Testing has 3 main purposes:
- To assert the correctness of your code.
- To prevent regressions.
- To encourage good design.
- 1) To assert the correctness of your code. Sometimes you might counter “well, it’s not always possible to know the result of a calculation to assert because the result is non-deterministic”. In this scenario, the following reasons to test still hold true!
-
- To prevent regressions. It really does build confidence in your code if you can quickly run a test suite to make sure you haven’t unexpectedly broken something.
-
- To encourage good design. It really does actually - there’s lots of supporting research that shows this. When you write tests, you put yourself in the caller's perspective, so you really do think about design such as loose coupling, separation of concerns, modularity, simplicity of API, and so on. If your code is hard to test, it’s likely too strongly coupled, and here Dependency Injection with abstractions can help you. Test Driven Development (TDD) is the ultimate testing practice, where you write tests first using an imaginary API, and then you fill in the details.
I don't subscribe to certain views that functions should be no longer than four or five lines myself - I find it too difficult to keep a good train of thought when hopping around many small functions. I think functions can be long when necessary, especially complex functions that require a sequential train of thought to build a mental model. As a general rule, most functions should normally not be much bigger than your screen’s viewport, but larger functions are fine when you need them.
Tip
Know that compilers can apply far more effective in-lining optimisations with smaller classes and functions.
If your language has named parameters with default values (e.g. Py/Kotlin), then you often encounter long function argument lists. For example, take the definition of k_means from scikit-learn shown below. This is fine as the named parameters clarify what the args are, whilst default values means you only need to provide new/custom values where the defaults are not suiable:
def k_means(
X,
n_clusters,
*,
sample_weight=None,
init="k-means++",
n_init="auto",
max_iter=300,
verbose=False,
tol=1e-4,
random_state=None,
copy_x=True,
algorithm="lloyd",
return_n_iter=False,
):
# ...However, if your language does not have named function arguments, then such a long argument list would be difficult to comprehend, and there is the risk of argument mix-up. There are several potential solutions that you can use to simulate an approximate solution, such as the builder pattern and DTOs used as parameter bags to wrap long argument lists.
Tip
Consider using immutable Data Transfer Objects (aka DTOs / Data Objects / Records) if your function has more than ~4 arguments and if you language does not support named and default function arguments.
In the following example, a simple mutable object bag is used to define a lambda to name optional arguments with default values. Using a mutable object here is ok because the lambda's scope doesn't leak beyond the method execution. In the future, Java records with 'withers' will likely become the recommended approach, until then, there are libs such as record-builder.
// public class for brevity, but this could be defined as a nested class for locality
public class MyFunction {
private MyFunction(){}
public static class Props{
public String name = "";
public int age = 0;
public String phone = "not Set";
public String idNumber = "not Set";
private Props(){}
}
static public void execute(String mandatoryParam, Consumer<Props> params){
var p = new Props();
params.accept(p);
//validation logic if required
if(p.age < 0){
throw new RuntimeException("age can't be negative");
}
// do something
}
}
void main(){
// The lambada simulates named function arguments
// Local scoping avoids issues with subsequent mutation of the properties bag
MyFunction.execute("this is mandatory", p -> {p.name = "Homer"; p.age = 30;});
}Functions should do one thing. For some functions this is true, especially low-level functions for example. However, I don't believe this is a unanimous rule. Higher-level functions (the outer layer in the dependency bullseye) often compose multiple child functions to achieve an end result. For example, consider a controller that composes multiple service facade calls to provide a final response.
Tip
I prefer: Functions should separate concerns to give Locality of Behaviour. LoB means that code is easily understood by looking at only a small portion of it (paraphrasing R. Gabriel)
The following example is from 'Modern Software engineering' (Dave Farley): There are three implementations of an add_to_cart method that performs three tasks:
-
- adding an item to the cart,
-
- persisting the card to a DB, and
-
- calculating the total cost of the cart.
Bear in mind that that the examples are intentionally short, imagine much longer functions. Example 1 is poor, it mixes all three concerns directly and you need to understand the low-level details which would be better located in a different persistence-focussed package. Example 2 is much better - logic is abstracted into private methods with descriptive names that 'reads the logic,' this is far easier to follow. Example 3 is even more loosely coupled - is this an improvement over version two, probably not for this simple example, but you can see how this further decouples concerns. Example 3 is a nice pattern for complex systems - it is easy to add new event listeners to react accordingly without having to change the logic of the function.
# Python
def add_to_cart_poor(self, item):
self.cart.add(item)
conn = sqlite3.connect('my_db.sqlite')
cur = conn.cursor()
cur.execute('INSERT INTO cart (name, price) values (item.name, item.price)')
conn.commit()
conn.close
return self.calculate_cart_total();
def add_to_cart_good(self, item):
self.cart.add(item)
self.store.persist_item(item)
return self.calculate_cart_total();
def add_to_cart_extensible(self, item, listener):
self.cart.add(item)
listener.on_item_added(self, item)TODO.
Tip
Code should be cohesive which is related to the Single Responsibility Principle. High cohesion means the methods and variables of a class are co-dependent and often change together. This can be paraphrased as "Changes to the code over here should not affect code over there" and "Code that changes together stays together." Another useful interpretation: "When you want to add something, you intuitively know which class should be modified."
Here’s the authoritative view from the famous Kent Beck from Nov (2022) and his ‘Tidy First’ approach to software development:
Examples of patterns that support cohesion include the State Pattern.
- Consider code that is widespread across many files with each file having 'switch' or 'when' statements that reference a centrally declared enum set. The switch/when statements execute different behaviours based on the current enum state value. If you add or remove a state enum option, you will need to update the all switch/when statements spread across your entire code-base. This is not a problem for small projects, but for large code bases it can require significant refactoring. The state pattern co-locates the state enum values with the associated behaviour - state objects are responsible for their own behaviour and for changing into another state. To do this, the central enum set could be replaced with a corresponding set of state objects, where each state object collects and implements the relevant state-dependent behaviour itself.
TODO - provide an example
OPP is about "Program organisation" (Bjarne Stroustroup).
The OOP data model of class inheritance hierarchies and interface implementations has, undoubtedly, proven to be highly effective in modelling real-word data, largely due to its frequent natural fit for representing real-world objects, and also due to its expressiveness. OOP still dominates compared to other programming paradigms such as functional and procedural. OPP helps you to organise programs around data and the actions that are intended to operate on that data. From my perspective, what I like about OOP is that it is really clear how to organise my program around classes - locating the data alongside the functions that operate on that data is very common and often feels natural. Without classes, you can lose some of this clarity, especially in large programs. However, OPP is not the only option. Other paradigms such as Functional, Procedural, and Data Orientated Programming exist. I'm not a purist - I believe you can and should apply all these paradigms when it is most convenient to do so. Most general purpose programming languages allow you to mix paradigms to some extent.
OOP bashing appears to be popular these days, but this is mostly because of the problems associated with hierarchical inheritance (see section on Don't Pay Too Much Inheritance Tax), and I agree that hierarchical inheritance should be applied judiciously. However, I believe that most of the other OOP principles are tried, tested, and proven. You can’t argue against three of the four 'pillars' of OPP: Abstraction, Encapsulation, and Polymorphism. These are good design principles largely available across all modern programming languages. Arguably, what pure OOP can over-emphasise is deep & brittle inheritance hierarchies, but instead you can favour composition instead in the right scenarios.
| Pillar | Implementations |
|---|---|
| Abstraction | Interfaces, Traits, Service Facade, Type-Classes |
| Encapsulation | Classes, Records/struts/nested-structs, Closures, Modules, Public/Private/Protected visibility, monads (wrap side-effects) |
| Polymorphism | Inheritance, ADTs, Type-Classes, Generics & Parametric Polymorphism, Interfaces/Traits, Dynamic-Dispatch |
| Composition / Delegation, Traits, Default Interfaces, Inheritance, Type-Classes |
I believe inheritance should be replaced with 'Sharing Behaviour': the other pillars are generic to accommodate multiple implementations and inheritance is not the only way of sharing code across types. Other ways to share code include Composition (available any language), traits (Scala, Rust), interfaces with default implementations and attributes (Java, Go, Py), and type-classes (Scala), to name a few. I agree that inheritance can be overdone, but it does have it place (see section on 'Don't pay too much inheritance tax').
For OPP, understand the principles of SOLID:
| SOLD | Meaning |
|---|---|
| Single Responsibility Principle (SRP) | A module should be reponsible to one, and only one, ACTOR (Bob Martin's reinterpretation) |
| Open-Closed Principle | Classes should be explicitly designed for extension and can't be unintentionally modified. |
| Barbara Liskov’s Substitution principle | A subclass can substitute for its parent, this mostly surfaces when chaining function calls. Requires all behaviour, public members (API and variables), invariants and post-conditions to be the same, commonly requires inheritance. |
| Interface Segregation Principle | Separate interfaces to focus on just a single concern. |
| Dependency Inversion | Abstractions should not depend on details, but details should depend on abstractions. To do this, an abstraction layer (interface) is extracted to separate high and low level modules. |
A common misinterpretation of SRP is "One function/class should only do one thing." While I don't disagree with this advice, its not actually what the SRP was getting at. The formal interpretation is "A module should have one, and only one, reason to change." This is a touch ambiguous, I prefer Uncle Bob Martin's reinterpretation "A module/class should be responsible to one, and only one, ACTOR" (because the reason to change comes from the actor, a useful clarification in my opinion). In Bob's great book, Clean Architecture: A Craftsman's Guide to Software Structure and Design, Bob illustrates a violation with an Employee class that mixes three business rule methods, where each is required by a different actor: calculatePay() for the Finance team, reportHours() for HR and save() for Operations. Under the hood, these methods could refer to some common private logic eg regularHours(), however, this type of de-duplication by consolidating common code into a shared function can often become a source of bugs, especially for complex code which spans different dev teams which requires merging of different VCS branches. SRP says don't do this, instead separate the code that different actors depend on, even at the expense of some intentional duplication.
Tip
SRP is closely related to cohesion: "Code that changes together stays together" and when you want to implement SRP think "It should be easy to identify which class should be modified if I want to change something."
A solution to the Employee class violation example would be to use a data object that holds nothing but the employee data (no methods) and move the behaviour into separate facade(s) for processing business rules, one for each actor for example (PayCalculator HourReporter and EmployeePersistor). Notice that this solution is the end of a spectrum of possible solutions - Bob does explain that some developers or in some scenarios you may prefer to keep the most important business rules close to the data. In this case, the original Employee class can still contain the three business rule methods, but these methods are very short and simply delegate to the appropriate business rule calculator facades under the hood (Composition).
Tip
Don't define a new data type (class) solely as a container for a single function to do something. This is overkill - just write functions and call them.
Don't do this:
class ToUpper : ICharacterTransform
{
public char Transform(char c) => char.ToUpper(c);
}
class ToLower : ICharacterTransform
{
public char Transform(char c) => char.ToLower(c);
}OCP is relevant only for classic hierarchical class extension, however, as we prefer object composition over inheritance these days, this principle has less relevance in modern software engineering. You should explicitly make classes non-extendable by default meaning you explicitly have to design classes for inheritance when you need to.
What does polymorphism mean to you? For those with background in OPP, you most likely think of inheritance using an abstract base-classes with sub-type specialisations. However, I agree with Bruce Eckle in that it should be regarded as a much broader term, and polymorphism can crop-up all over:
Tip
Don't limit your understanding of Polymorphism to inheritance alone, it broadly means that 'a single type can represent multiple types.'
If we acknowledge this interpretation, then polymorphism can include:
| Polymorphism type | Description |
|---|---|
| Inheritance | Classic inheritance hierarchies with base class & sub-type specialisations |
| Union/Sum Types | A type must be one from a selection of types to implement choice |
| Parametric Polymorphism | An interface is attached to a type to represent another type, often by composing objects |
| Ad-hoc Polymorphism / Type-classes | An externally implemented interface is attached to a type to represent another type, often by composing objects. Existing 3rd party types can be attached to the external interfaces. |
| Structural Typing / Protocols | A type is compatible if has methods that correspond to an interface protocol, with compile-time checks (eg Go) |
| Duck Typing | Same as Structural Typing with dynamic-dispatch (eg Python) |
Inheritance is Solely for Strong 'Is A' Type Relationships for State, State-Invariants, and Post-Conditions
Tip
A common mistake is to assume inheritance is mainly for sharing behaviour. It is not, it is for inheriting all public and protected members including state variables and, importantly, requries all invariants and post-conditions are maintained. Interfaces and type-classes don't allow state sharing by design, they are designed to be lightweight solely for passing around behaviour. When you need to share state, you need to rely on inheritance and/or composition.
The Liskov substitution principle requires all invariants and post-conditions are maintained when extending a parent - it is required that a parent object can be replaced with any of its sub-types. The sub-type must therefore maintain all public members including methods, state, all invariants (i.e. conditions and relationships across all state that hold true), and all post-conditions (i.e. the invariants hold after object construction and after applying behaviour). Inheritance is the mechanism that enables all of this. Inheritance should be reserved only for strong and natural 'IS A' relationships e.g. a swallow is a bird. This principle can be over-applied, in fact you often do not need inheritance and should instead favour sharing behaviour using parametric polymorphism and/or type classes instead. As a memory aid, if you define a super class Bird, you would think adding the fly() method would be appropriate, but no, not all birds can fly and some can swim, so flying is a behaviour better added as by the Flyer interface, so Penguine can omit fly() (and also apply the Swimmer interface). I would say that the semantics of the parent need to be maintained by the children on all of the 'features' for the relationship to be a natural 'IS A' relationship.
By default, in some languages you can extend a class by default (the wrong default), unless you explicitly disallow it e.g., using the final keyword in Java or through object and interface Sealing. In modern languages, classes are typically closed to extension by default. For example, in Kotlin you have to explicitly enable class extension using the open keyword to make it explicit that this class is designed to be extended. This makes SOLID’s 'Open Closed Principle' best-practice explicit in the language.
Tip
By design, inheritance is an intentionally strong 'Is A' relationship; For a sub-type to act as its parent type (the Liskov substitution principle), then it needs to maintain the same behaviour, AND the same invariants AND post-conditions. A common mistake is to focus just on behaviour, inheritance is a much stronger relationship than that.
Tip
I say "don't pay too much inheritance tax" intentionally because inheritance is great in certain scenarios such as in building frameworks, libraries, and for strong/natural "Is A" relationships e.g. "type A is a type of B." A great example is the 'Either' monad - the Left/error and Right/success objects inherit from the Either base class, so an Either can be Left or Right, but never both. Another good example is in ORMs (object relational mapping frameworks) where it is common to extend a base 'Entity' class that provides the primary key for the entity/relation and any the invariants associated with the PK.
However, deeply nested inheritance hierarchies where sub-classes extend super-classes can become very brittle. This is because you are structurally tied to the classes in the parent hierarchy:
- If you don't need all of the characteristics provided through inheritance, it can be difficult to 'split-out' what is not wanted without widespread refactoring.
- Some languages (eg Java) do not allow multiple inheritance which could encourage deeper inheritance hierarchies.
- If you don't have access to the src of the inheritance hierarchy, you may be forced to implement abstract methods you don't need, typically by throwing unsupported exceptions/errors.
If you have access to the src of the inheritance hierarchy, you may need to extract the required methods into a new level and inherit from that level to facilitate better segregation of concerns e.g., from the direct parent if you want all methods, or from a higher-level ancestor if you do not need every method. This can be an expensive refactor meaning deep inheritance is often considered an anti-pattern these days, especially when building applications rather than frameworks and libraries. A number of modern languages don't even support inheritance. Having said that, inheritance arguably does have its place when developing libraries and frameworks, and especially for relationships that have a strong and natural "Is A" type of relationship e.g., 'typeA is a genuine / real sub-type of typeB.'
Tip
Attach Interfaces/Traits to your types for new behaviour: Rather than pay too much inheritance tax, consider using 'Parametric Polymorphism' where a 'behaviour type' such as an interface or trait attaches a new contract to your types which allows a type to be represented as another type. This shared behaviour type augments your type-system for use in function parameters, return values, and attribute declarations, and is often used in conjunction with generics ('<holder_types>'). The canonical name is "parametric polymorphism" or "generic programming". It is typically implemented with dynamic-dispatch and is often used in conjunction with Composition under the hood where an object uses the functionality of another object to implement its behaviour.
Several modern languages don’t even support inheritance (Rust, Zig, Go), relying instead on parametric polymorphism. Despite the subtle differences across languages, they generally follow the pattern of externally implemented interfaces that you use to 'attach or graft' behaviour onto types: Interfaces (eg Java, Kotlin), Traits (eg Rust), Mixins (eg Python), Type-Classes and externally implemented interfaces (Scala, Kotlin).
| Parametric Polymorphism Pro | Inheritance Con |
|---|---|
| Interface segregation is finer grained - you choose what to implement | You may need to inherit everything from all ancestors (note, this might be the intention as per Liskov substituion) |
| More loosely coupled, changes to an implemented interface does not affect other types. | Tightly coupled - changes in the parent hierarchy can vertically affect all sub-types |
| More flexible - implement multiple interfaces | Single inheritance in some languages is more restrictive |
| Polymorphism - multiple implementations for a generic interface/trait. |
Some languages allow you to externally implement generic interfaces 'outside of a type' and associate implementations of these interfaces to other existing types without having to modify those original types. This is known as 'ad-hoc polymorphism' and comes from functional programming. Approaches differ across languages, and includes Traits in Rust and Type-Classes in Scala/Haskell/Kotlin, Protocols (Clojure/Swift), Implicits (Scala), Shapes and Extensions (C#).
Tip
A type class is an abstract, parameterized type that lets you add new behaviour (not state) to any closed / existing data type without using sub-typing.
Why? Ad-hoc polymorphism is very convenient and powerful for easily creating 'blanket implementations' for defining a new type for a whole group of types at once, even when you don't have access to the src code of those types. This can't be done with interfaces that are defined directly on types. Brian Goetz describes the limitations of traditional interfaces where you implement the interface directly on the type as follows: "Interfaces have limitations: they abstract over behaviours of instances, not behaviour of types which means you need to already have an instance of a type to use them. This is known as the expression problem." This basically means you need the src of a type in order to directly implement an interface to extend the type with new behaviour. To address this limitation, the Java architects are exploring the use of type classes, the current proposal is to move additional behaviour to a third-party 'witness' object and publish a witness to the existing/external type as explained here.
There are subtly different approaches used to 'associate' or 'scope' the externally implemented interfaces to existing types, as explored below. Languages typically lexically split their core data types from the implementation of the external interface, and some languages use parametersied generics on the external interfaces to provide an implementation for the (existing) type.
Tip
Lexically splitting a type from a new type that defines new behaviour defined in an externally implemented interface (a type class) and then associating the two together is different from directly implementing the interface on the original type declaration itself (eg implementing an interface directly on a class). The main advantages is that you don't need to modify an existing type which enables new blanket implementations. Recognise that there are at least 3 code fragments required to do this: 1) an existing type which can be a 3rd party type where you don't have access to the src, 2) an externally implemented interface/trait/type-class, and 3) code that 'associates' or 'scopes' them together. If that extra layer of indirection feels a touch unnatural or overkill, you can easily co-locate these individual component parts near to each other in the same file to achieve that familiar class feel.
The following Rust example provides a simple example. Note that in this example, Rust uses static dispatch or 'monomorphization' for performance reasons and not dynamic-dispatch (which allows you to create multiple implementations for the same type at small performance costs).
// Rust
use externalcrate::SomeStruct; // here is our existing type
trait Length { // A trait to declare shared behaviour
fn get_length(&self) -> u32;
}
impl Length for SomeStruct { // Implementation attaches trait 'Length' to type 'SomeStruct'
fn get_length(&self) -> u32 {
self.to_string().len() as u32
}
}Advantages of Ad-hoc Polymorphism:
- No modification of existing types - behaviour is attached without need to access or modify existing types types.
- New 'Blanket Implementations' - gives the ability to define a new type-class implementation for a whole group of types at once, rather than having to write separate implementations for each one.
Disadvantages of Ad-hoc Polymorphism:
- Increased code complexity.
- If dynamic dispatch is used, it has a small performance penalty (usually it is not a problem for the majority of code-bases).
- Externally implemented interfaces don't carry state (member variables). For example, as cited by the Rust book: "trait objects differ from traditional objects in that we can’t add data to a trait object. Trait objects aren’t as generally useful as objects in other languages: their specific purpose is to allow abstraction across common behaviour." Note that some languages allow variables to be declared on interfaces but with caveats e.g., Kotlin allows abstract properties or accessor methods with no backing fields (see Kotlin docs) to define what properties (and methods) a class must implement.
The following Kotlin example provides another example. You will notice that the approach is different:
- Rust: A trait implementation is linked to an existing type using a code fragment that references the trait implementation and type/struct:
impl <someTrait> for <someStruct>. - Kotlin: Separate objects are used to create implementations of our external interface, and these instances are then indirectly attached ('scoped' in kotlin terminology) to our existing type at the call-site using:
with(someScopeObj)'scope' function. This process uses dynamic dispatch to supply the correct implementation, not static dispatch. In this model, recognise that there are 4 separate fragments of code: 1) the existing type, 2) the interface, 3) different implementations of the interface, 4) a scope function (with()) at the call-site to supply the required implementation (scope object).
data class Tweet(val tweet: String, val retweet: String) // Existing type (assume Tweet is 3rd party)
// Generic extension interfaces
interface Summary<T> {
fun T.summarise(): String // <- extension function receiver on T
fun T.info(): String = "Default Information" // <- default impl of extension fun receiver
}
interface Farewell<D> {
fun D.sayBye(): String
}
// Externally implemented interfaces provide implementations (aka Scope objects)
object TweetSummaryScope : Summary<Tweet> {
override fun Tweet.summarise(): String = "Tweet: $tweet Retweet: $retweet"
}
object AnyFarewellScope : Farewell<Any> {
override fun Any.sayBye(): String = "Tat ta"
}
// E.g. of a function that declares multiple externally implemented interfaces as requried context,
// and multiple generic types as required parameters
context(summary: Summary<T>, farewell: Farewell<D>)
fun <T, D> globalFuncWithMultipleContexts(toSummarise: T, toSayGoodbye: D) =
with(summary) { // can access T's methods here
with(farewell) { // can access D's methods here
"Summary is: ${toSummarise.summarise()}, Info: ${toSummarise.info()} Bye: ${toSayGoodbye.sayBye()}"
}
}
// in some test case class:
@Test
fun `test attaching type classes to existing types String and Tweet`() {
val tweet = Tweet("my tweet", "my retweet") // Create existing type
with(TweetSummaryScope) { // scope Summary<Tweet> to provide extra behaviour on Tweets
with(AnyFarewellScope) { // scope Farewell<Any> to provide extra behaviour on any object
// if two implementations for the same type class are scoped with 'with' e.g. consider another nested 'with(AnyFarewellScope2) {}'
// then the last implementation wins.
assertEquals("Tat ta", tweet.sayBye()) // Add behaviour to existing type 'String'
// example local scoped anonymous object, prevents pollution of global scope (just to demo the concept)
val anyFarewellScope2 = object : Farewell<Any> { override fun Any.sayBye(): String = "Bye bye" }
with(anyFarewellScope2) {
assertEquals("Bye bye", tweet.sayBye())
}
val expected = "Summary is: Tweet: my tweet Retweet: my retweet, Info: Default Information"
assertEquals("$expected Bye: Tat ta", globalFuncWithMultipleContexts(tweet, "Bye: "))
}
}
}Pros of the Kotllin 'scope' (dynamic dispatch) approach:
- Polymorphism
- You can dynamically / conditionally bring into scope multiple type-class implementations when needed using the
with(scopeOb)scope function (see the@testfor an example). - You can implement the same interface multiple times for a single parameterised type class (not doable with Rust static dispatch). Multiple scope object implementations can then be combined as needed to supply the required implementations.
- You can dynamically / conditionally bring into scope multiple type-class implementations when needed using the
- Reduced scope pollution
- The
anyFarewellScope2example shows how to reduce pollution of the global scope by creating an implementation as a local scoped variable.
- The
Cons of the Kotlin 'scope' (dynamic dispatch) approach:
- Complexity - The Rust approach is simpler and the API is more self-describing. Similarly, implementing interfaces directly on types is much simpler (but you then lose the ability to augment 3rd party types).
- Reduced discoverability - You need to be explicitly aware of and import type-classes in order to use them (e.g. as function args, as context parameters, and within scope functions). This means knowing "what's available here?" can be a problem in a poorly structured project. Interestingly, at the time of writing, Java is exploring type classes for Java which proposes a solution for this findability problem where type class instances can be queried-for and discovered using new 'witness' keywords. https://www.youtube.com/watch?v=Gz7Or9C0TpM
- Performance - dynamic dispatch incurs a small performance overhead.
Note that these externally implemented interfaces cannot access private or protected members of the type they are augmenting. I'm not sure this should be regarded as a con, however; If you need to access private members, you should have access to the src-code and you should extend/implement directly on the type itself.
Some languages allow you to extend the existing types using simple extension functions and properties to create blanket implementations of functions e.g., C# and Kotlin. Extension functions and extension properties can be grafted onto existing types even if you don't have access to the source code for those types. Note that this technique does not create a new behaviour-type as explained above, rather you are simply extending your existing core types and abstractions.
Tip
One of the primary intentions of extensions is to keep core abstractions small by not polluting them with your own customisations, this way the original abstractions keep their original behaviour (Andrey Breslav, original Kotlin language designer).
Notice that in the example below, swap is grafted onto the existing MutableList interface. All other references to MutableList can call swap creating a blanket implementation without polluting the original type. Similarly, all list implementations can access the lastIndex extension property. Note that extension functions and properties do not have access to private members of the types they extend.
// Kotlin
// Extension property on Kotlin's existing List Interface to store a custom description
var <T> List<T>.description: String
get() = this.description
set(newDescription): Unit { this.description = newDescription }
// Extension function on kotlin's existing MutableList Interface to swap specified elements
fun <E> MutableList<E>.swap(index1: Int, index2: Int): Unit {
val tmp = this[index1] // 'this' corresponds to the list
this[index1] = this[index2]
this[index2] = tmp
}
fun <E> consumer(mutableList: MutableList<E>, index1: Int, index2: Int) {
mutableList.swap(index1, index2)
mutableList.description = "My cool new list to store bikes"
println(mutableList.description)
}Some languages have have ‘Duck Typing’ (e.g. Go, Py) where interfaces are used as function parameters or return values, or to parameterise other types such as lists, but importantly, you do not need to explicitly define an interface on a type itself for that type to conform to the contract. For example, you wouldn't declare: class A implements interfaceB (Java) or impl traitB for typeA (Rust). Instead, a particular type implicitly implements an interface if has the corresponding methods available: “if it walks and swims like a duck, it’s probably a duck.” If the type checking occurs at compile time, it is often called ‘Structural Typing/Polymorphism’ eg Template classes in C++ and in Go. If type checking occurs at runtime and produces runtime errors, it is called "Duck typing" which is more common in dynamic languages.
Tip
The ease and speed of development that duck typing brings is great for scripting and smaller code bases, but as the size and complexity of the code increases, I would advise using scaffolding code for stronger compile time type checking (for Pythonistas, you can always introduce static typing such as Python’s optional type hints, and for Go, see the code in the 'Go Example' above for a tip to check a type implements an interface).
Some languages (Java, Kotlin, I'm not sure if there are others) allow you to seal interfaces and classes in order to restrict the range of allowed subtypes for inheritance and interface implementations. Sealing provides more control over inheritance alone.
Tip
Why sealing? Limiting the extensibility of types is a useful feature for API design and domain modelling when you need to control code reuse. So, why would you want to control code reuse? The answer is that restriction of subtypes to a known set helps comprehension, clarity, type misuse, and security. Sealing mitigates the need for defensive coding for unknown subtypes, and is useful for increasing the security of libraries. It also facilitates exhaustive when and/or switch case patterns for Algebraic Data Types (see ADTs below).
For Java devs: A final class can't be subclassed. A package-private class (the default in Java, no modifier) can only have subclasses in the same package, not from another package. This means you need to make a class public to extend from it (or protected for a different package in the same project). If an abstract base class is public, this allows uncontrolled extended by anyone, which we want to prevent. So, how do we allow public access but prevent free/uncontrolled class extension or interface implementation? - the answer is sealing;
Tip
In Java/Kotlin, the main motivation behind sealed classes and sealed interfaces is to have the possibility for a superclass or interface to be widely accessible, but not widely extensible, allowing only controlled extension where you need it. Sealing opens up several advantages for API design.
As a simple example, consider a sealed result class that has a fixed number of subtypes to indicate different kinds of results. You can react to the different results via exhaustive when/switch patterns. In another domain modelling example, consider subtypes of the class Vehicle - the author may be interested in the clarity of code that handles known subclasses such as Car and Lorry, and is not interested in writing defensive code for every unknown type of vehicle.
package org.vehicles_sealing;
// file MOT.java
public sealed interface MOT permits Car, Lorry {
int getMaxMOTIntervalMonths();
default int getMaxDistanceBetweenMOT(){
return 10_000;
}
}
// In a separate file, same package: file Vehicle.java
public abstract sealed class Vehicle permits Car, Lorry {
private final String registrationNumber;
public Vehicle(String registrationNumber) {
this.registrationNumber = registrationNumber;
}
public String getRegistrationNumber() {
return registrationNumber;
}
}
// In a separate file, same package: Car.java
public final class Car extends Vehicle implements MOT {
private final int numberOfSeats;
public Car(int numberOfSeats, String registrationNumber) {
super(registrationNumber);
this.numberOfSeats = numberOfSeats;
}
public int getNumberOfSeats() {
return numberOfSeats;
}
@Override
public int getMaxMOTIntervalMonths() {
return 12;
}
}
// In a separate file, same package: Lorry.java
// The use of non-sealed allows for controlled extension points where you may want them, e.g. Truck.
public non-sealed class Lorry extends Vehicle implements MOT {
private final int loadCapacity;
public Lorry(int loadCapacity, String registrationNumber) {
super(registrationNumber);
this.loadCapacity = loadCapacity;
}
public int getLoadCapacity() {
return loadCapacity;
}
@Override
public int getMaxMOTIntervalMonths() {
return 12;
}
// In a separate file, same package: Truck.java
public final class Truck extends Lorry {
private final boolean articulated;
public Truck(int loadCapacity, String registrationNumber, boolean articulated) {
super(loadCapacity, registrationNumber);
this.articulated = articulated;
}
public boolean isArticulated() {
return articulated;
}
}Corresponding Test case:
package org.vehicles_sealing;
import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.*;
class CarTest {
@Test
void carTests() {
Car car = new Car(4, "MyReg");
Lorry lorry = new Lorry(5_000, "lorryReg");
Truck truck = new Truck(10_000, "lorryReg", true);
assertEquals(4, car.getNumberOfSeats());
}
private void exhaustiveDemo(Vehicle v){
switch (v) {
case Car car -> {
System.out.println("its a car, MOT due: "+car.getMaxMOTIntervalMonths()+" or after "+
car.getMaxDistanceBetweenMOT()+" miles");
}
// truck must come before Lorry to be exhaustive, as Truck extends Lorry
case Truck truck -> {
System.out.println("its a truck, is it articulated: "+truck.isArticulated() +
" MOT due:"+truck.getMaxMOTIntervalMonths()+ " or after "+
truck.getMaxDistanceBetweenMOT()+" miles");
}
case Lorry lorry -> {
System.out.println("its a lorry, MOT due: "+lorry.getMaxMOTIntervalMonths() + " or after "+
lorry.getMaxDistanceBetweenMOT()+ " miles");
}
}
}
}Tip
A complex term that in practice simply means that one class contains or is passed an instance of another to use its capabilities.
Warning
Unless your language supports duck-typing, composition alone may not establish a polymorphic type without the addition of additional scaffolding code such as an interface or trait.
I feel demonstrating parametric polymorphism across a selection of languages is useful to demonstrate several related concepts such as interfaces/traits, composition, extension methods, union types and sealing. I'm not going to cover ad-hoc polymorphism and type-classes as that is explained in more detail above.
Arguably, Rust has one of the most powerful type systems available (I think Kotlin is comparable). We will start with Rust and follow with examples from other programming languages.
The detailed example below demonstrates:
- Default method implementations on traits.
- Optional method overriding.
- Trait composition and implementing multiple traits - see
Product. - Extension Trait to augment existing types - see
StringExt. - Generic functions and trait bounds - see
filter_items. - Trait objects and dynamic dispatch is not shown, this is covered in the section Ad-hoc Polymorphism and Type-Classes.
// Rust Parametric Polymorphism
use std::fmt;
// Printable trait with default and required methods
trait Printable {
// Default method implementation
fn pretty_print(&self) -> String {
format!("[Default pretty_print: {:?}]", self.format())
}
// Required method to implement
fn format(&self) -> String;
}
// Serializable trait with default methods
trait Serializable {
fn serialize(&self) -> String;
// Default validation method
fn validate(&self) -> Result<(), ValidationError> {
Ok(())
}
}
// Custom error type for validation
#[derive(Debug)]
struct ValidationError {
message: String,
}
impl fmt::Display for ValidationError {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(f, "{}", self.message)
}
}
impl std::error::Error for ValidationError {}
// Product struct implementing multiple traits
#[derive(Debug, Clone)]
struct Product {
name: String,
price: f64,
quantity: i32,
}
impl Printable for Product {
// Implementing required format method
fn format(&self) -> String {
format!(
"{} (Price: ${:.2}, Quantity: {})",
self.name, self.price, self.quantity
)
}
// Optional override of pretty_print
fn pretty_print(&self) -> String {
format!("[Product: {}]", self.format())
}
}
impl Serializable for Product {
fn serialize(&self) -> String {
format!(
"Product{{name={},price={:.2},quantity={}}}",
self.name, self.price, self.quantity
)
}
// Custom validation implementation
fn validate(&self) -> Result<(), ValidationError> {
if self.price < 0.0 {
return Err(ValidationError {
message: "Price cannot be negative".to_string(),
});
}
if self.quantity < 0 {
return Err(ValidationError {
message: "Quantity cannot be negative".to_string(),
});
}
Ok(())
}
}
// Display trait for pretty printing
impl fmt::Display for Product {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(f, "{}", self.format())
}
}
// Inventory management trait
trait InventoryManager {
fn is_low_stock(&self, threshold: i32) -> bool;
fn restock(&mut self, amount: i32);
}
impl InventoryManager for Product {
fn is_low_stock(&self, threshold: i32) -> bool {
self.quantity < threshold
}
fn restock(&mut self, amount: i32) {
self.quantity += amount;
println!("Restocked {} by {} units", self.name, amount);
}
}
// Extension trait for string manipulation
trait StringExt {
fn truncate(&self, max_length: usize) -> String;
fn word_count(&self) -> usize;
}
impl StringExt for str {
fn truncate(&self, max_length: usize) -> String {
if self.len() <= max_length {
self.to_string()
} else {
format!("{}...", &self[..max_length])
}
}
fn word_count(&self) -> usize {
self.split_whitespace().count()
}
}
// Generic filter function similar to Go's FilterItems
fn filter_items<T, F>(items: &[T], predicate: F) -> Vec<T>
where
// Trait bounds used to constrain the generics F and T
F: Fn(&T) -> bool,
T: Clone,
{
items
.iter()
.filter(|&item| predicate(item))
.cloned()
.collect()
}
fn main() {
// Create products
let laptop = Product {
name: "MacBook Pro".to_string(),
price: 1999.99,
quantity: 5,
};
let keyboard = Product {
name: "Mechanical Keyboard".to_string(),
price: 129.99,
quantity: 2,
};
// Demonstrate trait methods
println!("Pretty Print: {}", laptop.pretty_print());
println!("Serialized: {}", laptop.serialize());
// Validation demonstration
match laptop.validate() {
Ok(_) => println!("Validation passed"),
Err(e) => println!("Validation Error: {}", e),
}
// Demonstrate extension trait
let long_string = "This is a very long string that needs truncation";
println!("Truncated: {}", long_string.truncate(10));
println!("Word count: {}", long_string.word_count());
// Demonstrate polymorhpic filtering
let products = vec![laptop.clone(), keyboard.clone()];
let low_stock_products = filter_items(&products, |p| p.is_low_stock(3));
println!("Low Stock Products:");
for mut p in low_stock_products {
println!("{} - Quantity: {}", p.name, p.quantity);
p.restock(10);
}
}Kotlin provides a similar approach to Rust's polymorphic traits through a combination of:
- interfaces with default methods
- optional overrides
- extension functions for extending existing and custom types directly, whether on the type or an interface (this is a slightly different approach to Rust which requires a trait definition)
- Sum/Union type implemented with sealing to model choice (also available in Rust)
This combination is very powerful, facilitating type augmentation and dot completion for extension method findability.
package org.example
import org.example.ValidationResult.Success
import org.example.ValidationResult.Failure
// Kotlin Parametric Polymorphism
// Rust-like trait implemented with a Kotlin interface using default methods on interfaces & extension methods
interface Printable {
// Default method implementation similar to Rust trait methods
fun prettyPrint(): String {
return "[$this]"
}
// Abstract method that implementing classes must define
fun format(): String
}
interface Serializable {
fun serialize(): String
// Default Validation method
fun validate(): ValidationResult {
return Success("Validation passed")
}
}
// Extension function to add functionality to existing types
// This is similar to Rust's trait methods that can augment existing types
fun String.truncate(maxLength: Int): String {
return if (length <= maxLength) this
else substring(0, maxLength) + "..."
}
// Extension function with more complex logic (see section on Extension Functions)
fun String.wordCount(): Int {
return trim().split("\\s+".toRegex()).size
}
// Sealed class to demonstrate exhaustive behavior (see section on Sealing)
sealed class ValidationResult {
data class Success(val message: String) : ValidationResult()
data class Failure(val error: String) : ValidationResult()
}
interface InventoryManager {
fun isLowStock(threshold: Int): Boolean
fun restock(amount: Int)
}
data class Product(val name: String, val price: Float, var quantity: Int) :
Printable, Serializable, InventoryManager {
override fun format(): String {
return "$name (Price: $price Quantity: $quantity)"
}
override fun serialize(): String {
return "Product{name=$name,price=$price,quantity=$quantity}"
}
// Custom validation implementation
override fun validate(): ValidationResult {
return when {
name.isBlank() -> Failure("Name cannot be blank")
price < 0 -> Failure("Price cannot be negative")
else -> Success("Product is valid")
}
}
override fun isLowStock(threshold: Int): Boolean {
return this.quantity < threshold
}
override fun restock(amount: Int) {
// TODO make thread safe
this.quantity += amount
println("Restocked $name by $amount units")
}
}
// Demonstrate Extension method on an existing List interface - similar to a Rust extension trait
fun List<Product>.filterValidProducts(): List<Product> {
return filter { it.isLowStock(3) }
}
// Demonstration of type augmentation and trait-like behavior
fun main() {
// Create products
val laptop = Product("MacBook Pro", 1999.99f, 5)
val keyboard = Product("Mechanical Keyboard", 129.99f, 2)
// Demonstrate extension functions
val longString = "This is a very long string that needs truncation"
println(longString.truncate(10))
println(longString.wordCount())
// Validation demonstration
val validationResult = keyboard.validate()
when (validationResult) {
is Success -> println("Validation passed: ${validationResult.message}")
is Failure -> println("Validation failed: ${validationResult.error}")
}
// Demonstrate trait methods
println("Pretty Print: ${keyboard.prettyPrint()}")
println("Serialised: ${keyboard.serialize()}")
// Demonstrate polymorphic filtering
val products = listOf(keyboard, laptop)
println("Low stock items: ${products.filterValidProducts()}")
laptop.restock(10)
}While Rust-like traits are elegant and powerful, you can achieve a similar effect using a combination of interfaces with default method implementations, and a mixture of composition and inheritance. As you can see in the Java example below, there is a lot more boilerplate. Java's approach is still not as powerful as Rust and Kotlin's approaches which allow type-classes for automatic 'blanket implementations' on existing types and trait/extension-function inheritance.
Limitations:
- Extension of existing types from another package may not be possible (eg final classes).
- No trait or extension-function inheritance.
- May need to use a combination of composition or inheritance to augment existing types, which may not be as convenient as attaching type classes.
import java.util.List;
import java.util.concurrent.locks.ReentrantLock;
// Java Parametric Polymorphism
interface Printable {
// Default method implementation - similar to Rust trait methods
default String prettyPrint() {
return "[Default Pretty Print: " + this.format() + "]";
}
// Abstract method that implementing classes must define
String format();
}
interface Serializable {
String serialize();
default Result validate() {
return new Ok();
}
}
sealed interface Result permits Ok, Err { }
record Ok() implements Result {}
record Err(String err) implements Result {}
interface StringExtensions {
// Default method that can be "mixed-in" to any non-final class
default String truncate(int maxLength) {
if (this.toString().length() <= maxLength) {
return this.toString();
}
return this.toString().substring(0, maxLength) + "...";
}
}
// Because String is final, we can't extend String so we have to compose it
// in order to implement StringExtensions
record MyExtendedString(String str) implements StringExtensions {
@Override
public String truncate(int maxLength) {
if (str.length() <= maxLength) {
return str;
}
return str.substring(0, maxLength) + "...";
}
}
interface InventoryManager {
boolean isLowStock(int threshold);
void restock(int amount);
}
class Product implements Printable, Serializable, InventoryManager {
String name;
Float price;
Integer quantity;
ReentrantLock re;
public Product(String name, Float price, Integer quantity){
this.name = name;
this.price = price;
this.quantity = quantity;
this.re = new ReentrantLock(true);
}
@Override
public String toString(){
return this.name;
}
@Override
public String format() {
return this.name+" (Price: $"+this.price+", Quantity: "+quantity+")";
}
@Override
public String prettyPrint(){
return "Product: "+this.format();
}
@Override
public String serialize() {
return "";
}
@Override
public Result validate() {
if(this.price < 0.0f) {
return new Err("Price cannot be negative");
}
if(this.quantity < 0){
return new Err("Quantity cannot be negative");
}
return new Ok();
}
@Override
public boolean isLowStock(int threshold) {
return this.quantity < threshold;
}
@Override
public void restock(int amount) {
try {
re.lock();
this.quantity += amount;
} finally {
re.unlock();
}
System.out.println("Restocked "+name+ " by "+amount+" units");
}
}
// Demonstration class
public class Main {
public static void main(String[] args) {
// Create products
Product laptop = new Product("MacBook Pro", 1999.99f, 5);
Product keyboard = new Product("Mechanical keyboard", 129.99f, 2);
// Demonstrate trait/interface methods
System.out.println("Pretty Print: "+laptop.prettyPrint());
System.out.println("Serialize: "+laptop.serialize());
// Validation demo with polymorphic Result and Exhaustive pattern matching
switch (laptop.validate()){
case Ok _ -> System.out.println("Validation passed");
case Err err -> System.out.println("Validation err "+err);
}
// Extension Trait with Composition demo
// Unlike Rust, you can't augment an existing final class like String.
// To impl ExtendedString interface you need to compose the String.
MyExtendedString longString = new MyExtendedString("This is a very long string that needs truncation.");
System.out.println("Truncated: "+longString.truncate(10));
// Demonstrate Polymorphic filtering
List<Product> products = List.of(laptop, keyboard);
System.out.println("Low Stock Products:");
List<Product> lowStockProducts = products.stream().filter(p -> p.isLowStock(3)).toList();
lowStockProducts.forEach(p -> {
System.out.println(p+" - Quantity: "+p.quantity);
p.restock(10);
});
}
}In Go, you can create trait-like behaviour using 'mixin' structs to provide default implementations of interfaces, but note that 'mixin' structs are not true type-classes in that you cant attach new behaviour to existing 3rd party types for blanket implementations.
Limitations
- No extension/augmentation of existing types in another package.
- No trait inheritance.
- No Generic constraints.
// Go Parametric Polymorphism
package main
import (
"fmt"
"unicode"
)
// Interface for Printable trait
type Printable interface {
// Method that can have a default implementation
PrettyPrint() string
// Required method to implement
Format() string
}
// Interface for Serializable trait
type Serializable interface {
Serialize() string
// Default validation method
Validate() error
}
// Mixin struct to provide default implementations
type PrintableMixin struct{}
// Default implementation of PrettyPrint
func (p PrintableMixin) PrettyPrint() string {
return "Default Pretty Print"
}
// Validation mixin
type ValidatableMixin struct{}
// Default validation implementation
func (v ValidatableMixin) Validate() error {
return nil
}
// Custom error type for validation
type ValidationError struct {
Message string
}
func (e *ValidationError) Error() string {
return e.Message
}
// Product struct implementing multiple "traits"
type Product struct {
PrintableMixin
ValidatableMixin
Name string
Price float64
Quantity int
}
// Implementing required Format method
func (p *Product) Format() string {
return fmt.Sprintf("%s (Price: $%.2f, Quantity: %d)",
p.Name, p.Price, p.Quantity)
}
// Override PrettyPrint method
func (p *Product) PrettyPrint() string {
return fmt.Sprintf("[Product: %s]", p.Format())
}
// Custom Serialize method
func (p *Product) Serialize() string {
return fmt.Sprintf("Product{name=%s,price=%.2f,quantity=%d}",
p.Name, p.Price, p.Quantity)
}
// Custom Validate method
func (p *Product) Validate() error {
if p.Price < 0 {
return &ValidationError{"Price cannot be negative"}
}
if p.Quantity < 0 {
return &ValidationError{"Quantity cannot be negative"}
}
return nil
}
// Extension-like functionality using generics (Go 1.18+)
func FilterItems[T any](items []T, predicate func(T) bool) []T {
var filtered []T
for _, item := range items {
if predicate(item) {
filtered = append(filtered, item)
}
}
return filtered
}
// --------- 1. Wrapper type extension of existing type--------
type ExtendedString string
func (es ExtendedString) Truncate(maxLength int) string {
s := string(es)
if len(s) <= maxLength {
return s
}
return s[:maxLength] + "..."
}
// --------- 2. Interface based extension---------
type Capitializer interface {
Caps() string
}
type CapitalizeString struct { // no 'implements' needed
original string
}
// Implicit implementation of Capitializer.Caps()
func (cs CapitalizeString) Caps() string {
if cs.original == "" {
return cs.original
}
firstChar := []rune(cs.original)[0]
return string(unicode.ToUpper(firstChar)) + cs.original[len(string(unicode.ToLower(firstChar))):]
}
// --------- 3. Util function is more idiomatic in Go---------
func TruncateUtil(s string, maxLength int) string {
if len(s) <= maxLength {
return s
}
return s[:maxLength] + "..."
}
// Inventory management "trait"
type InventoryManager interface {
IsLowStock(threshold int) bool
Restock(amount int)
}
// Implement InventoryManager for Product
func (p *Product) IsLowStock(threshold int) bool {
return p.Quantity < threshold
}
func (p *Product) Restock(amount int) {
p.Quantity += amount
fmt.Printf("Restocked %s by %d units\n", p.Name, amount)
}
func main() {
// Create products
laptop := &Product{
Name: "MacBook Pro",
Price: 1999.99,
Quantity: 5,
}
keyboard := &Product{
Name: "Mechanical Keyboard",
Price: 129.99,
Quantity: 2,
}
// Demonstrate "trait" methods
fmt.Println("Pretty Print:", laptop.PrettyPrint())
fmt.Println("Serialized:", laptop.Serialize())
// Validation demonstration
if err := laptop.Validate(); err != nil {
fmt.Println("Validation Error:", err)
}
longString := "This is a very long string that needs truncation"
// -------- Demonstrate 1. Wrapper type extension of existing type --------
extString := ExtendedString(longString)
fmt.Println("Truncated:", extString.Truncate(10))
// -------- Demonstrate 2. Interface-based Extension--------
capitalized := CapitalizeString{original: longString}
// You can verify that capitalized type implements Capitalizer
// with the following trick, compiler errors if not:
var _ Capitializer = capitalized
fmt.Println("Caps:", capitalized.Caps())
// Filtering with generics
products := []*Product{laptop, keyboard}
lowStockProducts := FilterItems(products, func(p *Product) bool {
return p.IsLowStock(3)
})
fmt.Println("Low Stock Products:")
for _, p := range lowStockProducts {
fmt.Println(p.Name, "- Quantity:", p.Quantity)
p.Restock(10)
}
}Coming soon.
If you can do OOP, you can do DOP: DOP advocates for cleanly separating data from behaviour. Data is modelled using hierarchically nested structs/records/data-objects, and methods that operate on those data are typically extracted into top-level or module/package level functions. DOP tends to adopt more noun orientated naming approach whereas OPP tends to adopt a mix of nouns and verbs. DOP tends to violate the Open-Closed principle compared to OOPs Visitor pattern. For example, given a bunch of 3rd party types, with DOP you can exhaustively pattern match if a new 3rd party type was added and you would need to update your code to handle that new type (whereas visitor pattern can ignore the new type), so there is a trade-off. However, it can produce simpler code that is easier to reason about.
Tip
There is no need to extract logic such as data validation and invariant checking into utility functions, this logic should be co-located with your data types, in class/record constructors for example.
Here's an interesting comment I noticed after watching a Java update newscast from Nicolai Parlog from the Java DevRel team regarding whether to use DOP for the pending JDK standard JSON API (i.e. a core native Java standard library, not a 3rd party API like Jackson), paraphrasing: "A sealed interface hierarchy with record chosen over class implementations (i.e. a DOP approach) is not appropriate for the JDK JSON API because of the lack of record encapsulation (and extensibility via open-closed), even though it seems like a perfect fit. This is because the lack of encapsulation makes internal evolution tricky which is not appropriate for a for long-lived public/stable API like the proposed Java JSON API. However, the DOP approach is great for application code though."
TODO: CaseyM: Compile time OOP encapsulation hierarchies are bad for performance code such as physics simulations and games. Casey advocates for an 'NP Component System' for better performance where 'Discriminated Union' types such as RustEnums, Java Records, Kotlin Inline value classes and data classes used with 'match/swtich-case' expressions are more performant than 'Fat Structs' that mix multiple members of different types behind interfaces. The former enables better performance as calculations can be efficiently applied to a bunch of these (anaemic) types by architecting the system's core logic around more relevant 'engineering boundaries' that type-safe and have no dynamic-dispatch, rather than as individual polymorphic domain objects. A nice analogy is that of column vs row based storage in DBs (ECS - Entity Component System as an analogy to column based storage while row based storage is OOP).
Anaemic types that have one core value type, outlinking
https://youtu.be/wo84LFzx5nI?si=oaVFo33k59VX_5LW
The Dependency Rule states that: "Source code dependencies must point only inwards." This is illustrated in the following diagram.
On the right side of the diagram below ('Crossing Boundaries' and 'DI'), remember the following:
- Interpret the grey 'Flow of control & invocation' arrows as: The side with the arrow-foot calls a method on the arrow-head side. The arrow-foot side must therefore have access to the object on which to call the method.
- The grey 'Flow of control & invocation' arrows are 'wiggly' because they touch each box in the general flow of control.
- Blue arrows represent UML source code relationships such as 'implements' and 'has a..' for composition.
- Crossing boundaries goes in both ways, from outer to inner and vice-versa.
- For inner to call outer, use dependency inversion.
Tip
"For inner to call outer, use DI - Make the outer implement methods owned by the inner." Uncle DaveM.
- If you are confused by the direction of the arrows, remember that they point in the direction of source dependencies (imports, includes, composition), not in the direction of data flow.
- Structuring code this way does have an overhead - it takes more time and effort.
- As mentioned, this would be overkill for scripts. My advice would be to apply the rule pragmatically: Create a partial boundary instead so that it is easier to refactor as the code grows ('Extract Interface' refactoring)
Tip
For Pythonistas and Go fans, DIP and DI are less obvious because of their duck typing - there are fewer frameworks compared to statically typed languages. Nonetheless, DIP and DI are still very valid principles to follow in these languages.
Here's a simple example to help build a mental model ❤️ 🧠
- We have a
BigHeartsingleton at the centre of our architecture that implementsHeart. Outer layers such asLegscan callHeart.excercise()to inform the heart we're starting to exercise. To do this, our outer body parts can have a reference toHeartwithout breaking DIP - our outer layers only depend on our inner layer. - However, what if the Heart needs to broadcast out to our body parts its response to exercise? We do not want our
Heartto import specific body parts, that breaks DIP as our inner layer would depend on the outer layer. - To resolve this, we define the
HeartBeatReceieverinterface that defines theheartBeat()function within the inner source code package i.e. the one that ownsHeart. - All our body parts implement the
HeartBeatReceiverinterface and the correspondingheartBeat()method to respond to changes in heart beat . - In terms of source code dependencies, our heart does not depend on specific body parts, instead it depends on
HeartBeatReceiverreferences. - Notice that this enables a flow of control (method invocation) going from outer-to-inner (
BigHeartcallsHeartBeatReceiver.heartbeat(). - How does
BigHeartacquire references to itsHeartBeatReceiverimplementations? - This can be done by wiring all our dependencies together into a dependency graph, typically from the main function. If needed, we can use globally declared abstract factory implementations that serve up body part implementations.
- If you don't like the idea of sprinkling your domain objects with references to abstract factories, this is where Dependency Injection (DI) comes into play. Basically, a DI framework takes the role of wiring together the dependency graph - see IoC.
Here is our inner heart package, notice we have no dependencies on our outer bodyparts package - this doesn't break DIP.
// Java
package heart;
import java.util.List;
public class BigHeart implements Heart {
List<HeartBeatReceiver> heartBeatReceivers;
private int heartBeat = Heart.defaultRestingHeartBeat;
public BigHeart(List<HeartBeatReceiver> heartBeatReceivers){
this.heartBeatReceivers = heartBeatReceivers;
}
public int getHeartBeat(){
return this.heartBeat;
}
@Override
public void exercise() {
this.heartBeat += 10;
System.out.println("Heart: responding to exercise - raising pulse, heart beat: "+this.heartBeat);
for(HeartBeatReceiver receiver : this.heartBeatReceivers){
receiver.heartBeat();
}
}
@Override
public void think() {
this.heartBeat += 2;
System.out.println("Heart: responding to thinking - raising pulse, heart beat: "+this.heartBeat);
for(HeartBeatReceiver receiver : this.heartBeatReceivers){
receiver.heartBeat();
}
}
}package heart;
public interface Heart {
public static final int defaultRestingHeartBeat = 60;
public void exercise();
public void think();
}package heart;
public interface HeartBeatReceiver {
public void heartBeat();
}package heart;
import java.util.List;
public class HeartFactory {
private static Heart heart;
public static synchronized Heart create(List<HeartBeatReceiver> bodyParts){
if(heart == null){
heart = new BigHeart(bodyParts);
}
return heart;
}
public static Heart getHeart(){
return heart;
}
}Here is our outer bodyparts package. Notice the src code dependencies on our inner layer - this doesn't break DIP.
package bodyparts;
import heart.HeartBeatReceiver;
public class Arms implements HeartBeatReceiver {
@Override
public void heartBeat() {
System.out.println("Arms: hurting");
}
}package bodyparts;
import heart.HeartBeatReceiver;
public class Brain implements HeartBeatReceiver {
@Override
public void heartBeat() {
System.out.println("Brain: clearing");
}
}package bodyparts;
import heart.HeartBeatReceiver;
public class Legs implements HeartBeatReceiver {
public void run(){
System.out.println("Legs: running");
HeartFactory.getHeart().exercise();
}
@Override
public void heartBeat() {
System.out.println("Legs: tiring");
}
}Here is our main class/module. Main belongs at the outer edge of our architecture and depends on our inner layers. Without an IoC container, this is where you 'wire-together' the application's dependency graph. If you use an IoC container, the container does this for you - hence we're giving control to the IoC.
import bodyparts.Arms;
import bodyparts.Brain;
import bodyparts.Legs;
import heart.Heart;
import heart.HeartBeatReceiver;
import heart.HeartFactory;
import java.util.List;
public class Main {
public static void main(String[] args) {
Legs legs = new Legs();
Arms arms = new Arms();
Brain brain = new Brain();
List<HeartBeatReceiver> bodyParts = List.of(legs, arms, brain);
Heart heart = HeartFactory.create(bodyParts);
legs.run();
System.out.println("\nMain: Thinking");
heart.think();
}
}The application prints the following:
Legs: Running
Heart: responding to exercise - raising pulse, heart beat: 70
Legs: tiring
Arms: hurting
Brain: clearing
Main: Thinking
Heart: responding to thinking - raising pulse, heart beat: 72
Legs: tiring
Arms: hurting
Brain: clearing
Basically, this means that your application code does not itself create instances of business objects directly. Instead, these objects are created separately by 'wiring logic' that sits outside of your immediate business/domain code. As a result, business and domain objects are then automatically injected into other business/domain objects. Injection occurs via class-constructors for required dependencies, or setter methods for optional dependencies. The wiring code can be implemented manually by passing dependencies to objects. This typically occurs: from a main module/function, via the Factory Pattern, or using Inversion of Control containers such as Spring.io that manage the wiring for you.
For the most part, the lifetime of a class is typically either singleton-scoped a.k.a., ‘application scoped’, where a single instance is created and managed (which means it must be thread-safe), or ‘prototype-scoped,’ where a new instance is always created & injected. However, there are several other specialised lifetimes such as ‘session-scope’ or 'transaction-scope' depending on your requirement.
So, what is the benefit of IoC?
- Separation of Concerns: First, the ‘wiring logic’ is cleanly separated from your domain logic; wiring logic resides in Factory classes or is managed by the IoC container.
- Decreased Code Coupling: This really is an excellent approach for decreasing code coupling if you use abstractions/interfaces. Dependent code only needs to know about higher-level abstractions, not concrete details. This really helps reduce complexity.
- Late Binding: By not depending on concrete classes, it creates the opportunity for IoC to read in class libraries that have been dynamically dropped into your server at runtime without having to make any changes to your logic (see late-binding).
- Advanced Life Cycle Management: The wiring code in your IoC container or Factory can be used to manage the life cycle of your dependencies which enables several possible optimisations and actions. For example:
- 'Just in time' creation of objects enables late-binding for dynamic use-cases such as dropping in new libraries without server re-start,
- Objects can be cached within object pools which helps reduce potentially expensive object creation for increased performance.
- Managing the life cycle of objects outside of your injected classes provides a 'point-cut' for custom actions. For example, when an object is returned to the pool, you can fire an event, for example.
- Object destruction can be handled under an external managed process.
- More effective testing: You can mock different dependencies/classes and inject them into your objects as needed to test different scenarios. A very simple example from Dave Farley’s ‘Modern Software Engineering’ book is given below:
// Java
// Two versions of Car and BetterCar,
// after Dave Farley's Modern Software Engineering Book.
// BetterCar has constructor injected engine which is easier to test.
class Car {
private final Engine engine = new PetrolEngine();
public void start() {
applyBrakes();
putIntoPark();
this.engine.start();
}
private void applyBrakes(){ /*...*/ }
private void putIntoPark() { /*...*/ }
}
class BetterCar {
private final Engine engine;
public BetterCar(Engine engine) {
this.engine = engine;
}
public void start() {
applyBrakes();
putIntoPark();
this.engine.start();
}
private void applyBrakes(){ /*...*/ }
private void putIntoPark() { /*...*/ }
}For the Car class, unless we decide to break encapsulation and make engine public, you can’t test the engine separately. Our BetterCar allows you to mock or create a different engine implementation and test that separately as shown below:
@Test
public void shouldStartBetterCarEngine() {
FakeEngine engine = new FakeEngine();
BetterCar car = new BetterCar(engine);
car.start();
assertTrue(engine.startedSuccessfully());
}In the example above, lets assume we remove the interfaces so that we deal just with concrete types, we could then define a circular dependency; body part implementations depend on the heart implementation and vice-versa. This raises the question; how do we give our heart implementation a list of concrete body parts, and how can our body parts be given a reference to our heart. If we use constructors for dependency injection, we can't create create one without creating the other because our dependency graph has a circular dependency, it is not acyclic. So, how can we address circular dependencies:
- Remove circular dependencies: I agree with most of what Rob Pike recommends in his justification for not supporting circular dependencies, but I also think accommodating circular dependencies sometimes does crop up in certain use-cases, most commonly at the class level within a single layer.
- Setter Injection: Body part implementations could receive a heart through a setter method rather than through their constructors. This enables us to create our body parts first, our heart can be created second, and third we pass our heart to each body part. This is OK in some scenarios if the dependency is optional. However, the heart is required and we do not want to create a body part in a partially complete state.
- Lazy Initialisation via the Static Method Pattern: Body Parts can request a reference to a heart through a static factory method. This allows us to create body parts and heart independently; the heart is created lazily the first time a reference is requested. Static methods are sometimes considered as being dirty because they can break encapsulation, create tight coupling between classes, and make code harder to test and maintain. https://www.baeldung.com/cs/factory-method-vs-factory-vs-abstract-factory
- Lazy Initialisation via Static Factory Pattern and Abstract Factory: Creation of objects and aggregates of objects is managed by a factory service.
- Lazy Initialisation via Lazy Metadata/Annotations: If you are using an Inversion of Control container for dependency injection, the IoC container likely provides annotations to indicate the order of initialisation. For example, Spring provides the
@Lazyannotation that can be used on dependencies declared at the injection site to delay their creation until they are needed. This can be very effective, you don't then introduce static methods which some consider as being a dirty approach.
// Java
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Lazy;
import org.springframework.stereotype.Component;
@Component
public class BeanA {
private BeanB beanB;
@Autowired
public BeanA(@Lazy BeanB beanB) {
this.beanB = beanB;
}
}Tip
One situation where I find separate Factory classes appropriate is for returning an aggregate (see DDD) when the final object you are creating relies on several other objects as dependencies.
This is a big topic, but as ever, there are trade-offs between the different approaches which ultimately depends on your use-case:
- Dynamic/late binding: Allows new code to be deployed without having to re-build and re-start your runtime. This literally means that your application code can evolve over time without re-building and re-starting your application. To do this, you require a dynamic language and runtime such as languages built on the JVM, JS, C# Python and Ruby. A classic example of a dynamic application is Minecraft where player-made mods or custom maps and APIs can be added to a running Minecraft realm. Another example where late binding is for applications that have plugins, this is common in business applications that need to be easily extended such as adding new file parsers or algorithms without having to re-build and re-start the app server. Typically, new functionalities are loaded using class loaders through special pluggable library files e.g.,
.jarfiles containing dynamic proxies and implementations of Service Provider Interfaces (SPIs). For these types of use-case, late binding is very powerful, but in my opinion, for the majority of applications, it is overkill. Late binding is commonly implemented using a technique called reflection which is not as performant as static binding because code needs to be introspected in order to invoke the new functionality. Polymorphism also allows method overriding for new sub-types to be loaded at runtime using dynamic dispatch. A classic example is dynamically loaded component pallets in a CAD application containing new object sub-types.
Classic examples of dynamic applications include:
-
Game Engines like Unity and Unreal Engine use dynamic binding to load and unload game assets, scripts, and plugins at runtime which enables modular game development and creation of customizable games.
-
Web browsers that interpret and execute JavaScript code dynamically to enable web sites to create interactive web pages.
-
AI and Machine Learning Frameworks often use dynamic binding to load and execute different models and algorithms at runtime, allowing for experimentation and customization
-
Virtual machines like the JVM and the .NET CLR (Common Language Runtime) use dynamic binding to load and execute classes at runtime.
-
IDEs like Intellij IDEA use dynamic binding for their plugins.
-
Frameworks like Spring often rely on dynamic binding to allow developers to extend the framework's behaviour without modifying its core code. For example, you can create custom components and inject them into the framework's configuration.
- DI/IoC containers have changed a lot recently. In the past, they have largely been built for dynamic binding using reflection, but these days you can choose the IoC implementation depending on your requirements, dynamic or static.
-
Security applications for applying patches and updates without restarting.
-
Operating systems like Windows and Linux use dynamic loading to load device drivers and system libraries at runtime.
-
Static binding: If you do not need to modify runtime behaviour with plugins or highly dynamic late binding, for the majority of simpler use cases choose static binding for more performant Ahead of Time (AOT) compilation. Late binding is generally more secure considering the additional potential to load malicious code.
Nuff said. Just to be clear, you should not be able to construct an object in an invalid state. There are several ways to prevent invalid object creation, including the following approaches (I'm sure there are others too):
For example, you could throw an IllegalArgumentException runtime exception from within a constructor. However, remember that exceptions should be reserved for truly exceptional circumstances and often it is better to model invalid states explicitly in your type system as potential domain errors. In this case, consider the following approaches.
In the example below, we use a Kotlin smart constructor to hide the primary constructor and model an invalid author name explicitly in our type system. This way, we create an Author object using a familiar constructor call syntax and is clear that a potential outcome is an InvalidAuthorName. You can of course apply the same approach in the standard factory pattern.
// Kotlin
object InvalidAuthorName
data class Author private constructor(val name: String) {
companion object {
operator fun invoke(name: String): Either<InvalidAuthorName, Author> = TODO()
}
}
val authorResult = Author('Homer')
var? author = when (authorResult) {
InvalidAuthorName -> TODO() // return early, throw runtime ex, return null
// (Kotlin models null explicitly for us with var? syntax)
Author -> authorResult.value
} See the builder pattern. The final build() function should be used to check complex invariants and can return a smarter return type such as an Either or an ADT. An invariant is a condition that must hold true typically across multiple state variables.
There might be a tried & tested design pattern for the problem you’re tackling. Some patterns are probably overkill, but some genuinely useful patterns include Factory, DTO, Observer, Strategy, Singleton, Repository, Stateless Façade, Visitor. Have a look at the recommended texts in the appendix. Gang of Four (GoF) is kinda regarded as old school these days.
This pattern abstracts logic behind a common abstraction such as a SAM interface (Single Abstract Method interface) so that an implementation can be chosen at runtime. This makes the code more flexible and reusable. In the Kotlin example below taken from Dave Leeds, we use validation as an example, where any of the validators can be passed at runtime to the FormField class.
Here are two more Kotlin examples that are more idiomatic which reduce boilerplate, again from Dave Leeds:
An even more concise example:
Note you can use an extension function to easily create an optional version:
At the call site:
The visitor pattern is used to separate business logic from objects on which they operate. Typically, objects define an accept method then call method(s) on the accepted visitor. The calling object is typically passed to the visitor as an argument so the visitor can access the object's public state, as in the pseudo code: accept(Visitor v) { v.visitDoLogic(this); }. New logic can easily be added to the visitor's visitDoLogic(callerObj) without having to update the calling objects which illustrates an example of the open closed principle in SOLID. This pattern uses a double-dispatch logic: first an object's accept(Visitor) method is invoked, then the visitor's visitDoLogic(obj) method second.
The visitor pattern is typically invoked for large cascading / nested object trees; an accept method can pass the visitor instance to all its member objects that also define an accept method, for example:
// C#
public class Addition : Expression {
public Addidtion(Expression left, Expression right){
Left = left;
Right = right;
}
public override void Accept(Visitor v) {
Left.accept(v);
Right.accept(v);
v.vist(this);
}
// get values etc elided
}
// invoking code would create a Visitor implementation and invoke the double dispatch logic by calling `Addition.Accept(visitor);`Languages implement the visitor differently. For strongly typed polymorphic languages that support method overloading (Java, C#, Kotlin), interfaces can be used simplify the double dispatch logic where accept and visit methods can be overloaded using different argument types. Languages that do not support polymorphic overrides e.g., Go and Python, typically need to define different visit-method names e.g.
// Go
type Visitor interface {
visitWheel(wheel Wheel) string
visitEngine(engine Engine) string
visitBody(body Body) string
visitCar(car Car) string
}Builders are especially useful if the dependencies of your class have complex invariants. Basically, this means that if you class can only be constructed with a particularly complex combination of dependencies such as ‘my object requires A and B and either C, D, or E and F, but never G if D is present’ (I’m sure you get the idea), then the Builder pattern can help you. How to implement this in your chosen language varies of course e.g., in Go check out the Functional Options Pattern where you pass functions to modify the state of a struct. This is a nicely explained example, but it has some issues, first it lacks a build() function that is typically used to validate invariants before returning a valid 'built' instance. Second, having a bunch of standalone functions in the Functional Options pattern gives poor discoverability, it can make more sense to use a Builder type bunch of setter functions to set invariants - this means you can easily discover and chain setters using dot notation and code-completion e.g. with pseudo code:
configOps = NewConfigOptionsBuilder().id("someId").maxConnections(10).prefix("somePrefix").build();
e.g. builder pattern in Golang
todo
Your first instinct should be to make a method/member/variable private first, then increase visibility as required, not the other way around.
Bugs can't hide in simplicity.
Duplicating chunks of code is odorous - don’t do it.
Following Agile processes (i.e., ‘Feedback Driven Development’) should trap and prevent unnecessary code.
You don’t retrospectively comment your code, you just don’t. Using sensible names should prevent long-winded doc strings. Use xDoc tools e.g., PyDoc, JavaDoc, xDoc etc. Document the intent of the function/class, not the implementation details.
Leave code in a better state than you found it & don’t comment bad code, re-write it with good descriptive names.
🚃💥🚃💥🚃💥🚃
A module should not know about the innards of the objects it manipulates. Important: by ‘objects’, I mean objects that have state and methods that operate on that state. It is quite normal to have deeply nested data objects/structs/records call methods to access data - for these types of data carrying objects, the Law does not apply. https://en.wikipedia.org/wiki/Law_of_Demeter
Tip
Method f of class C should only call the methods on these:
C- An object created by
f - An object passed as an argument to
f - An object held in an instance variable of
C
f should not invoke methods of any other objects returned by any of these approaches. In other words, talk to friends, not to strangers.
To quote Eric Evans, an expert in both OOP and Functional paradigms and of 'Domain Driven Design' book fame: "At times, I found FP an awkward fit. The problem would have fit OOP better. I'm happy that it is easier than it used to be to move between both those ways of thinking."
I completely agree with Eric - if the language permits, combining functional approaches with OOP is powerful in the right scenario. In fact, modern languages are more of a hybrid mix of OOP, Procedural & FP e.g. Rust, Kotlin, modern Java (others languages too). Hybrid approaches borrow concepts from FP such as ADTs, default immutability, functional composition such as map filter collect.
For the majority of developers, myself included, I also believe that adopting a pure functional language is a stretch - their popularity are still low representing ~5% of mind share according to IEEE, Tiobe. I think the primary reason is that people just comprehend the world more readily in terms of objects, state and procedures, compared to purely functional/mathematical ways of thinking such as recursion and 'higher order functions'. To boil it down further, basically loops are difficult or even unavailable in pure FP, and I think allowing some interior mutability is often a small price to pay for simplicity.
Therefore, if your language, my recommendations are to mix some functional concepts when appropriate:
Tip
- Use a naming convention to identify pure functions, see using 'Calculations to limit side effects'
- Push out side effects to the out boundaries of your code (see the Dependency Rule / Bullseye) so they become 'intended effects’ and not nasty interleaved ‘side effects’.
- Aim for a core of pure functions.
- Aim for immutability as your default.
Be Careful Not to Pollute Pure Functions with Hidden Mutable State
Pure functions need to remain pure; you really don’t want to pollute your pure functions with hidden shared mutable state across threads. Consider the following example - one is broken, the other is OK, the difference is subtle. So, while combing FN + OOP is powerful, be very careful.
ParallelStream will split work across a thread pool, this means the list, which is not atomic and is our external mutable state, is subject to a whole host of complex threading issues (overwrites, ghost-reads, race conditions etc). Replacing the forEach with a functional ‘reducer’ operation (e.g., toList()) and converting the call chain from a statement to an expression solves the issue - there are no statements in pure functional code.
Modern languages make class members private by default, at the individual class level. Older languages like Java make member variables private at the wider package level by default, which is too visible in my opinion. Recommendation is to make members private at the class level.
- Shared and mutable global State is evil and will cause you problems in a concurrent environment.
- Modern languages are immutable by default. For example, in Rust and Kotlin, you need to specifically ‘opt into’ mutable variables using special keywords such as ‘mut’ (Rust) and ‘val’ (Kotlin) which define immutable state instead of ‘var’ which is mutable.
- Return immutable defensive copies of data from methods and functions rather than the original mutable data (e.g. return a immutable copy of an object from a HashMap instead of the actual object). Defensive immutable copies are thread-safe, while our mutable references stored within our HashMap can stay safely encapsulated.
- This is very much a Rust way of thinking - in Rust you have to think more deeply about who owns what and when. More specifically, Rust allows only one mutable reference to be valid at any one time OR multiple immutable references, but never both (one mutable & many imutable) at the same time. It is possible to manually represent this concept in other languages, as described our HashMap example in the previous bullet (note however, that this example does not go as far as borrowing vs ownership when passing data to functions).
When mutability is necessary, try to encapsulate it within a function so any mutable state is not leaked. This is known as ‘Interior Mutability’. To do this, you’ll likely need to take defensive copies of the input parameters to reduce the risk of side effects.
Calculations or ‘pure functions’ have no side effects and are ‘idempotent’. This means given the same input arguments, idempotent functions will return the same value regardless of when you call them (incidentally, this means you can cache the results of expensive calls in a lookup table, for example). Operations may have side effects such as updating a database, writing a file to disk, calling a remote service, even printing to the console is a side effect (maybe another process is reading your stdout/err?). Operations return values may change depending on when you call them. For example, consider calling a travel service with the same function parameters – results will probably vary depending on the time of year or time of day. One approach to help make code that depends on randomness (e.g., a random number is required), is to provide a seed value, and make that part of the external API which allows you to test using the same seed values.
See prior bullet. Functions should either do something such as create side effects (operations) or provide an answer to something such as returning a result from a stateless calculation. Try to separate functions that have side-effects from pure functions (aka calculations) using a naming convention. Try not to do both in a single function. In some languages you can be explicit– e.g., in Kotlin, a common convention is to reserve single expression functions only for calculations – you can quickly/easily see this in the first line of the expression function signature – no need to understand the function body for side effects. Nevertheless, unless you are using a pure functional language, this is still only a convention.
Functions that only have string arguments and string return types aren't great, please try to avoid them, they are just too loosely typed - a string can literally describe anything. I concede that string input parameters are sometimes necessary, but often you can constrain input parameters and return types beyond strings. One great tool used to describe types are Algebraic Data Types. Please read on.
ADTs combine ‘Product Types’ for modelling aggregation such as a C/Golang/Rust ‘structs’ or Java's Record type with ‘Sum Types’ for modelling choice, also known as ‘Union Types’ or ‘Tagged Unions’. This simple combination of aggregation and choice is deceptively powerful and shows up in many programming languages to model data i.e., domains, return types and function arguments:
-
Product Types are great for modelling aggregation, and include immutable data classes such as records, data objects, and structs. They are called ‘Product Types’ because their state ‘when considered as a whole’ is the Cartesian product of their data.
-
Sum Types can be used to represent choice and are polymorphic - an abstraction such as a marker/type interface with a fixed set of implementing subtypes (e.g., ‘sealed’ classes or interfaces in Kotlin/Java, ‘enum’ in Rust, ‘Union’ types in Python). They are called Sum Types because the set of possible types is the sum (union) of the total allowable set.
In some modern languages (e.g Rust, modern Java, others), ADTs can be efficiently processed using de-structuring and exhaustive pattern matching with when & switch statements. Exhaustive matching means the compiler will generate a compilation error if not all types are explicitly handled. As highlighted by Gavin Bierman in his Devoxx talk, you can spot many 'lightly disguised abstractions such as JSON' and model them as ADTs as shown in the following diagram:
From Gavin Bierman's Devoxx talk, Java Language Futures: https://www.youtube.com/watch?v=NNPN5tvjzqA&t Using the sealed interface, it would be simple to permit a range of additional custom types that implement JsonValue such as MyCustomString and ThingArray for example.
-
Unrecoverable / Fatal: Is the error recoverable? If not, then let the program crash/panic/throw (runtime exception). For example, a
FileNotFoundExceptionshould crash if that file is the application's mandatory config file - you can't continue without it, but for a file browser application, probably not. -
Recoverable problems: For example, if a remote service is temporarily unavailable, you could introduce a retry before showing an error to the user.
-
Errors that need to propagate to the user: An error-as-value would be suitable if you are building a file-explorer GUI - you don’t want your program to crash if a file gets deleted by another process. In this scenario use a value-error or catch the exception and convey a sensible message to the user.
-
Programming Mistakes / Boneheaded Exceptions: Let the program crash/panic/throw (runtime exception), you’ll be motivated to fix the problem quickly.
These problems are similar to Eric Lippert's Four exceptions of the Apocalypse (former designer of original C# compiler and language designer):
-
Fatal Exceptions RIP (e.g. OOM, rare, don't catch these)
-
Boneheaded Exceptions & Coding bugs (don't catch these either)
-
Vexing Exceptions - Those exceptions that aren't really exceptional and result from unfortunate/poor design choices. Advice is to always handle these i.e. catch close to cause and handle/contain or re-throw something more appropriate to your application layer.
-
Exogenous ie External Exceptions - These are external exceptions that are beyond your control, e.g. from interacting with an external system. Advice is to catch and handle these, potentially wrapping them and add extra context for re-throwing.
Tip
Guard clauses should be defined early - do not define lengthy conditional success blocks when you have can have short fail blocks defined early.
// bad ❌
fun badGuardClauseExample(val arg1)
if(valid(arg1)){
...success path...
} else {
... error handling...
}
}
// better ✅
fun betterGuardClauseExample(val arg1)
if(notValid(arg1)){
// ... error handling ...
}
... success path...
}Consider the following scenario: Within your application boundary, null is passed as function parameter when a list is expected. Within the function, defensive checking and converting from null to an empty list might seem an appropriate strategy. Nope, this can potentially hide higher level bugs. You have to ask yourself Q. 'what is the intention of declaring a list object as a parameter?' A. It is not to accept null. Throwing an IllegalArgumentException with a good contextual message is appropriate in this scenario.
Tip
- Polluting your 'inner' business logic with defensive checks is unnecessary and can obfuscate genuine bugs.
- Validate at the edge of your application: Postel's Law applies at the edge of your application, at the boundary when receiving incoming data: "be liberal in what you accept and conservative in what you return."
- Being defensive when using 3rd party libs is also OK.
Note, the application boundaries are illustrated in the application bullseye diagram, and in the classic 'Hexagonal Architecture'.
Don't assume you can if-check your way out of all scenarios. A good example is the TOCTOU scenario (time of check time of use) when checking if a file exists before doing something with the file - its better to use a try/catch block which is atomic and safer. Example:
// bad ❌
string TryGetFileContents(string path) {
if(!File.Exists(path))
return null;
// file could be deleted here by other process
var stream = File.OpenRead(path);
...do something here...
}
// better ✅
string TryGetFileContents(string path) {
try {
var stream = File.OpenRead(path)
... do something here...
} catch (FileNotFoundException ex){
return null;
}
}Handling null depends on the language and programming style you are using:
-
Nullable languages (C/C++/Java): Dereferencing a null pointer causes bad things to happen. This is known as ‘the billion-dollar mistake’ coined by Tony Hoare, in 1965. In your code, be sure to make it clear when null is meant to represent the ‘absence of value’ (e.g., with @Nullable annotations for example). At the boundary between layers of your code and when using a 3rd party library for example, it is OK to be ‘defensive’ and check for nulls.
-
Many argue null is an acceptable way to represent the absence of value, it’s just a fact and is too fundamental in many layers. They argue the real billion-dollar mistake is not null itself, but in the failure of the language to do type-safe handling of null.
-
Some languages e.g., Kotlin, Rust & Python have ‘safe nullability’ baked into their type-systems e.g., None to mean no value and optional ‘?’ on variables (‘var?’) to indicate this variable might be null.
-
Functional languages commonly use
Eithermonads to wrap errors andOptionalfor the absence of value. With these monads, the programmer is forced to handle the occurrence of no value or an error, typically using a pattern-matching style of syntax which means errors can’t be ignored, mistakenly or otherwise. -
Various modern languages use Algebraic Data Types (an extension of the Special Case pattern and return types that wrap either a successful result or an error (Go, Rust, Kotlin, modern Java).
-
ADTs combine ‘Sum Types’ and ‘Product Types’ and are excellent for representing multiple special cases, including multiple error states.
Tip
If a value can legitimately have no value, ensure safe nullability. In order of preference: 1. Use the languages native safe nullability type system (e.g. var? in Kotlin); 2. Use known zero values, Optionals, ADTs to model absence of value explicitly; 3. If your language's type system does not support safe nullability implicitly, make it clear when null is meant to represent absence of value e.g. with @Nullable annotations.
Tip
Don't initialise struct values with null.
Errors as values vs exceptions is a hotly debated topic in programming communities. There are pros and cons to each approach as discussed below. My personal opinion is that if used correctly, (runtime) exceptions are an effective error handling strategy, and there are compelling reasons why exceptions are idiomatic in many popular languages - language designers aren't dumb. However, I also recognise that exceptions can be both intentionally abused and used incorrectly, and so care should be used in their application. I also recommend that errors-as-values should be used to model known business errors to 'make (known) invalid states unrepresentable' in your application - the two approaches need not be mutually exclusive.
Example Errors-as-Values:
// Go
package main
import (
"errors"
"fmt"
)
func f(arg int) (int, error) {
if arg == 42 {
return -1, errors.New("can't work with 42")
}
return arg, nil
}
func main() {
for _, i := range []int{7, 42} {
if r, e := f(i); e != nil {
fmt.Println("f failed:", e)
} else {
fmt.Println("f worked:", r)
}
}
}Example Throwing of Runtime Exception:
// Java
import java.util.Arrays;
import java.util.List;
public class Main {
public static int f(int arg){
if(arg == 42) {
throw new IllegalArgumentException("can't work with 42");
}
return arg;
}
public static void main(String[] args) {
List<Integer> list = Arrays.asList(7, 42);
for(Integer i : list){
try {
int r = Main.f(i);
System.out.println("f worked "+r);
}catch(IllegalArgumentException ex){
System.out.println("f failed");
}
}
}
}-
Fans of errors-as-values argue that functions should return either a success value OR a failure value for known business errors. In doing this, the potential for failure is made explicit in a function signature. I agree - it is is commonly regarded as the more reliable approach to handling known business errors because you are forced to handle errors, typically using a conditional to test for error or success. This ensures error handling is not an afterthought.
-
Supporters also argue that there is less uncertainty compared to throwing exceptions because it can be challenging to determine all the exception types that can be thrown by a deep call stack. This has given rise to some saying: 'Exceptions suck because they are a non-local goto.' Also recognise that unhandled (runtime) exceptions do not create compilation errors, meaning the compiler can't help you discover all of the different types of exception that could be thrown. You often need to dig and read the docs of the APIs you are using. Personally, I disagree with the 'non-local goto' opinion - we're not really jumping to another location in the code to continue with the application logic, instead we're going back the way we came, unrolling the call-stack.
-
Another issue of a specific type of exception known as a 'checked' exception is that they prevent functional composition. This is because the compiler forces you to handle checked exceptions wherever they can be thrown, but they are not considered as part of a function's return signature and type system. Instead, exceptions invoke orthogonal flows that 'break out' of your regular functional flow. Checked exceptions therefore breaks 'referential transparency' (see discussion below on Error Monads such as
Either&Validated). Checked exceptions are generally not recommended these days, except for certain special use-cases where they still have their supporters.
-
Fans of exceptions argue that by forcing you to interleave error checking at function call sites throughout your code obscures the code's happy path and readability. I tend to agree, especially when adding boilerplate to manually pass an error back up the call stack.
-
Exception fans also argue that exceptions centralise your error handling code which gives a clean separation of concerns.
-
For low-level code, exceptions are largely considered an effective strategy for surfacing underlying issues such as low level operating system issues which may be mistakenly obscured by the errors-as-values pattern (although the same could be said by mindlessly catching all exceptions).
-
When used correctly and with discipline, exceptions can also be more performant than interleaved error-value checking. This is because languages like C++ and Java have 'zero cost exception handling.' I think this is a misleading term, what it actually means is 'zero cost to the happy path provided no exceptions are thrown.' Assuming no exceptions are thrown, quite simply, there is less for your code to do as there are no interleaved conditional error checks. While any performance hit from interleaved result checking is likely to be marginal for the majority of use-cases, it may become more pronounced in deeply nested code or tight compute loops. However, this can be mitigated with good code structuring by moving error checks out of and before any performance critical-sections.
Tip
Quite simply: use errors-as-values to model known business errors, and runtime exceptions only for genuine exceptional conditions and programming bugs.
Tip
try/catch and panic/recover are very different strategies, a key difference is the resulting control flow after they are triggered and opportunity to react to an exception.
In a try/catch/finally block, unless you re-throw or return from within the catch or finally block, code coming after the try/catch/finally block will still execute. This does not happen with panic/recover - a function that is aborted via panic begins to unwind the stack, running deferred blocks/functions as it encounters them (in Go, this is the only place recovery takes affect, although use of recover is not widespread in Go and panic is typically used to end a program). Thus, panic/recover is very different to try/catch stemming out of the fact that it is built around deferred logic as a recovery mechanism (e.g. Go & Zig).
As presented by the AWS Prime Video app developers in their experiences in re-writing the AWS TV app from JS/C++ to Rust/Wasm, "writing panic free code is really hard." Despite all the benefits of the re-write to Rust, the "lack of exceptions has been a real pain-point" for them as a panicking app crashes without having the opportunity to catch and respond to an exception in a user friendly way.
Whether to use exceptions has profound implications on your API design and performance, be aware of the issues highlighted above. Some modern languages, e.g. Mojo, go as far as trying to address the choice for you by compiling exception handling code under-the-hood to use errors-as-values. I think the aim is to allow you cleanly separate the happy path from exception handling code (clean separate of concerns).
Of course, choice between exceptions or errors-as-values depends on the language and environment - you don't get exceptions support on every architecture and platform. The result pattern is much more flexible especially on embedded systems.
-
Yes, depending on your language of choice and what is considered idiomatic. Some modern languages support both approaches. For example, to support interoperability with Java, the Kotlin language supports unchecked exceptions as well as its own
Resulttype which is intended for low-level code rather than for modelling business errors. For modelling business errors, Jetbrains/Kotlin recommend using sealed class hierarchies and exhaustive pattern matching to handle errors (see discussion on data oriented programming). -
At the time of writing, a Kotlin team are working on a union type for capturing a result OR one or more errors: Kotlin roadmap and Rich Errors.
Error Handling - Exceptions Should Not be Used for Flow Control - Exceptional Does Not Mean Conditional
Passing around a deeply nested stack trace within conditional and control logic is very expensive, don't do it. Instead, model your (known) business errors as values (no need to pass around exceptions), and leave exceptions for coding errors and exceptional situations. If you want control flow logic that says "if success do this..., but if an error occurs then do this..." then use the result pattern.
Error Handling - Only use Exceptions for Exceptional Situations Such As Coding Errors and Unexpected Errors
For example, an invalid object posted to your API is not exceptional, this should be handled as a potential business error. In the situation where some code throws an exception such as a parse error, catch it locally, extract the useful information, and return an error-value. In general, the result-as-value pattern is appropriate where the problem is the fault of the caller and not a programming mistake e.g., invalid input / form data.
If you use exceptions (not all languages have exceptions e.g., Rust, Go), define Exceptions in terms of a caller’s needs and wrap 3rd party library APIs including their exceptions. Often, only a few custom exception classes are needed for a particular area of code.
Generally, pushing genuine runtime exception handling code (for unexpected problems) up to the ‘outer layers’ of your code toward the boundaries is usually a good approach. It also helps cleanly separate the ‘happy path’ from interleaving error handling code. However, this is not a hard rule, in some situations you may need to try/catch/finally at the source of the error to take important corrective actions such as closing an IO resource or rolling-back a DB transaction.
Don't add e.g. email addresses to exception messages, they will end up in a log and you don't want emails and usernames in logs.
Error Handling - Do not create custom exception types when the standard library exception types handle your cases
Assertions are for developers and for documenting the invariants of the system. For example, replace comments like 'this should never happen' with asserts. They are compiled out in release code anyway.
TODO
With ADTs, for any single function, you can replace thrown exceptions with return values using a single generic abstract data type that wraps the exception or error. For example, using a sealed interface, all possible success AND error variations can be modelled using polymorphism. Our abstract data type becomes an algebraic data type (ADT), also known as a 'nominal' or 'named' union type (an ADT provides the combination of aggregation and choice to model all possible variants). The ADT replaces exceptions with an 'errors-as-values,' approach that can include multiple optional success and error types, if required.
This is super-powerful because you can add new success and/or error/exception return types through polymorphism, and also intentionally restrict all calling clients to a single permissible set. This is invaluable for developers of libraries where you explicitly want to limit the return types of your library functions and prevent clients overriding with their own implementations.
To process the function's abstract return type with very little boilerplate, exhaustive pattern matching with switch ensures all possible variants are handled - the compiler will produce an error if any case is unhandled. There is a lot to unpack here, but the example given below from Gavin Bierman clearly demonstrates this approach using modern Java (2024). The use of ADTs with pattern matching is being coined in the Java community as 'Data Orientated Programming,' and can be applied for modelling many types of data type hierarchy (Gavin uses converting JSON to Java types as an example). I suspect that this approach will become very popular in the Java community in the future, moving away from exceptions in higher level application code (note, as discussed above, exceptions will still have their place in lower level library and framework code).
// (java 23 with previews enabled)
import java.util.concurrent.TimeoutException;
public class ResultsAsValuesDemoMain {
// main method prints:
// handledsuccess: success[result=all good]
// timeout: java.util.concurrent.timeoutexception: too long
// interrupted: interrupted[result=i was interrupted]
// failure: java.lang.exception: failed
public static <V> void main(String[] args) {
MyResult<V> result = updatedLegacyFunction(4L);
printResult(result);
result = updatedLegacyFunction(15L);
printResult(result);
result = updatedLegacyFunction(10L);
printResult(result);
result = updatedLegacyFunction(7L);
printResult(result);
}
static <V> void printResult(MyResult<V> result){
// Switch uses exhaustive pattern matching and deconstruction.
// A compile time error is generated if not all variations are handled.
switch (result) {
case Failure<V>(Throwable cause) -> System.out.println("Failure: "+cause);
case Interrupted<V> val -> System.out.println("Interrupted: "+val);
case Success<V> val -> System.out.println("Success: "+val);
case Timeout<V>(Throwable cause) -> System.out.println("Timeout: "+cause);
}
}
// legacy function that throws exceptions
static String legacyFunction(long timeout) throws InterruptedException, TimeoutException, Exception {
if (timeout < 5) return "All good";
else if(timeout > 10) throw new TimeoutException("Too long");
else if (timeout == 10) throw new InterruptedException("I was interrupted");
else throw new Exception("Failed");
}
// legacy function can be re-written to return an ADT 'error as result'
static <V> MyResult<V> updatedLegacyFunction(long timeout){
if (timeout < 5) return (MyResult<V>) new Success<>("All good");
else if(timeout > 10) return (MyResult<V>) new Timeout<>(new TimeoutException("Too long"));
else if (timeout == 10) return (MyResult<V>) new Interrupted<>("I was interrupted");
else return new Failure<>(new Exception("Failed"));
}
}
// MyResult<V> is an Algebraic Data Type.
// ADTs combine power of union/sum types for modelling choice with
// product types to model custom wrapped errors.
sealed interface MyResult<V> permits Success, Failure, Timeout, Interrupted { }
record Success<V>(V result) implements MyResult<V> {}
record Failure<V>(Throwable result) implements MyResult<V> {}
record Timeout<V>(Throwable result) implements MyResult<V> {}
record Interrupted<V>(V result) implements MyResult<V> {}Using ADTs to model better return types. After Gavin Bierman's Devoxx talk, Java Language Futures
Before I get to error monads such as Either and Validated, I'll try to briefly explain what Monads are. Monads are notoriously difficult concept to grasp, but once you have, its pretty easy to hold onto and it is a very useful concept (opinion) that can be implemented in most programming languages, not just in FP.
Tip
Error monads aren't strictly necessary if using ADTs as return types or if your language has its own approach to 'errors-as-values.' However, monads add some extra smarts to facilitate happy-path functional composition. Please read on.
A monad is a burrito 🌯 (a better analogy is a bento-box 🍱 😊). If you've looked into functional programming, you'll understand this aphorism because monads are a notoriously difficult concept to explain: Like a burrito, a monad is a wrapper object (the tortilla) around a type (the filling). This sounds like ADTs right? yes, but monads also add additional 'mapper' methods that are used to apply passed-in computations on the monad's wrapped type in order to transform it into a new result type (or produce a wrapped error). These mapper methods can be chained together as needed. Monads also allow some additional behind the scenes logic to be applied in addition to the passed-in transformations/computations - these are extra 'super-powers' that a particular monad provides. A simple example is the writer monad that appends to an append-log behind the scenes whenever writer mapper methods are called.
Tip
Here's my definition: A monad is a design pattern that wraps a type so that operations can be chained together to transform that type while also allowing additional processing behind the scenes, such as generating additional side-effects, for example.
I prefer the bento-box analogy, because there are more moving parts to a bento-box which better describes a monad (opinion). Having a basic understanding is a useful concept to grasp.
A core tenant of the functional paradigm is to produce a more declarative and expressive 'happy path' of composed computations that isn't polluted with interleaved error handling logic. In a monadic call chain, you define ‘what to do’ by chaining functions that return monads to achieve an end result, not ‘how to do it’ as in more imperative approaches. The happy path self-documents, it screams what the business logic does. In more imperative approaches, you often see that each result is checked using a conditional before continuing with the next computation. Some devs like this approach, ok cool, but others argue that polluting the happy path leads to unreadable code, especially for large call chains.
Here are some features of a monad:
- A monad is concept with a standard API - you can implement monads in almost any language, its not just for pure functional languages.
- A monad can be used as a return type for your functions and goes beyond errors-as-values by adding the ability to do type-safe functional composition.
- A monad is a parametrised 'wrapper' type that is typically implemented with generics.
- An 'Either' monad is a very common monad, its wrapped type is either a successful result OR some form of failure result that is returned from a passed-in computation, never both. For example:
Either<LeftErrorValue, RightSuccessValue>(note that Rust is opposite, where left is success and right is error). - The simple choice between error and success is essential for enabling functional composition in a consistent way: this is a core distinction between monads and ADTs - ADTs lack the simple monad API (required
mapandflatMapmethods). - The computation is passed-in using a standard API that defines two mapper methods -
mapandflatMap.- The passed-in function argument is often named
nextto indicate that it is the next bit of computation to apply. - The 'next' computation operates on the monad's existing wrapped type to create the next result.
- The 'next' computation can optionally generate custom side-effects e.g., calling out to another system or writing a file to disk for example.
- The 'next' function can be a normal function that returns a plain type, it does not have to be 'monad aware.' Alternatively, it can also be a function that itself returns another monad.
- The passed-in function argument is often named
- The monad API is designed to enable functional composition in a standard and consistent way from left to right, transforming each monad's wrapped success type along the way to achieve a final end result.
- As already mentioned, to do this in a consistent way, a monad must have two 'bind' methods called
mapandflatMapthat each accept a computation. The computation is typically a function-reference or a lambda meaning map and flatMap are 'higher-order' functions. The given computation 'maps over' the monad's wrapped type to transform and return a new success type wrapped in a new monad instance (or a new error type wrapped in a new monad). - If a mapper function returns an error type, subsequent calls in the chain will short-circuit the computation and will simply return the erroneous
Either. Short-circuiting continues until the end of the call chain.
- As already mentioned, to do this in a consistent way, a monad must have two 'bind' methods called
- Monads exist for several common patterns of computation - you've likely used monads in several libraries and have not even realised.
Here is a very basic sample implementation of Either in Kotlin, inspired by the Arrow2 library:
// Kotlin
// A basic implementation of Either
sealed class Either<out A, out B> {
data class Left<A>(val value: A) : Either<A, Nothing>() // left for error
data class Right<B>(val value: B) : Either<Nothing, B>() // right for success
fun <C> map(next: (B) -> C): Either<A, C> = flatMap { Right(next(it)) } // notice map automatically wraps the 'next' function in a Right
fun <A, C> flatMap(next: (B) -> Either<A, C>): Either<A, C> = when (this) {
is Right -> next(this.value)
is Left -> this as Either<A, C>
}
}-
A monad wraps a type e.g.
<A>or<B>. Typically, this wrapped value is not a primitive type (int, float etc), but a type that requires its own constructor, hence 'higher-kinded' type. -
A monad has a standard set of 'mapper' methods, also known as 'bind' methods ('map' and 'flatMap'), and 'unit' methods sometimes called 'of' or name after 'of' e.g. 'ofLeft()' and 'ofRight()':
-
- The 'unit' methods initialise a monad
Mby wrapping the given type<B>within the monadic context and has the form (whereMstands for Monad):
M<B>.of(B)
- The 'unit' methods initialise a monad
-
- The 'flatMap' method has the form:
M<B>.flatMap(next: (B) -> M<C>): M<C>where:M<B>is the 'receiver' or 'subject' monad and wraps typeB.M<C>is the next monad result in the call chain and wraps a transformed typeC.next: (B) -> M<C>is known as the 'next' functor which itself must return a new monad.- Explanation: Flatmap accepts a 'monad-returning' function that transforms
<B>to<C>only IF our receiver monad's wrapped value is a success - if receiver monad's wrapped value is a success, flatmap applies the 'next' computation and returns/relays next's response, which is a monad of the same typeM(i.e. function chaining); if receiver monad's wrapped value is an error, flatmap short-circuits and returns the subject-monad and its existing wrapped error value. Note that this passed-in 'next' function must itself return the wrapped<C>value within a new monad result i.e.M<C>, typically to indicate the success or failure of the applied function. The return value is directly (i.e. 'flatly') returned by flatMap, so I think a more accurate name for flatMap is 'mapAndFlatReturnAMonad'. FlatMap has the following form, notice the 'next' function's return type is the same as map's return type.
-
- The 'map' method has the form:
M<B>.map(next: (B) -> <C>): M<C>where:M<B>is the receiver/subject monad and wraps typeB.M<C>is the next monad result in the call chain and wraps a different typeC.next: (B) -> <C>is a plain/vanilla function, it returns a 'plain' type, not a monad.- Explanation: Map accepts a 'plain/vanilla' function that transforms
<B>to<C>. Map then always returns a new success monad that wraps the result of that transformation and has the same type as the receiver monadM. This passed-in 'next' function must return the plain type<C>, not a monad as in flatMap, so before map returns its new monad value it wraps the plain return value within a newly created success (Right) monad instance. This means map's return type is consistent with flatMap i.e.M<C>(I think a more accurate name for map is 'mapWrapAndReturnAMonad').
-
Tip
map(b -> c) is for transformations only and is 'Right bias' - i.e. it always stays in a "Right" and is designed specifically for applying plain functions for value transformations that cannot fail (map always wraps the plain function's result in a 'Right'). If your transformation can fail, you must apply extra logic within flatMap to 'lift' the error into a Right or Left monad to maintain the integrity of your error handling as shown in the example below:
val endResult = validateIngredients(ingredients) // returns Either<BakingServiceError, OkVal>
// cooKPlain returns false on error, so we need to 'lift' this plain error into an Either.Left
// using flatMap, as shown below using the nested if:
.flatMap { u ->
if(cookPlain(ingredients, temperature = 130)) Either.Right(true) else Either.Left("temp too low")
}
.map { packPlain(pie, isFragile = false) } // map returns Right<String>
.map { deliver(pie) } // map returns Right<true>
.map { it -> "Blue Blah" } // map Returns Right<String>
when (endResult) {
is Either.Left<*> -> assert("temp too low" == endResult.value)
is Either.Right<String> -> fail("unexpected right")
}Tip
In the examples above, the composed functions are globally declared for the sake of simplicity. You could easily create custom business objects that declare their own monadic functions for chaining. This is a more hybrid approach that spans both OPP & FP, and is especially useful for encapsulating more complex stateful business logic.
Regarding exceptions in functional composition: If your language uses ‘Checked Exceptions’ (e.g., old-style Java or when using other JVM languages that call out to underlying old-style Java libs), you can’t throw checked exceptions during functional composition as they force you to handle the error and break the call chain with try/catch or throws statements. In this scenario, wrap the exception in the monadic context and return a left error. Note that throwing unchecked exceptions is OK in a functional call chain as they don’t force you to pollute the happy path with try/catch or throws, but you likely still want to wrap the error in a monad to return an error-as-value e.g., if that error is not a programming error or an exceptional circumstance.
In the following examples, we will use monads to simplify a functional call chain. The functions that we will compose include monad aware functions and basic functions that return plain types. The example repo for code below here and here is the same in Rust.
// Kotlin
// Global helper functions, the first two return Either monads, the last returns a plain type
fun validateIngredients(ingredients: List<String>): Either<BakingServiceError, OkVal> { }
fun cook(ingredients: List<String> , temperature: Int): Either<BakingServiceError, OkVal> { }
fun pack(pie: String, isFragile : Boolean): Either<BakingServiceError, OkVal> { }
fun deliver(pie: String): Boolean { } // note, plain return type is not 'monad aware'We use an ADT to model our errors - BakingServiceError with sub-types:
// Kotlin
sealed class BakingServiceError {
object BadIngredients: BakingServiceError()
object TemperatureTooLow: BakingServiceError()
object PackingFailed: BakingServiceError()
data class PoorRating(val minRequiredScore: Int = 3): BakingServiceError(){
companion object
}
}
data class OkVal(val message: String)- Check as we go - In the first example test, we check for success or failure for each step as we go. We don't need to use a consistent error type in this example:
// Kotlin
@Test
fun `demo interleaved result checking`(){
val ingredients = listOf("sugar", "water", "flower")
val pie : String = "baked cherry pie"
val bakePrepped = validateIngredients(ingredients)
when(bakePrepped) {
is Left<BakingServiceError> -> fail("unexpected left error")
is Right<OkVal> -> assert(bakePrepped.value.message == "Ingredients ok")
}
val cookResult = bakePrepped.flatMap { cook(ingredients, temperature = 180) }
when(cookResult) {
is Left<BakingServiceError> -> fail("unexpected left error")
is Right<OkVal> -> assert(cookResult.value.message == "Cooked ok")
}
val packResult = cookResult.flatMap { pack(pie, isFragile = false) }
when(packResult) {
is Left<BakingServiceError> -> fail("unexpected left error")
is Right<OkVal> -> assert(packResult.value.message == "Packed ok")
}
val deliverResult : Either<BakingServiceError, Boolean> = packResult.map { deliver(pie) }
when(deliverResult) {
is Left<BakingServiceError> -> fail("unexpected left error")
is Right<Boolean> -> assert(deliverResult.value )
}
}- Just the happy path - The above test can be simplified into the following, the 'happy-path' becomes far more readable.
- We need to use a consistent error type (
BakingServiceError) for this to compile. However, as in this example, this type can of course be polymorphic such as a custom ADT or an abstract base class. In doing this, you can model multiple success and/or error types . In a different example, the wrapped Left instance could be anything that extendsThrowable, for example. - We only need to test for our final expected result (or the existence of an error). We can safely ignore the interleaving success results - they act as carriers to deliver the final result.
- If you need to extract the error sub-type to perform different error handling behaviour, you can drill down and extract the error sub-type as shown below.
// Kotlin
@Test
fun `demo happy path and check last result`(){
val pie = "baked cherry pie"
val ingredients = listOf("sugar", "water", "flower", "cherries")
val bakePrepped = validateIngredients(ingredients)
val cookResult = bakePrepped.flatMap { cook(ingredients, temperature = 180) }
val packResult = cookResult.flatMap { pack(pie, isFragile = false) }
val deliverResult = packResult.map { deliver(pie) }
when (deliverResult) {
is Right<Boolean> -> assert(deliverResult.value == true) // expected final result
is Either.Left<BakingServiceError> -> {
when (result.value) {
BakingServiceError.BadIngredients -> fail("error handle here")
BakingServiceError.TemperatureTooLow -> fail("error handle here")
BakingServiceError.PackingFailed -> fail("error handle here")
}
}
}
}
// We only need to extract the final result - map/flatMap will short-circuit on an error which allows us to compose the 'happy path' as shown above. On error, extract the error sub-type to do error-specific handling. Tip
Monads enable handling specific errors after the happy path.
- Condensed happy path - The above example can be shortened further:
// Kotlin
val deliverResult = validateIngredients(ingredients)
.flatMap { cook(ingredients, temperature = 180) }
.flatMap { pack(pie, isFragile = false) }
.map { deliver(pie) }
// Extract our final result, or pattern match the error, same as aboveYes. Our parameterised Left error and Right success types can each wrap a polymorphic type such as an ADT used to model all possible error and success variations. To continue our example, we add a new PoorRating data object to our ADT to return the minimum required rating should the pie receive a poor rating:
// Kotlin
sealed class BakingServiceError {
object BadIngredients: BakingServiceError()
object TemperatureTooLow: BakingServiceError()
object PackingFailed: BakingServiceError()
// If we encounter a PoorRating Error, optionally provide the minimum required score
data class PoorRating(val minRequiredScore: Int = 3): BakingServiceError(){
companion object
}
}
data class OkVal(val message: String)A pattern I've seen before is to specify an abstract base class such as Throwable so that any exception type can be returned (not thrown) by a business function wrapped in a Left e.g.
// Kotlin
fun someBusinessFunction(): Either<OkVal, Throwable> {
try{
... processing logic detects an error or throwns an exception
} catch(ex: SomeException) {
return Left(ex)
}
}Monads exist for several standard patterns of computation. For example, we've already seen the Either monad for describing a success or failure result (see above), but others exist:
-
Validation Monad: Unlike the Either monad, a Validated monad does not short-circuit on the first encountered error. Instead, the purpose is to capture all the possible errors encountered in the entire call chain. The error type is thus typically a non empty list. A validated monad can only be used if the computations are independent where input to the next computation does not depend on the successful output of the previous. A simple example is capturing all the errors on a form. In fancy functional speak, the Validator is an applicative functor.
-
Ior Monad: Provides both a successful result and additional list of context messages collected along the way. An example would be a successful code compilation that has several deprecation warnings. Overall the routine was successful, but you also want to collect the additional context.
As described above, monads are designed to be chained together. Depending on the logic you want to compose, this can sometimes require nesting of flatMap and map calls to achieve the desired result. Let's assume the following functions :
fun intToString(n: Int): Either<Error, String> { /*... note String success val...*/ }
fun stringToThing(s: String): Either<Error, Thing> { /*... note Thing success val...*/ }
fun Thing.summarize(): String { /*...simple extension function returns plain String...*/ }To get the desired result, our first attempt at functional composition produces the following example. Note that we mix flatMap and map as required depending on the return types of the composed functions. We use map when our computation (summarize) does not itself return a monad, and flatMap when the computation does return a monad. The resulting composition requires some nesting, urgh:
fun foo(n: Int): Either<Error, String> =
intToString(n).flatMap { s -> stringToThing(s).map { t -> t.summarize() } }To address this, several languages have introduced sequential computation blocks that flatten the call chain. For example, this includes Scala's for comprehension, Haskell's do notation and Kotlin's Arrow2 Raise DSL. An idiomatic Arrow2 example is shown below using the Raise DSL which includes the either block builder:
fun foo(n: Int): Either<Error, String> = either {
val s = intToString(n).bind() // use assignments in the call chain if needed
log(s)
val t = stringToThing(s).bind()
t.summarize()
}No nesting, much nicer, but where did those bind() methods come from, they aren't declared on Arrow2's Either type and what do they do? Well, those bind functions are extension functions that can be grafted onto Either if used within the scope of the either block. Recall from our discussion on pervasive polymorphism above, that some languages (Kotlin, C#) use extension functions to allow you to extend existing types.
- Q. So, how does this either block flatten the nested call chain?
- A. Within the
eitherblock, instead of potentially having to nest function calls like you would withmapandflatMap, you just usebind()on monadic functions which does two things: 1) it short-circuits theeitherblock in the case of an error, and 2) it returns the wrapped/regular type, not a monad. Returning the wrapped/regular type frombind()enables flattening, it allows you to inline optional variable assignments e.g., to interleave some logging as in the example above . If an error does not occur, theeitherblock then wraps the result oft.summarize()within the returnedEitheras a success. For this to compile, the monadic functions need to share the same Error type - this is because theeitherblock needs to return a known typeEither<Error, String>in our example above. Remember though, Either's success and error types types can be polymorphic as in the baking examples above.
The Arrow2 implementation is an example of extensions with extension receivers and function literals with receiver. It get quite complex with several variations, so please refer to the Kotlin and Arrow2 docs for more info and for tutorials. Hopefully however, this shows how errors are generally handled using monads in a functional call chain.
In the functional error handling examples above, we use the Either monad as a smarter return type to bring back our happy path by allowing us to compose a selection of global functions. It is important to remember that we can apply the monad API in custom business objects to encapsulate custom operations with controlled side-effects:
Tip
Recognise that the monad API (mainly map() and flatMap()) can be used to build your own smart custom types that have controlled side-effects. We refer to this monadic custom business logic as a 'Higher Kinded' type.
Tip
"Monads allow you to write happy path code within the context of a monad. A monad wraps all the errors that could possibly go wrong, groups them all up, and lets you deal with that failure outside of the monad's business logic. It basically encapsulates all the side-effects and possible errors. The net result is more reliable and readable higher-level happy-path code." Paraphrasing Kris Jenkins (who hosts the Developer Voices podcast) in this episode of Happy Path Programming podcast
Kris goes on to explain that there is "a heck a lot of information communicated in a Haskell type/function signature - Haskell type signatures are so semantically dense, which is why some new users to Haskell think the documentation is so thin."
I'm not going to use Haskell, I don't know enough, but I do recognise the elegance of a custom 'Higher Kinded' monad for modelling a domain in a reliable way. In the example below, I use a custom higher kinded business object called Account. Note:
- The smart constructor used to return the
Eitherthat wraps anAccountor aNegativeAmounterror on account creation. - The custom business functions
withdrawanddepositthat operate on an instance ofAccount. - An ADT wraps all known business errors -
AccountError - The
transferMoneystatic method which can fail due to a flaky network. If the transfer fails, we catch the offendingIOExceptionand return aAccountError.TransactionFailed(ex)and wrap the original cause.
// Kotlin
sealed class AccountError {
object NegativeAmount : AccountError()
object NotEnoughFunds : AccountError()
object AccountNotFound : AccountError()
data class TransactionFailed(val exception: Exception?) : AccountError()
}
@ConsistentCopyVisibility
data class Account private constructor(val balance: BigDecimal) {
companion object { // functions within the companion are statics
operator fun invoke(initialBalance: BigDecimal): Either<NegativeAmount, Account> =
applyAmount(initialBalance) { Account(it) } // smart constructor
fun create(initialBalance: BigDecimal): Either<NegativeAmount, Account> =
applyAmount(initialBalance) { Account(it) }
private fun applyAmount(amount: BigDecimal, fn: (BigDecimal) -> Account) =
if (amount < ZERO) Left(NegativeAmount) else Right(fn(amount))
/**
* Convenience function to create a new Account or throw IllegalArgumentException.
* Not recommended for production use, use Constructor instead.
*/
fun createOrThrow(initialBalance: BigDecimal): Account {
var accountResult = applyAmount(initialBalance) { Account(it) }
return when(accountResult) {
is Right<Account> -> accountResult.value
is Left<*> -> throw IllegalArgumentException()
}
}
/**
* @return Updated debtor and creditor Accounts in a Pair instance, or an AccountError if the transfer fails,
* for whatever reason such as flaky network.
*/
fun transferMoney(debtor: Account, creditor: Account, amount: BigDecimal): Either<AccountError, Pair<Account, Account>> {
return try {
// In the real-world, this might be a flaky network operation than can fail
debtor
.withdraw(amount)
.flatMap { d -> creditor.deposit(amount).map { Pair(d, it) } }
} catch(ex: IOException){
Left(AccountError.TransactionFailed(ex))
}
}
}
fun copyAccount(balance: BigDecimal): Account = this.copy(balance = balance)
fun deposit(amount: BigDecimal): Either<NegativeAmount, Account> =
applyAmount(amount) { this.copyAccount(balance = this.balance + it) }
fun withdraw(amount: BigDecimal): Either<AccountError, Account> =
applyAmount(amount) { this.copyAccount(balance = this.balance - it) }
.flatMap {
if ((balance - amount) < ZERO) Left(NotEnoughFunds) else Right(Account(balance - amount))
}
}Here are some corresponding tests that show Account usage:
// Kotlin
@Test
fun `should fail creating an account with a negative amount`() {
assert(Account.create((-100).toBigDecimal()) == Left(NegativeAmount))
}
@Test
fun `should withdraw money from an account`() {
val account = Account.createOrThrow(100.toBigDecimal())
val updatedAccount = account.withdraw(50.toBigDecimal())
assert(updatedAccount == Right(buildAccountViaReflectionForTestsOnly(50.toBigDecimal())))
}
@Test
fun `should fail withdrawing a negative amount to an account`() {
val account = Account.createOrThrow(100.toBigDecimal())
val fail = account.withdraw((-50).toBigDecimal())
assert(fail == Left(NegativeAmount))
}
@Test
fun `should fail withdrawing when there is not enough funds`() {
val account = Account.createOrThrow(100.toBigDecimal())
val fail = account.withdraw(200.toBigDecimal())
assert(fail == Left(NotEnoughFunds))
}
@Test
fun `should transfer money across two different accounts`() {
val debtor = Account.createOrThrow(100.toBigDecimal())
val creditor = Account.createOrThrow(100.toBigDecimal())
val result = Account.transferMoney(debtor, creditor, 50.toBigDecimal())
assert(result == Right(Pair(buildAccountViaReflectionForTestsOnly(50.toBigDecimal()), buildAccountViaReflectionForTestsOnly(150.toBigDecimal())) ))
}A bento-box is better analogy for describing higher kinded types - why? coming soon (TODO).
In computing, concurrency is not parallelism, despite the two terms having very similar dictionary definitions. Concurrency is a software concern involving context switching of processes on a single CPU core via ‘kernel threads’ (these types of thread are also commonly referred to as 'process threads,' 'carrier threads,' and 'platform threads'). Context switching gives the illusion that multiple things are happening at once because the time slicing is so small. True parallelism is both a software and hardware concern which requires increasingly more hardware to do more things at once. This can range from multiple cores on one CPU, multiple CPUs, multiple nodes, remote actors, remote VMs, clustered VMs, cloud functions such as AWS Lambda and more.
For example, parallelism with increasingly heavyweight implementations from small to large could range from:
- Single host with shared memory parallelism using low level platform threads and locks (mutexes), where threads map to cores. Programming with platform threads and locks is low level.
- Single host with shared memory parallelism using software abstractions built on lower level platform threads, such as suspending functions commonly called co-routines, virtual threads a.k.a. fibers, async/await methods/functions, pragmas to parallelise tight loops such as used in in OpenMP.
- Multi-processing where each child-process has its own memory space where data is separately distributed to each process for processing, or data can be shared using inter-process-communication mechanisms (IPC) such as memory-mapped files (memory and disk) or messaging over sockets.
- K8s worker nodes/pods hosted in the same K8s cluster with communication over ethernet.
- HPC using compute clusters where 'tightly coupled' workloads message-pass over a high performance interconnect (MPI), VM clusters (e.g. Apache Terracotta).
- Geographically distributed compute nodes/servers such as remote Actors / FaaS / Grid computing / service mesh.
- Geographically distributed multi-clustering (e.g., Grid, federated HPC - a potential route to exa-scale).
Some general recommendations:
-
Keep platform threads as isolated as possible & limit mutable global state:
- Sharing of fixed immutable state is fine.
- Taking defensive copies of data can help prevent race conditions and other concurrency related ‘spooky actions at a distance’.
- Try to be more functional and limit your use of global mutable state.
- Understand the pitfalls of multi-threaded code such as race-conditions, ghost reads, dirty reads, dirty writes, and thread deadlock.
-
Keep synchronised critical sections as small as possible:
- Amdahl’s law: "The overall performance improvement gained by optimising (i.e. parallelising) a single p art of a system is limited by the fraction of time that the improved part is actually used," or more simply: even a small amount of synchronization _significantly_ affects performance.
- Here are some examples of Amdahl's law:
- If 95% of the time spent by your code is parallel, throwing more processors at the problem does not improve speed up beyond ~256 processors.
- If the amount of time spent by your code is <50% parallel, adding more processors won't speed up your code at all.
- If the amount of time spent by your code is ~95% parallel, the maximum speed up is only 20 times and this takes 2048 processors.
- Understand the pitfalls of multi-threaded programming. If deadlock, live-lock, ghost-reads, dirty reads, and atomic vs composite actions don’t make much sense to you (do you think ‘i++’ is atomic? – no it is not), then you will no doubt run in to problems. For application developers rather than lower level library developers, there is often a much better approach to coding low-level multi-threaded and shared memory models.
- When testing, use more threads than processors – running with more threads than processor cores encourages task swapping. The more frequently your tasks swap, the more likely you will find issues.
IO Bound Tasks require asynchronous patterns to achieve concurrency. The solution patterns include:
- Async/Await (e.g., Rust/C# async/await functions and Kotlin’s suspending functions). These are often referred to as ‘coloured approaches’ because your functions are typically split into two types; 1) red functions for asynchronous code typically requiring special keyword modifiers to annotate functions and their call-sites e.g.
async(function) andawait(call-site) and; 2) blue functions for plain synchronous code having no modifiers. Here is the original and now famous blog - What Color is Your Function. If await is called on an async function, this often causes other async functions to be suspended. - Colourless Coroutines (e.g. Go's Goroutines). Go's Goroutines are not coloured because they only require the
gokeyword at the function call-site, there is no need to annotate a function as asynchronous which allows you to call regular functions in an asynchronous way (although you frequently need to pass in concurrency primitives to share data such asChannels). - Virtual-Threads / Green Threads / Fibres such as the ‘colourless’ Project Loom for the JDK where the virtual threads are implemented in user space using continuations. With the release of Loom, the venerable JVM platform arguably boasts one of the most advanced approach to concurrency. JVM languages can implement a (partly) colourless programming approach on top using a combination of platform and virtual threads and higher-level abstractions such as the streams API.
- Continuations are very low-level and are used to implement patterns such as coroutines & virtual threads. Continuations can be suspended, stored on the heap, and restarted. Typically, you would not directly use continuation APIs using a Continuation Passing Style (CPS) in your own application logic, although some languages do have public APIs for CPS.
- Async-Wrapper types such as Futures & Promises. Note that these are really just higher-level synchronisation primitives, and the task that you await itself would need to be non-blocking to achieve high levels of concurrency.
- Call-Back functions (beware ‘call-back hell’ as often seen in JavaScript). In fact, more friendly ‘synchronous’ patterns such as Coroutines and Continuations simply abstract much of the lower-level call backs from the programmer.
- Non-blocking IO primitives where libs & APIs are non-blocking instead of blocking.
- Reactive Frameworks e.g., project Reactor. Performant, but typically requiring a horrible API that has a lot of 'hidden magic.' In the JDK space, with the delivery of Project Loom (see above), I believe reactive approaches will decrease in popularity.
CPU Bound Tasks have different solution patterns:
- Platform/OS/POSIX/Kernel threads (pthreads). These are good for numerical computations that do not require much, preferably no, IO. An example would be a ‘tight computational loop’ that performs an in-memory calculation that has no side-effects. Note that platform threads are rather heavyweight and so should be pooled for effective resource usage (see Fork-Join-Pools). For example, on Linux, each thread typically requires ~1MB of memory per thread – this is because they are controlled by the operating system, and the OS has to be generic enough to handle a variety of use-cases, so 1MB was assumed to be a catch-all default. Platform threads are very different from Virtual Threads in terms of how they are implemented. A platform thread is very similar to a process in terms of resource-cost, except that threads allows memory sharing between multiple threads while multiple processes do not share the same memory space. For multiple processes, you need some other mechanism to share state and messages between processes, such as a common memory-mapped file(s), file system, databases, and message brokers.
- Shared memory frameworks such as OpenMP which make multi-threaded programming simpler by avoiding low-level synchronisation primitives.
- Fork-Join-Pools and related ‘Work Stealing’ patterns that involve task queues. FJP is a highly recommended way to achieve best possible performance because low-level Kernel/OS threads are re-used to maximum effect.
- Horizontal scaling with Pub-Sub and Competing Consumers. This is where multiple compute nodes subscribe to a message channel and pull messages from the channel. If the queue-depth gets too high, you add more consumers to process the messages.
- Lazy Parallel Streams in functional approaches. Functional streams are typically executed lazily, importantly after the whole computation has been fully defined. This allows the caller or runtime to perform optimisations such as automatic parallelisation. This can only be achieved because the full stream is defined lazily, ahead of time.
- Message Passing e.g., the Actor model (e.g., Akka) & Message Passing Interface (e.g., OpenMPI) in HPC are both examples of message passing. Note that the Actor model is actually the canonical parallelism pattern, while MPI is quite niche (largely just the HPC community).
If you must use low-level locks and synchronization primitives with critical sections, try to use ‘re-entrant’ locks for better composability and performance over non-re-entrant synchronized blocks. Check if the languages mutexes (semaphores, count-down latches) are re-entrant (language agnostic advice).
Tip
Coroutines (concurrent routines) are NOT lightweight threads, they are an entirely different abstraction from threads. A coroutine is a lightweight and suspendable function call or sequence of calls that run on-top of 'carrier' platform threads.
Coroutines are generally implemented as 'coloured' functions i.e. functions that use prefixes such as 'suspend' (Kotlin) or 'async' (Rust/others). The underlying carrier thread that runs a coroutine can be different depending on your application's configuration. For example, the actual carrier thread may come from named thread-pool can can be scaled independently, or it may be the main UI 'application' thread in a GUI or Android application for example.
Tip
Coroutines are suspended at their function boundaries.
This means that a single function split into multiple smaller functions enables more suspension points. As a result, some thought about function granularity and logic splitting is needed when designing coroutines. Depending on configuration and design, the next suspending function may run on an entirely different carrier thread from the previous.
Coroutines are generally started using a construct that launches the coroutine as they can't simply be started by calling the suspending function from a regular 'blue/sychronous' function. The launch construct implementation varies across languages; in Golang, you typically prefix a non-returning function invocation at its call site with the 'go' prefix which works in conjunction with an underlying thread pool configuration. While this is great because you don't need to split your functions into different types (async vs regular synchronous/blocking), you cannot simply use 'go' on everything and expect it to work - the go keyword doesn't return the function's result, so you need a way for the function to communicate with the use of Channels or WaitGroups.
In Kotlin you need to create a 'coroutine builder' such as Coroutine.launch() and a coroutine 'Dispatcher' which controls which thread or thread pool the coroutines use for their execution. This makes it simple to change the underlying carrier thread across a sequence of coroutine calls i.e., conceptually: 'run this part of the coroutine on the main UI thread, then switch and run this part of the coroutine on a named thread pool, then switch back and finish on the main thread'. This is very powerful, and this requirement typically surfaces more often in GUI and Mobile applications (where you need to be careful not to block the main UI thread) compared to server-side applications that typically process a request within a thread started by the application container.
Tip
A running coroutine can be suspended if another coroutine is awaited i.e. by calling await() on a coroutine can suspend another coroutine.
Virtual threads are NOT coroutines. A virtual thread (see Java's Project Loom) does not manually declare the suspension points with keywords like 'async' and 'suspend' like coloured functions do. Instead, a blocking sequence of calls is declared within a thread and suspension handled automatically by the runtime so that whenever a blocking call is encountered, such as a network or socket call, it is automatically unloaded from the carrier thread to allow another virtual thread to run. This is very powerful especially for server applications as virtual threads massively increases concurrency.
In Loom, the virtual thread API re-uses the existing thread API. A positive is that it does not split functions into separate blocking and non-blocking colours, a negative is that it does not provide a lower-level thread switching capability provided by Kotlin's coroutine dispatcher API. It could be argued that Loom's virtual threads are not entirely colourless (which is why I say 'partly' colourless in the section above). This is because when handling thread processing logic in functions/methods, typically you must mark the method with a throws InterruptedException. This throws clause is part of the method signature, it is a contract indicating that this method can be interrupted by another thread and marks the method as asynchronous, which means the method is not itself colourless, just like async/suspend functions.
Coming soon TODO
- Never add plain-text credentials including username/passwords and ssh keys/tokens into version control.
- Sensitive data such as credentials can be stored locally in your local dev environment using ephemeral sources such as environment variables, command line arguments, local files such as '.env' files that are git ignored (use a '.gitignore' file) to ensure they are not committed to VCS, and local key-chains / credential stores such as HashiCorp's vault and https://github.com/openbao/openbao .
- Do not hard-code secrets in code. For production, use well-established secret serving methods such as creating a secret object in OpenShift that configures environment variables for your running pods.
- Public URLs should always be secured using TLS/HTTPS. Host certificates can be freely obtained from https://letsencrypt.org
- Always consider linting and scanning your code for vulnerabilities and anti-patterns using well-established tooling such as FindBugs, Snyk for containers, OWASP's Dependency Check tool suite: https://owasp.org/www-project-dependency-check
- Familiarise yourself with OWASP's Top Ten security risks for webapps: https://owasp.org/www-project-top-ten
- Always update default passwords that are shipped with products e.g., 'admin' is sometimes used default username and password pair.
- To minimize injection attack surface, don’t use your own variable binding or hard code parameters using string concatenation – use the supported variable binding tooling to ensure values are always escaped.
For the Hartree Centre, we propose an Agile methodology as it largely suits the type of projects we do. Agile is an overused term, so for Hartree’s purposes, a good definition is ‘Feedback Driven Development’. Iteration and customer feedback really ARE essential if we are to successfully address real customer needs. Know that industry data shows that even for the best software companies in the world, two thirds of their ideas produce zero or negative value so continuous feedback is essential to mitigate the risks: Online Controlled Experiments at Large Scale: http://ai.stanford.edu/~ronnyk/2013%20controlledExperimentsAtScale.pdf
According to the values of the original Agile manifesto (search the original ‘Snowbird meeting’), agile development practices include risk-taking, rapid-feedback, frequent and high-bandwidth communications across the whole team, and collective project ownership. This means full stakeholder involvement with everyone: developers, testers, scientists, end-users, and business-development managers. It emphasises ‘individuals and interactions over processes and tools.’ Agile is especially relevant for greenfield and relatively short-lived projects which describe many of the projects we do at the Centre.
We recommend weekly or fortnightly iterations involving customer playbacks and demos. Anything longer than 2 weeks can require significant course correct if/when you go in the wrong direction - agile aims to catch problems early and to course correct. According to Uncle Bob Martin, the emergence of agile was to “find out how screwed we were as early as possible, it was not just about writing software quickly”.
Design Thinking puts you in the shoes of the customer so that you can understand their pain points. This helps design solutions that really address customer needs. Hartree have a set of recipes for activities that you can use to conduct DT workshops. The activities don’t have to be applied religiously and you can adapt as needed. The activities include As Is Scenario Journey Map, Empathy Maps, User Persona and Problem Statements, User Stories, Ideation, Prioritisation, Ideal To-Be Scenario Journey Map, Outcome Statements, Cupcake road maps.
Hartree also has a scoping doc that you can send the customer ahead of time to help focus minds.
Epics are like Work Packages. Typically, they require multiple tasks and span multiple sprints.
Define user stories with the INVEST Framework or Who-What-Why or the Connextra Card Template – all are good and you do not need to be too rigid
-
“As userType [X], I need a way to do [what?] so that I can [what’s the benefit]”.
-
Who, What Why
Tip
Break up large stories into smaller stories.
Tip
Don't define technical task as stories - stories should be written using a user-friendly vocabulary.
- INVEST:
- Independent - this means we try to design stories that do not need to be implemented in a particular order (a soft rule as there may well be stories that need to be prioritised).
- Negotiable - to retain agility, we recognise that requirements often/inevitably evolve and so we don't focus overtly on getting the details right up-front (i.e., Waterfall).
- Valuable - must have a clear and quantifiable benefit to the client.
- Estimable - a story must be concrete enough that developers can estimate it.
- Small - a story must not be larger than one or two developers can implement in a single iteration.
- Testable - when a developer says that its 90% ready, nobody really knows how close it is to being finished.
Arrange core user stories into a Journey Map with a narrative flow or backbone of Big Activities moving from left to right
Beneath each big activity, define short verb phrases to describe what the user does to achieve each big activity.
Create a list of tasks and use ‘planning poker’ / finger-waving to estimate effort – after a ‘3, 2, 1’ countdown, everyone at the same time provides an estimate of the difficulty of a task between 1 and 5 or holds up a card. This ensures honest estimates from everybody which is Important because different team members may have different experiences/specialities of the task area. See https://www.evernote.com/l/AWQ6FGRtfrNI1az21FVp9aosQ9zu8b-4CXg
Break up the Backlog into sprints to deliver your cup-cake roadmap. Provide effort estimations for tasks using ‘planning poker’. More than two weeks generally gives enough time for software to deviate without requiring significant refactoring and course-correction, so we don’t recommend more than 2 weeks.
Test the critical path and be pragmatic about coverage - 80% coverage often not feasible or even useful. Develop tests in-line with the mainline branch. TDD helps us think about the public interfaces / API to the code under development.
At the end of the sprint, demo your progress to the client. This is important. Agile can be paraphrased as ‘Feedback Driven Development’. It is essential to get that customer feedback early and continuously.
- If possible, get the client to sign-off work every month (PMO have a ‘Decision Point Review’ template).
- Use the Cucumber approach for acceptance testing i.e., ‘Given, When, Then’. For example: ‘Given [a particular context/scenario], When [something happens], Then [this is the result]’.
- Recognise that we need both iterative & incremental approaches to building complex systems. Incrementalism == modularity, which helps break down complexity.
- Review the Backlog, revise and plan your next sprint, jump to 7.
- Plan a cup-cake dev roadmap. A cup-cake won’t feed everyone, but it can have core ingredients - it’s a whole-product that a user can taste sooner rather than later.
- If the cup-cake tastes good, proceed with the vanilla sponge, hopefully ending with the multi-tier wedding cake that can feed everyone. Iterate your development roadmap and keep soliciting user feedback.
Thanks for reading, comments/feedback most welcome. Have fun !
