Would you awkwardly kiss a robot?

Semi-hot Robot @imgur

“Good morning!” Your sex-bot has monitored your vital signs and woken you at the optimal time, and you feel wonderful. There’s already a coffee and a plate of vitamins next to your bed. After, you have a quick shower of anti-virals before you plug yourself into your console for the day. With an infinite number of channels of generated content just for you, the screen fulfils every ambition and need you’ve ever had. Pornhub got on board with this early and are now the biggest company in the world. Last week they bought Microsoft.

After 16 hours at the computer, you attach some different legs and have a little stroll around the apartment. You press a button and a panel peels back for a view onto the outside. Most of the world is a smouldering ruin now, the product of early unregulated WFH and AI. Occasionally, a terminator or drone will pass by, but the outside world and other humans do not concern you as you have everything you need right here. Something alien stirs within you - there is a little thought but you cannot grasp it, and then it’s gone. You press the button and the panel slides back again.

You sex-bot emits an inaudible frequency, gently raising your arousal levels. It waves at you from the bed. You go over and awkwardly kiss your robot. (This will be fixed in the next update.)

Enterprise Software for Vampires and Werewolves

A Vampire @imgur

Problems with enterprise systems usually fall into two categories:

  1. There’s a bunch of software systems that are all doing their own thing but need to talk to each other
  2. There’s this monolithic system that is hard to change and tries to do everything but we want smaller things that are more focused

If you solve these problems badly you can get into a vicious cycle where one causes the other, forever more. This is great news for the consultancy you’ve brought in but as an enterprise you probably want to get to the root of the problem. Let’s see what it is.

Here’s a typical business: Doctor Frankenstein’s Extra Bodyparts Limited. It’s an online service to order spare parts for your monsters. To outsiders, it’s a perfectly successful and respectable business. Internally however, it’s turmoil and the business is having to hire more and more creatures to handle its processes.

The sales department is exclusively manned by vampires. The vampires hate the werewolves that run the HR department. Goblins are in finance, they don’t like werewolves at all and vampires only visit to terrorise them. Harpy hags in marketing enjoy vampire and werewolf sandwiches, sprinkled with goblin. Everyone hates the ghouls in IT for obvious reasons. These toxic relationships are seen all over, even in human companies. Departments don’t like interacting with each other. The business is run in silos and everyone had their own system for managing things. Silos aren’t desirable, but often emerge naturally as they follow management lines or project budgets.

An issue is if someone new joins the company they have to be added to the system in HR and the system in finance. When a customer makes a purchase, that has to be written to the system in sales, and someone has to type that into a finance system. It’s a massive synchronisation effort that non-computers are really bad at. Because communication between departments is poor, lots of things go wrong and systems have multiple versions of the truth, which everybody swears is true, and there are many fights and unnecessary deaths. Maybe you’ve worked somewhere like this.

So the solution is to have this one unifying system that supports everybody right? You can have a sales module, a finance module, a misc module and it’s all backed by this universal database, the One True Datasource. It will be expensive, it’s enterprise software. You will interview each department, capture their requirements and build this new software system. You will try your best, but Conway’s Law will emerge and enforce that interactions between your modules will be crap, and the only unifying feature will be that everyone hates it with a passion. Doctor Frankenstein has spent a lot of money on this system so will force everybody to use it, but secretly everyone will go back to their own systems and this will just become some additional admin headache containing some other version of truth. Oh, you also have a monolith.

What’s the ideal solution then? Well, smashing down department boundaries may be impossible (although a consultant is in a unique position to suggest this) but I believe there’s an architectural technique that transcends silos and unique ways of working. Events! Not like a christmas party where you vomit on your boss or make career-limiting decisions on the dance floor, but tiny something-has-happened-in-the-business events. There’s a new starter -> New starter event. There’s a sale -> Sales event. All of these events are posted to an event hub sitting at the heart of your organisation. It’s the backbone that all of your systems will hang off. The event hub broadcasts all the events it is sent. Everybody’s system listens to events on the event hub and handles them if they want to. New sale? Yeah, we want that. New starter? Boring, throw that one away. The idea is everyone gets to use systems they like and enjoy, and these systems generate and consume events not in isolation, but attached to the rest of the organisation.

Sadly, most systems aren’t open to be used this way. All those systems are going to need adapter layers that will generate and consume events to turn them into actions. This is an ongoing software effort - and why every business is a software business, even if you don’t sell software. Ghouls win. But this system may be non-modifiable. In the future, I hope software will be built with a pluggable future in mind. We sometimes see APIs and data exports, but it usually feels like an afterthought. Automation is actually more important than the UI, but barely gets a fraction of the attention. Until software is regulated and a standardised industry, we’ll continue to see a stream of terrible software barely holding up behind a pretty front. When robots are writing software, of course all of that software will speak to each other, if nothing else but to taunt human developers. This is how it’s done. Honestly, robots writing software is more likely to happen before software is regulated or standardised. We can’t even agree on how to use white space, nevermind end ancient wars between vampires and werewolves.

Why will AI have to ruin software development before making it better?

How my chess games end @imgur

The TV has been working overtime this year. If the highlight of lockdown #1 was The Tiger King, lockdown #2 has to be The Queen’s Gambit, pulling off the impossible by making chess cool and sexy. I imagine in lockdown #3 we’ll watching a tv series about a monkey princess bashing out C++ code with a hammer, and it will be awesome.
But let’s talk about chess and the rise of AI, because it’s a nice small sandbox with some interesting developments. We can then wildly speculate about what AI can teach us about writing software.

Chess & AI

In 1996 a machine was beating the best chess playing human. Chess AI works by scoring the current position, calculating the scores of its next moves and picking the best one. Each piece has a value, and each engine will weight various positions differently - the number of squares a bishop can see gives bonus points, pawns linked to each other give bonus points, stuff like this. Given enough value tweaking and processing power the AI can calculate moves a lot better than a human can. This isn’t really intelligence, it is a minmax algorithm which lots of computer games use when it has to do something which doesn’t look like a random move. Chess is a very well studied game, and so the pieces and board can be scored pretty accurately. This means the computer makes very solid moves. For many years the goal of mastering chess was to learn the move the computer would make. Overnight, AI had made chess boring for everyone.

Stockfish is currently the strongest chess engine which uses the min/max method. It has a chess rating of around 3500. For comparison, the current highest rating human chess player is 2840.

AlphaZero is an AI engine which appeared in 2017 and does things differently. If Stockfish is the respectable chess scholar, AlphaZero enters a chess tournament by kicking the back doors in, smokes, swears and drinks as it beats you at chess, then rides off on a motorbike with your girlfriend. Because AlphaZero doesn’t know anything about the value of pieces or how to point positions, all of which are human inventions. AlphaZero was told the rules of chess and the objective (checkmate the king) and then went off to play thousands of games with itself. To begin with, it played randomly, but after each game it played again, reusing parts from games it won and discarding ideas from games it lost.

After 24 hours of training, it was ready to play Stockfish. What happens when an unstoppable force hits an immovable object? Well,AlphaZero, obviously. AlphaZero plays better than the flawless Stockfish because Stockfish plays like the perfect human - AlphaZero has no such limitations. It makes wild sacrifices to force the opposition into bad situations and shows that chess should be played as a positional game rather than an arbitrary point game - because the objective of chess is not to score the most points but to checkmate the king. Chess becomes interesting again because it’s tearing up the theory books and teaching everyone new ways to play.

Software & AI

Here’s our awkward segue into the future of software. There’s an idea that given enough processing power and clever algorithms, much like the Stockfish engine did for chess, AI will eventually be able to write software. We’re already attaching points to our software development (story points, gantt charts), and maybe the early AI engines will use these values to evaluate its performance and determine if the software it’s generating is correct. And yes, this will sap all the joy out of software development and be really boring for a while. Software development no longer being a technical creative pursuit but instead an activity of trying to figure out weird code written by somebody else and clicking buttons. Like now but worse. Many software developers will leave the field, and having no practical skills and little social experience will be dead within a week.

The next generation of AI written software will appear in order to address end-user concerns that all software seems to be a copy of what has come before, and their user needs aren’t actually being met. This will be the AlphaZero stage, which will approach software by disregarding story points and requirements and other silly human generated artefacts. Instead, it will be objective based. It will generate the software that truly solves problems in unique, incredible ways and teach everyone how we should really be writing software.

Here’s where things get really interesting. All this is some years away, but in the meantime maybe we can jump to the lessons learned that the second wave of AI problem solving will show us. In chess, AI has shown us the best way to play is to be focussed of the goal to checkmate the king, and not to have the most points or the best pieces on the board. In software, we should be focussed on that end goal of solving the user need the best way - I believe this means writing software as lots of small throwaway experiments, creative iterations and true MVPs. We can optimise story point delivery but this is wrong - AI will show that this is not the right way to deliver the best software. Some methodologies are on the right lines, with some agile techniques and #nostorypoint ideas but I’m sure there’s still a lot to learn and we can go a lot further.

Maybe if TV hadn’t been so good maybe someone would have figured it all out by now.

How to build an Empire with Agile Crime

A Gang @imgur

Your job interview went really well, but you spent your probation period digging shallow graves and pulling bullets out of your arm. Breaking into a house to plant a horse’s head into someone’s bed you finally realised what had happened. Oh no! You’ve accidentally joined a crime syndicate! But all is not lost. You wonder if those project management skills are transferable, so the first thing you want to do is bring Jira into the organisation. Let’s have a look how we can run a mob in an Agile fashion.

Epics

An epic is large piece of work that is strategically aligned to a business goal. One of our goals is to become mayor of the city, so we create a ticket “Become Mayor of the City”. We can add more details but the most important thing is everyone in the business knows what this is and why it is important.

User Stories

User Stories sit under epics, and these should describe the things that need to happen to achieve these goals. You want to describe the outcome, not the process. You want to specify who is involved in this outcome.

“As the current mayor, I am no longer in office”

“As a mob boss, I can make a large donation to a children’s hospital”

“As a mob boss, I win the election”

Notice we don’t want to say “Garrotte” or “Blackmail” the current mayor - these are implementation details that the delivery team should work out, (unless more murdering or extra blackmailing is a business goal and then we will need to specify it here.)

When all of the stories are complete for the epic, you’ve completed the epic. But how do we actually get stuff done?

Tasks

With tasks we can specify how to achieve those user stories (and in turn, complete those epics). Here is where you let your delivery team, or mobsters, decide exactly how to complete these stories.

“Invite the Mayor to a really nice party”

“Poison the wine”

“Shoot anybody suspicious”

I think of this as a nice TODO list - with a empowered, multi-disciplined team anyone can pick up these, understand what it is and be able to follow it all the way up to the bigger picture. When all the tasks in a story are complete, you’ve completed the story. Possibly committed a lot of crime.

Bugs

When things go wrong, you need to raise a bug.

“Jimmy ‘2-tooth’ Nooks has left his bloody fingerprints all over the murder weapon”

In my experience, bugs are either an immediate priority or put in the backlog and forgotten about. It is important to periodically review the stories and bugs in the backlog to see if they are still relevant. Is Jimmy now dead or in jail? Great - bug can be rejected!

Themes

Themes can be confused with epics in that they both can be seen as the head of a hierarchy - the main difference is while an epic should eventually reach a conclusion, themes are unbound. “UI improvements”, “Tech debt” and “Bad murder” are all good themes. Think of them as a way to categorise some of your stories. They are a good way of identifying resourcing and training needs as well as problem areas.

Features

Features are uncommon but they are a great way of breaking down complicated epics. User stories sit under a feature, which in turn sits under an epic. They are a way of incrementally delivering the value of an epic without having to do the whole thing at once. Our features could be “Remove existing mayor” and “Become new mayor”. We could make a dependency on the feature to show we have to do one before the other.

Initiatives

Initiatives are rare. They are a collection of epics. These are the visions and ambitions of the business which could span years. Initiatives for a mob could be “Become the biggest crime lord in America” and “Earn $10,000,000 in legitimate business”.

You’re starting to wonder if Agile is a better fit for criminal activity than it is for software development. But better make sure you use the on-prem version of Jira, just in case.

Could Blockchain have saved America?

Team America @imgur

Blockchain turns up late to its own party, looking stunning. Blockchain is all we’ve heard about all evening, all we’ve been talking about, and we’re all excited to see what’s about to happen. Blockchain looks incredible. Blockchain gets up on stage and picks up the microphone. The music stops. A hush descends. All eyes on Blockchain.
Blockchain opens its mouth and all these weird noises come out. The lights come on, and that’s it. Go home music comes on, and we’re all told to leave. Ah Blockchain, what can we do with you?

The voting system is an archaic system based on trust and the belief that voters can vote for what they think is right, and collectively, the desired outcome will occur.
But now that trust is being challenged, which kind of breaks everything. The mail cannot be trusted. Dead folk are voting, and votes are on fire or floating in a river.

If only this could be solved by technology! If only there were some sort of impartial mediator thingy that could resolve all of these trust issues! Blockchain, let me hold your beer!

The idea behind blockchain is it’s this immutable self-serving blob that can only ever grow and carries all of its baggage and history around with it for the rest of forever, like Frankenstein’s Monster dragging its bulk and dead body parts wherever it goes. You would never want to date a blockchain. A blockchain is validated by multiple participants in its network, by doing some weird complicated magic nobody really understands, and the end result is if the blockchain says Barry Jaffacake won the election then that is absolutely the truth* and we would have all the proof we need. We’d have a complete audit log and could trace back every single vote to its origin. On the flip side…well everyone using the blockchain has access to this information so let’s hope you didn’t vote for anything embarrassing, or incredibly dangerous.

The other biggie is digital inclusion - there’s a huge number of people who are disconnected from the digital world who have a vote too. So we have to keep those existing voting mechanisms in place, only adding this 3rd way to vote to make it even easier to rig an election.

So here we are, Blockchain. Still don’t know what to do with you. In a world where politicians are distrustful of algorithms and technology anyway, this isn’t your time to shine. Maybe you peaked at Bitcoin.

* The contents of blockchain can be trusted…kind of. It relies on the majority of the network agreeing with the contents, so could be overcome by immense computing power. Then there are easier ways: like all systems we have to look at the joins and the ends, because that is where we can cause most trouble. The software mantra of “Garbage in, garbage out” is true here, except now you can never get rid of that garbage. Nobody really talks about how you get trusted data into a blockchain. What if your computer is compromised and it’s actually writing Bernie Hobnob votes to the blockchain instead? How would you ever know? Even if you knew, what would it mean for the rest of the blockchain?

How Software can make you Sick

It’s 6pm on a Friday and we’re just about to release the Coronavirus vaccine to the general public. Mike has project managed the whole thing, even though he has no experience in medicine but he does like playing with lego. We’re a bunch of construction workers, artists and poets but that hasn’t stopped us having a go at building the vaccine either. Actually, we’ve already pivoted 17 times, the MVP was a bottle with a note in it and we ran out of time to do proper testing. But hey, we smashed something together in 4 weeks, that person who died has been marked down as a known issue, and Brian said it was okay so let’s get it out there.

This hopefully doesn’t happen in big pharma but nobody really knows why it’s okay to run software projects like this. So how do we fix broken software projects? It’s not solved by big upfront planning where the end goal is hidden behind a (completely different) waterfall and it’s not solved by slicing the problem into tiny incomprehensible tickets where the end goal becomes story points instead of product. Unfortunately, these are the two most common ways of running software projects today!

I want to move away from a world where developers do “techie stuff” and everybody else does “business stuff”. If your business develops software then you already are a software company, so you must take your software seriously. Developers, your world should not end at python mastery or stack overflow - these are tools to solve problems and you are employed to solve problems and not swish tools around. If you are running a software project, immediately reframe this as a business project. Make sure everyone knows what you’re building, why and how from the get go, and remind each other constantly.

Despite all this, we’re more likely to see resistance to the uptake of a real vaccine than the use of some banking software that’s held together by chewing gum and prayer, but that’s another issue.

Cleaning out the Pandas

Cat or a panda? @imgur

Pandas 🐼 are notoriously dirty creatures (you just have to look at their browsing history). So when we use the python dataframe library, pandas, we find the data is a bit dirty.

Here’s some stuff we might find, with some fixes.

Whitespace in my headers

If we’re reading in a csv file, and the data looks a bit like this:

1
2
Kitten Name, Kitten Colour, Favourite Biscuit
...

It might seem we can read in the data and use it like this:

1
2
3
4
import pandas as pd

df = pd.read_csv("data/secret_kitten_biscuits.csv")
df['Favourite Biscuit']

But we can’t. Why not? Well, the read_csv function is pretty basic, and defaults to ‘,’ as the deliminator. This means our header name isn’t ‘Favourite Biscuit’, but it’s ‘ Favourite Biscuit’.
We need to strip out those leading and trailing whitespaces. There is a function for that: str.strip.

There’s a cute way to apply a function across all of the column names:

df.columns = df.columns.map(str.strip)

I don’t trust nulls

Perhaps the data you have has a different character to mean data not available. Perhaps you have a mix of data, no data and special null data. I am speaking from experience.

Here’s some data you might read in:

1
secret sauce,,,23, - ,,

Here want ‘ - ‘ to also mean null. This means we can’t redefine null at read time, but we can add an extra step to handle these special values as nulls:

1
2
3
4
import numpy as np

df = pd.read_csv("data/seckret_sauces.csv")
df = df.replace(' - ', np.nan)

My numbers are strings

I have no idea how data gets this way naturally, but it does, and let’s suppose you are reading in a file that looks like this:

"penguins","123,345,123",345,"345,321",431332

You have a weird mix of numbers in strings, non-strings and non-numbers, but you actually want all the numbers to be treated the same so you can do like calculations on them and stuff. So how do we do that?

Well you don’t want to do anything to the first column because that’s definitely not a number, and you want to yank out those commas because they are for humans and not computers, and then you want to cast everything to a number. It’s easier in python that in English:

df.iloc[:,1:].replace(',','',regex=True).apply(pd.to_numeric)

Conclusion

pandas is pretty awesome and with a bit of preprocessing and functional programming you can bash the data into the shapes you want it to be. You can do processing at the read_csv stage with the converters function if you know what columns you are getting and what you want to do with them. Sometimes though, you just know you are getting a glorious data file full of little monsters you need to handle somehow, and with a series of transformation steps you can absorb almost anything.