The molten core of agile

gold-earths-core-gold-facts

Having spent the second half of my software engineering career doing ‘agile’ development, I think I’ve spent more time trying to convince people what agile is not than explaining to the uninitiated what it isSince the so-called manifesto was published in 2001, the whole thing’s been so abused, monetized and co-opted that, for all intents and purposes, it seems to have lost any tangible meaning. By 2014, I had basically decided I would simply use the word ‘lean’, since the culture around agile was so polluted with nonsense…

What made this thing so popular in the first place? I remember reading about it and feeling like it was a bit revolutionary… I was excited by it. Excited because, in many ways, it was a repudiation of decades of stodgy and bureaucratic software practices that I always despised. My blog is called ‘On the Contrary’, after all, so I naturally found this upending of the table appealing.

If I peel back the layers and get at the hot molten core of agile, though, what is it really? What was its intent?

Continue reading

Build a testing database with SSDT and NuGet

My central idea was to use SSDT to build a dacpac file that would define the schema of our testing database and wrap that DACPAC artifact in a nuget package that testing projects could reference.

It’s not uncommon during early attempts to bootstrap CI and automated testing for an engineering team to start with ‘unit’ tests that read and write to a common development database. Truly, these should be considered more integration tests than unit tests, since they touch more than a single layer of the architecture, but that’s not the main drive of this particular post. My primary goal in adopting testing culture is increasing the amount of reliable and automated testing. Moving towards more pure unit tests will be a drive towards making our automated tests faster (and probably improved architecture)… but that is a problem for a future day.

Continue reading

Build culture not process

When launching a new business or a new initiative, I see lots of teams make the mistake of obsessing over what process they are going to use to manage the software production efforts.

They start, typically, with some ‘textbook’ example of SCRUM©… this quickly leads to obsessive debate over what ticketing tools to use… which then devolves into silly discussions over what ‘states’ the ticketing process should have (‘waiting for tests’, ‘testing in progress’, ‘testing failed’, etc)… Add in some arguments about what roles would be on the team and the delicate parsing of the responsibilities that do, and do not, belong to each… it doesn’t take long before I’m wondering: ‘When did we stop being software engineers and become process engineers?’

The problem is that we’ve forgotten the FIRST principle of the agile software movement:

Individuals and interactions over processes and tools

Oh yeah… forgot about that.

Continue reading

Management by PowerPoint

Serious problems require a serious tool… Meetings should center on concisely written reports on paper, not fragmented bulleted talking points projected up on the wall.

-Edward Tufte

If every software meeting that you attend is being driven by a PowerPoint deck, don’t kid yourself: you’ve got yourself a serious problem. This is a sign of an acute infection of bureaucracy.

Let’s face it, most software decisions… whether it be design, prioritization, process, budget… are conversations that need to be had at a detailed level. PowerPoint presentations, almost by design, forces your meeting participants to gloss right over those details. It implies tacit agreement with high level, and mostly generic points and statements that often bear little to no significance to the problems being discussed.

Let me illustrate by identifying some glaring symptoms of the PowerPoint infection:

Continue reading

The Power and Simplicity of the Data Warehouse

“In many ways, dimensional modeling amounts to holding the fort against assaults on simplicity”
– Ralph Kimball, The Data Warehouse Toolkit

 

Although there are many reasons that an organization may consider building a “data warehouse”, most of the time the overwhelming reason is performance related… a report or web page is just taking too long to run and they want it to be faster. If that data could be pre-computed than that page would be really fast!

I’m here to tell you that speed is not the most important reason for building the warehouse.

When you’ve got a system where multiple consumers are reading the data from the transactional store and doing their own calculations, you create a whole bunch of other problems beyond just the speed issue:

  • You create the potential for multiple different and conflicting results in that system. At least a few of those consumers will inevitably do something wrong.
  • You put a considerable burden on your transactional system to recalculate that data over-and-over again for each custom request.
  • While those consumers are running their long running queries, that data is being simultaneously updated by a multitude of data collection and transformative processes… you are not only reading inconsistent sets of data in the consumers, you are blocking the collection and transformation processes from doing their job, getting in their way and slowing them down… and sometimes even causing them to fail.
  • You’ve created a multitude of intertwined dependencies in that system. This makes attempts to improve or otherwise change that transactional system extremely difficult to achieve without breaking things… and having to make massive system wide changes to accommodate even the smallest change.
  • The bottom line is this: You’ve just got a greatly over-complicated system that is much more prone to performance problems and defects. As Ralph Kimball states so eloquently, data warehouse efforts are a giant move towards simplicity. And simpler systems are better systems.

We recently launched a major warehouse initiative to, once and for all, pre-compute and store all our portfolio-level data. Although that data is already computed from data that’s been pre-aggregated at the account level, there is still considerable additional work required to aggregate that data further to the portfolio level.

Primarily, that’s a major performance problem. Pulling a single portfolio’s data can take as long as 5-7 minutes for larger portfolios. That’s a serious problem with our scalability and an overall burden on our system.

I’m happy to report that the portfolio warehouse initiative is nearing its conclusion and am confident it will do things for us far beyond the performance improvements we hoped to gain:

  • With every consumer pulling from the same warehouse for portfolio level information, we can guarantee they will get the same results… they are all “drinking water from the same well.”
  • The portfolio data can now be processed “incrementally”… i.e. rather than have to recalculate that data from the beginning of time for every request, we can reprocess only that data that has changed. This pays huge dividends on overall system performance and greatly decreases the burden on the system.
  • Our data will now be pulled from a snapshot-enabled data warehouse. This guarantees clean and consistent reads without blocking the transactional store from doing its work.
  • By having one system that reads transactional data and compiles and stores the portfolio data, we only have one system to change when we want to change something in the transactional store. This is hugely liberating to us when we want to modify those underlying systems.
  • The new published warehouse structure for portfolios is simple and easy to understand. It therefore opens up consumption of that data in new ways with less effort, opening the doors to new possibilities that were otherwise impossible. Looking at data for all the portfolios in a firm in one pass is now possible, performing cross-firm analytics that were unthinkable before. This opens a myriad of optionsfor us that we intend to take advantage of.
  • Oh, and the speed is nice also… it’s really fast!

While we are still in the final stages of implementation, we hope to bring this system fully into production over the next few months and are very excited about the possibilities… we hope you are too!

And if you’d like to read about data warehousing, I highly recommend what is, in my opinion, the bible of data warehousing:

The Data Warehouse Toolkit – By Ralph Kimball and Margy Ross

Fortigent

“In many ways, dimensional modeling amounts to holding the fort against assaults on simplicity”
– Ralph Kimball, The Data Warehouse Toolkit


Although there are many reasons that an organization may consider building a “data warehouse”, most of the time the overwhelming reason is performance related… a report or web page is just taking too long to run and they want it to be faster. If that data could be pre-computed than that page would be really fast!

I’m here to tell you that speed is not the most important reason for building the warehouse.

When you’ve got a system where multiple consumers are reading the data from the transactional store and doing their own calculations, you create a whole bunch of other problems beyond just the speed issue:

  • You create the potential for multiple different and conflicting results in that system. At least a few of those consumers will inevitably…

View original post 587 more words