Eriksen Costa

Introducing Kotlin Money

Tue, 08 Oct 2024 07:00:00 -0300

Manipulating monetary amounts is a common computing chore. However, no mainstream language has a first-class data type for representing money, it’s up to programmers to code abstractions for it. This isn’t an issue per se until dealing with rounding issues from operations like installment payments (e.g., buy now, pay later), foreign exchange, or even simple things like fee processing and tax collection.

Inspired by my days at N26 dealing with these challenges, I introduce Money: a Kotlin library that makes monetary calculations and allocations easy:

val price = 100 money "USD"                     // USD 100.00
val shipping = 5 money "USD"                    // USD 5.00
val subtotal = price + shipping                 // USD 105.00
val discount = 10.percent()                     // 10%
val total = subtotal decreaseBy discount        // USD 94.50

val ratios = listOf(60.percent(), 40.percent()) // [60%, 40%]

total allocate 2                                // [USD 47.25, USD 47.25]
total allocate ratios                           // [USD 56.70, USD 37.80]

The library supports mathematical operations with monetary amounts, calculations with percentages, and allocation, making it simple to model use cases like those mentioned. Cryptocurrencies are also fully supported out of the box:

val price = 0.01607580 money "BTC"           // BTC 0.01607580
val transactionFee = 1.25.percent()          // 1.25%
val total = price increaseBy transactionFee  // BTC 0.01627675
val installments = total allocate 3          // [BTC 0.00542559, BTC 0.00542558, BTC 0.00542558]

val rate = 62_555.60 money "USD"             // USD 62555.60
val totalInUsd = total exchange rate         // USD 1005.63

Allocation

One of the nicest features of the library is its allocation capability. Allocation allows the distribution of a monetary amount into parts while guaranteeing that the sum of the parts equals the original value. For example, a retailer may accept purchases by credit card installments or by buy now, pay later (BNPL). What happens when a customer makes a purchase totaling USD 100.00 to be paid in three installments?

val price = 100 money "USD"
val number = 3
val installment = price / number
val installments = List(number) { installment } // [USD 33.33, USD 33.33, USD 33.33]
val total = installments.sum()                  // USD 99.99

As you noticed, there is a loss of USD 0.01. A penny here and there may seem a slight loss, but it may be costly over time. But there are other complications as well, such as overcharging a customer (which can be an infringement of consumer rights in several countries) due to rounding issues. The library provides a handy allocate() method that guarantees the result won’t differ from the original amount:

val price = 100 money "USD"
val installments = price allocate 3          // [USD 33.34, USD 33.33, USD 33.33]
val total = installments.allocations().sum() // USD 100.00

To allocate in proportional parts, pass a list of Percentage values to the method:

val amount = 2345.89 money "USD"
val result = dueAmount allocate listOf(50.percent(), 30.percent(), 20.percent())
val allocations = result.allocations() // [USD 1172.94, USD 703.77, USD 469.18]
val total = allocations.sum()          // USD 2345.89

As you can see in the previous examples, both results totaled up to the original monetary amount. No cent was lost or gained. By default, the library automatically allocates the difference. But you can tweak how the difference is allocated in the allocations list. For example, suppose your company requires the difference to be always allocated to the last item. You can do it by creating the allocator object directly with the desired allocation strategy:

val price = 100 money "USD"
val allocator = EvenAllocator(OnLast)
val installments = allocator.allocate(price, 3) // [USD 33.33, USD 33.33, USD 33.34]
val total = installments.allocations().sum()    // USD 100.00

Wrapping up

This post is just a glimpse of the library’s capabilities. I intend to keep the library’s API concise and to expand its capabilities gradually, including supporting Android development and out of the box persistence and serialization. Nevertheless, I hope it’s useful in its current version for people manipulating monetary amounts in Kotlin projects.

Refer to the usage guide on how to work with Money. The library has built-in support for 306 circulating currencies and 2283 cryptocurrencies. The installation procedures are explained in the project’s README. Give it a shot!

Platform engineering at N26: how we planned and launched it

Tue, 31 Oct 2023 13:00:00 -0200

This article is part of a two-piece series on platform engineering. The first piece explains platform engineering and its capabilities, roles, and benefits for DevOps maturity.

At N26, we can spin up a new service in up to 1 hour in the production environment. You read it right. A new service with all its infrastructural dependencies provisioned and fully operational, with a dedicated database cluster and with its API accessible for authenticated requests.

In 2021, when we started to assemble the team to work in our Brazilian operation, we knew that we would need to build a robust culture and environment to match the business’ ambitions. In a crowded neobank market, being able to release new products and services quickly is vital to acquiring and retaining customers.

Our current startup funding winter shifted the attention to survivability and, thus, to profitability. Delivering fast isn’t enough anymore. Startups must be able to anticipate customers’ needs instead of using their nimbleness to experiment randomly just to discover failure retrospectively. Anticipation is faster and cheaper than experimentation.

That’s why platform engineering is in all the rage now. Platforms are essential to raising the productivity of product teams, letting them focus on what matters, which is how to satisfy better customers’ needs and deliver business outcomes. Robust platforms take the pain away from dealing with infrastructure by automating repetitive tasks and by dealing with cross-cutting concerns like security, compliance, and observability.

Two years later, our platform engineering strategy is paying off. We’re a high-mature DevOps organization, performing better than 97% of the 2022 State of DevOps Report respondents in the financial services industry. Our journey may provide you with interesting insights on how to bootstrap your engineering platform. In advance, you’ll need to flex your product management muscles. After that, we’ll see how N26’s platform works.

Our Software Engineers deployed to production at least once every two days (deployment frequency). Each deployment took 9 minutes on average (lead time). Only 1.2% of the deploys led to incidents (change failure rate) that had a restoration time of 21 hours on average (time to restore) — data from July 2023.

Culture eats strategy for breakfast

Let’s time travel again. In July 2021, when we started to lay out the Engineering strategy for N26’s Brazilian operation, we were at the peak of activity to deliver the first version of our mobile app in Brazil. Unlike the European expansion, in Brazil, we built everything from scratch. This first version would be a closed beta available only for family and friends. Our goal was to test our operation and set the groundwork to comply with BACEN regulations (Brazilian Central Bank).

You may be wondering if this is a fortunate case of a greenfield project. It isn’t. When I joined N26, all the product and engineering efforts were outsourced. We even depended on our partners’ infrastructures.

This arrangement was good enough for the first release of our app, as it was a closed beta experiment. However, it wouldn’t suit our business ambitions. In a crowded neobank market, the unique value proposition requires a dedicated staff with skin in the game to learn continuously and propose solutions to actual customers’ problems.

At the time, we were a five-person Engineering team. I set time apart to run a three-day workshop to formalize our strategy and culture. Before diving into our strategy, we discussed our culture. Defining the social game was vital to us as we were on the brink of scaling up the Engineering team for the next phase of our operation. Hiring quickly and onboarding people while speeding up productivity is a challenging feat.

While discussing our principles, we have found ourselves repeating how responsibility mattered. We wanted autonomous and responsible teams at N26. For us, teams must maintain what they deliver. This approach brings teams in contact with the operational part of their applications and the customers, both feedback loops that lead to building quality into what is delivered. This approach is known as you build it, you run it.

Other principles came naturally to us. One was that all the infrastructure should be abstracted in a platform. Another important principle was to use proven management practices, including product management and techniques, to manage our Engineering department. So, platform engineering was codified organically into our culture.

After setting our culture and with the pillars of our strategy in place, it was time to start the strategic planning.

Strategic planning

To start the strategic planning, I ran an Impact Mapping session. The technique is highly collaborative and consists of answering four questions: “Why?”, “Who?”, “How?” and “What?”. It’s a simple yet effective technique. Let’s learn by following our impact map.

The most important question is, why are we doing this? Knowing the goal lets people adapt to unforeseen circumstances. An impact map always starts with the why.

We were on the path to rapid growth. The business plan was aggressive on customer growth, and we anticipated a 20 times increase in our team, from 5 to 100 people¹ until the end of 2022. It is a massive growth in just 1.5 years. More than a robust platform, we needed excellent developer experience.

Our goal was to make it virtually easy to deploy on day one.

Our answer to the “Why?” question was to make it virtually easy to deploy on day one². But before jumping into which features we needed to develop to achieve this goal, we discussed who would be impacted by this goal. Who was our customer?

Our customer was the product engineer.

Our answer to the “Who?” question was the product engineer. It may seem odd to identify an internal customer, but at this phase, I knew we should refrain from accruing organizational debt. As Steve Blank — the Lean Startup movement’s godfather — points out, organizational debt can kill a company quicker than technical debt. Our main contribution to the business as an Engineering department was to staff the teams adequately and make them productive. The teams would contribute directly to the business by solving specific customer problems.

Then, we followed up to understand which behaviors we wanted the product engineers to have to help us achieve our desired goal.

We wanted the product engineers to maintain what they built and to be productive on their first working day.

Our answer to the “How?” was twofold. First, we wanted our product engineers to be responsible for maintaining what they built. Secondly, we wanted them to be productive on their first working day. These were the impacts we desired to create. Finally, we discussed what we could do to support the expected impacts.

Deliverables or features support impacts (click or tap to enlarge).

Our answer to “What?” was a list of features. But note, these features weren’t a shopping list of what we wanted to do. These were fully contextualized features aligned with the desired impacts and the business goal. The presence of non-technological features (e.g., getting started instructions and on-call documentation) makes it more evident.

With an essential part of our strategy well understood by the attendants, it was time to understand better which kind of experience we envisioned delivering to our fellow product engineers.

Mapping the Developer Experience

The impact map we created revealed two features that made us wonder which kind of developer experience we wanted to provide to the product engineers. They reinforced the infrastructure abstraction principle (the low-level details of the infrastructure should be abstracted for the product engineers) and the homogeneity and consistency³ principle that we set in our culture document.

The features that guided the mapping of the developer experience.

So, we started discussing the vision we had for the platform. We wanted to provide an experience that would let product engineers be productive on their first working day. We then started discussing our previous experience with environment provisioning processes.

Have you ever stopped to think about how the provisioning of new services works at your company? Is it entirely automated? How much manual work does it require?

I worked before in a neobank with almost twenty million customers and 350+ services when I joined. The few who knew the required steps of a new service provisioning were proud: it seemed like tribal knowledge. The engineering platform was incredible, but this part of the developer experience lacked self-service capabilities. I remember people saying it took at least two weeks on average to deploy a new service in production due to the amount of undocumented manual work.

But if two weeks seems too much, do you imagine waiting three months to have a working environment to deploy your service? A recent study by Rafay Systems found that one in four organizations takes three months or longer to deploy an application or service from code-complete to production. 9% of organizations take six months or longer. That’s a huge waste of resources.

A quick provisioning process seemed a good idea, as we could even deploy a new temporary service to production on the onboarding classes if we wanted to. We then mapped the journey of our product and platform engineers for the provisioning of a new service. We imagined a journey to provide a provisioning experience with the least effort possible to the product engineers.

The user story map of the developer experience we envisioned creating (click or tap to enlarge).

What we found in our User Story Mapping session was a vision where our platform would solve this provisioning problem through extensive automation. We wanted a PaaS-like experience: product engineers would need to open a pull request with some lines of YAML code in a repository that would orchestrate the bootstrapping of new services in our cloud environment. We even discussed a draft of how this file would look.

After the pull request approval, the orchestration would prepare the production environment. We wanted it to take care of things like computer cluster management, database provisioning, setting up the messaging infrastructure, and so on. Also, we wanted the orchestration to bootstrap the project. It should create the repository of the project based on a service chassis and make the first deployment into production to guarantee the environment was configured correctly. Then, the product engineers would clone the repository of the newly created service and start working on it. Every push into the trunk branch would deploy the service into production.

With a broad overview and shared understanding of the developer experience, we were ready to dive deeper into what we needed to build.

Planning to build less and the Thinnest Viable Platform

Our deep dive started with the provisioning process and the service chassis as the starting point. We used the Story Workshop technique to decompose both features into three slices: good enough (what are the minimum characteristics to make the feature functional?), better (what would make it better?), and best (what would make it fabulous or ideal?). Slicing techniques help to understand the scope size and to prioritize better.

Slicing the platform helps to prioritize the Thinnest Viable Platform (click or tap to enlarge).

The workshop reminded us of how complex software engineering is. We discussed simple concerns like database migrations and logging and more advanced things like circuit breakers and security on the Software Delivery Lifecycle (SSDLC). We had some idea that part of the discussed features would be required early on, like adopting the Money pattern to prevent rounding bugs in operations like installment calculation and credit financing. They’re costly and more frequent than we think.

However, we also knew we neither had enough people nor had time to build everything we wanted. That’s the harsh truth of software development: you never have enough people, time, and money to build everything you want. So, focus and prioritization are key. And that’s the strength of the Story Workshop technique, as it helps us define the Thinnest Viable Platform (TVP).

A TVP is the simplest platform you must aim to build. In some cases, the TVP may be a documentation of components or services that serve as building blocks for software development. Nonetheless, defining a TVP is important because software engineers love to build technical things, and a platform has the potential to be a never-ending journey of half-baked features. Without focus, there’s a risk of delivering a bloated platform that ignores the developer experience.

We set a goal to prioritize which features to deliver in our TVP. We wanted to create a provisioning process that could be run within a week⁴. The initial feature set would help us build the backbone of our platform.

Our first goal was to create a TPV that supported provisioning a service within a week.

So, we prioritized the setup of the computing cluster (Kubernetes), the orchestration automation⁵ (which we named “Project Bootstrapper”), and the first version of the service chassis. But how could someone deliver something without any kind of database support? We taught the team that they could use in-memory repositories for these purposes. Shortly after the TVP release, the platform team released the database provisioning feature. From there on, the product teams could deploy fully operational services in production.

That’s an important product management lesson for anyone doing platform engineering: your releases can solve the problem partially. Plan to build less and provide alternatives for the customers and stakeholders that they may implement while you work on the definitive solution. Alignment and communication are essential to work around temporary dependencies.

Our platform evolved dramatically since then. But every step of its progress was marked by the diligent work of the platform team, who always planned to advance the platform’s capabilities iteratively. Central to this process is the understanding of the business goals, product engineers’ needs, and regulatory requirements to prioritize features that deliver value to the company.

How our platform works

Fast forward to July 2023. Our platform evolved, and the orchestration makes everybody’s lives easier. To create a new service, the product engineer clones the repository that manages the infrastructure and runs a make command:

$ git checkout -b overnight
$ make create-team spaces

The command scaffolds a directory where the product engineer’s team will edit a single configuration file to bootstrap the new service. The file⁶ allows configuring things like the Service Level Objectives (SLOs) and the database cluster size:

locals {
    projects = {
        overnight = {
            description = "Calculate the balances' CDI-indexed yield."
            repository = {
                bootstrap = true
                language = "kotlin"
            }
            service = {
                bootstrap = true
            }
            monitoring = {
                bootstrap = true
                error_rate = "0.5"
                read_latency = "0.4"
                write_latency = "4"
                apdex = "0.94"
            }
            database = {
                bootstrap = true
                instance = "db.r6g.large"
                replicas = 4
                engine = "aurora-postgresql"
            }
            queues = {
                bootstrap = true
                names = ["invoice-paid", "yield-calculated"]
            }
            logs = {
                archive = true
            }
            cdn = {
                bootstrap = false
            }
        }
    }
}

Then, the product engineer commits the files and creates a pull request, which the platform team reviews. We have a policy in place to prevent microservices envy. As Domain-Driven Design practitioners, we prefer to design our services around bounded contexts and split the services only after thoughtful consideration. The platform team review checks the conformity to the policies.

After the pull request is approved, the orchestration is triggered. The orchestration:

Creates a service user for the application in AWS IAM.
Creates a Kubernetes service with horizontal and vertical pod autoscaler enabled.
Integrates the API Gateway with the Kubernetes ingress to expose the service’s API.
Creates an Aurora cluster in AWS RDS with horizontal autoscaling enabled.
Creates messaging queues/topics.
Creates an S3 directory integrated with a Content Delivery Network (CDN).
Stores credentials (API keys, database passwords, and so on) in the secrets management system.
Bootstraps the new service codebase using the service chassis.
Creates the new service repository in BitBucket.
Configures the BitBucket repository variables.
Creates the CI/CD pipelines.
Configures the quality and security gateway.
Runs the first deployment of the service.
Runs the first deployment of service workers in sidecars.
Creates a Service Level Objective (SLO) in Datadog.
Configures service alarms in Datadog.
Runs health checks to guarantee the provisioning worked.

The service chassis is a Ktor-based opinionated framework that packages a curated set of open-source libraries and internally developed components to reduce the cognitive load on concerns like persistence and logging.

A simplified diagram of how the orchestration works.

After the provisioning finishes, the product engineer clones the repository and starts working on the new service. The service chassis automatically generates a Docker Compose file to support local development. All projects share the same bootstrapping structure, which helps the product engineers when they switch between projects. Furthermore, less variability means faster optimization cycles for the platform engineers when enhancing the underlying tooling used by the Engineering team.

The services are also entirely instrumented. Early on, product engineers can monitor the application performance and troubleshoot issues like slow queries and low Apdex scores. Alarms are configured automatically and sent to standardized Slack channels (each team has a standard set of Slack channels, one dedicated to observability alarms).

To fine-tune the available computing capacity, the product engineer may update the configuration file to increase the number of Kubernetes pods required to run its service. The only way to set up the production environment from a product engineer’s standpoint is through this configuration file that abstracts the underlying cloud infrastructure. This guarantees that all changes to production are transparently audited.

We also have a set of things that require manual work but must follow guidelines or standards. Some guidelines have emerged from peer-reviewing processes, like the API design review. We strive to design our API semantically. Initially, we peer-reviewed changes to the API contract. Currently, the team follows the conventions that are available in a design document.

We can’t stress enough the importance of documentation. We documented the most critical parts of the product engineer journey, including the entire Software Development Lifecycle (SDLC). For example, product engineers can learn how to create new credentials and store them secretly in our cloud environment or adjust alarm thresholds by reading the documentation. Automated processes and documentation are the base of a self-service platform.

Summary table of the platform’s features with links for further information (click or tap to enlarge).

When a new person joins our team, our identity provider sends a welcome email with instructions on how to activate the laptop that is sent to the person’s house some days before their first day of work. The laptop is fully configured and updated to the latest version of the operating system and development tools. The only work the software engineer needs to do is to clone the repositories and run the starting scripts (generally Docker Compose commands) available in the README files.

To make our web-based tools readily available (e.g., Rancher, Datadog, Jira, Confluence, 1Password, Slack), we used Okta as the identity provider. With a single password, the person unlocks the laptop and access all the tools. Expanding our platform scope to issues like access management and laptop setup helped us to implement Engineering-wide changes like the rolling of tooling updates and enhancements without interrupting people to run setup wizards manually.

Closing remarks

Platform engineering at N26 has a long and winding road. From custom-built orchestration tools to standardization of manual processes, we found a way to make deployment virtually easy on the first day. In the same journey, we eliminated the dependency on third-party infrastructure, which resulted in an incredible drop in our API latency. The final payoff was an immense increase in developer productivity and a highly mature DevOps organization.

The main takeaway is to manage your platform strategically. There is much more to build than enough people, time, and money. A product mindset is paramount. Take advantage of available tools, especially open-source tools, to cut out scope. Focus on the developer experience you want to deliver and the outcomes your company pursues. Invest in building a resilient team and a developer experience that enables it to anticipate customer needs.

Acknowledgments

The platform engineering journey described here was possible due to the efforts of many people. Our strong foundation was born from the minds of seasoned professionals. Antonio Spinelli, Henrique Sloty, and Thiago Costa contributed heavily to strategic planning. All of them worked on the first features of the platform before moving on to work as the first engineering managers of the company.

Henrique Sloty has been managing the Engineering Platform area since the release of our TVP. Abner Maioralli joined the team and worked as the Technical Product Manager, supporting the planning, discovery, and communication processes. Although not discussed in this article, we also have a platform for mobile software development. Matheus Schmidt and Lucas Calandrine are the minds behind this platform.

Footnotes

In late-stage and geographically distributed startups like N26, it is common practice to use business plans to define the funding needs for projects. The leadership of the Brazilian operation had to plan and determine what products and services we envisioned to deliver to our local customers. I established a simple guideline and explained to the CEO and the leadership team that teams must have a minimum size to work on the entire SDLC. The projected growth was a tenfold increase in the Brazilian staff, increasing to 300 employees by the end of 2022. The Engineering team would grow to 100 people or a 20 times increase. ↩
The inspiration came from Etsy, one of the early DevOps pioneers. ↩
At the time, we had previous experience in both homogeneous and heterogeneous environments. We found ourselves biased towards homogeneous environments due to hurdles we felt in previous experiences, like services built around personal technology preferences. The lack of strategic management in Engineering may leave a trail of chaos: services that nobody wants to support because of inexperience in the programming language, database technology, and so on. We were not against the usage of technologies best suited for the problem at hand (use the right tool for the job), but we would bet on building a homogenous and consistent platform first and then branch out as needed. ↩
Remember that we depended on third-party infrastructure? We had three main issues with it: it was a proprietary platform (which created non-portable workflows), it had high latency and a slow development cycle. Provisioning a new service within a week seemed fair. ↩
Initially, the platform engineers of the Cloud Platform were unsure if they could deliver this orchestration. Then, Henrique Sloty, who would be the Engineering Manager of the entire platform team, ran a spike solution. Basically, he developed a bash script that made the orchestration possible using the BitBucket pipelines. The result was that the first release of the platform provisioned new services in under 15 minutes. ↩
The TVP used YAML for this file. Later, the file syntax was ported to HCL to simplify the platform maintenance. ↩

Platform engineering: capabilities, roles, and its benefits for DevOps maturity

Mon, 25 Sep 2023 06:30:00 -0300

This article is part of a two-piece series on platform engineering. The second piece describes how N26 planned a robust platform, improved productivity, and reduced costs.

Imagine you work in a modern technical environment, be it a startup or not. It’s Wednesday, and you’re ready to release a much-waited feature to the customer base.

You and your team then start checking out item by item of the release plan. You then check the last one just after switching on the feature flag. Now, it’s time to monitor the customers using the feature.

Some minutes later, you spot some warnings in the observability tool. You investigate the issue while the CX personnel open tickets with customer complaints. You check if the number of running Kubernetes pods is healthy and see that the cluster is starting more pods to cope with the load increase caused by the spike of errors. The database load is adequate, but the dead letter queue accumulates messages in a repeated pattern.

You add some extra lines to log what is happening. You copy and paste error-handling code from another project using the same programming language used by your microservice project. Then, you commit the code and wait while the CI/CD tool deploys it to the cluster.

Later on, you find that the access policy of the production environment is stricter than the sandbox environment. It seems the service isn’t able to publish messages to a secondary queue, which is used to eventually migrate required data for the new feature to work with a specific segment of the customer base. It isn’t working because you didn’t follow the naming rules.

You then decide to rename the queue by changing some lines of Terraform code and submit it to the approval of the Infrastructure team. Once approved, you redeploy the service with the fixed queue name, and everything starts working as expected. You’re happy because you solved the incident in under a couple of hours but also frustrated with how social media complaints overshadowed the launch campaign. Besides all, you all felt overwhelmed.

You build it, you run it

Our introductory tale has shown many attributes of companies running modern technical environments and practices. The team was able to deliver software using automated processes and was also able to set up the production environment by making changes to Terraform, an Infrastructure as Code tool. The Operations are available as a self-service, automated, and on-demand available dependency. A DevOps culture is established.

Besides DevOps, the team developing this feature is fully responsible for its software’s operations. They monitor it actively and immediately act when things go wrong since smooth operation is the top priority. Investments are made in quality to lower the maintenance burden.

The team owns the entire product cycle. The team creates a sense of how regular operation feels by working over the whole software development lifecycle (SDLC). Also, it can understand better how customers react to changes by correlating operational (application telemetry) and business metrics (KPIs, health indicators, and improvement drivers). This company adopts the “You build it, you run it” (YBIYRN) approach.

However, even with these practices, our tale spots hardships common even in highly mature DevOps organizations. The lack of a comprehensive engineering platform increases cognitive load, affecting productivity. Let’s understand what is the cognitive load before discussing platform engineering.

Cognitive load

Have you ever felt mentally exhausted after learning something new, like a new language, technology, or a different way of working? While learning, have you ever felt tempted to check a notification on your mobile phone?

That’s the forces of the different cognitive loads acting on your cognitive system. The Cognitive Load Theory was introduced in the 1980s to explain how our learning ability is heavily constrained by our working memory. Learning is a process that requires our cognitive system to process information in the working memory and then store it in long-term memory.

However, the working memory is highly constrained in capacity and duration and only holds information briefly. The exception is when the working memory deals with abilities previously learned since it retrieves information from long-term memory. That’s why while you’re learning something new, you practice it until you do it automatically. You experienced it many times in your life while learning to speak, read, write, dance, you name it.

The sensory memory filters out most of the incoming information. Learning is the act of encoding new schemas in the long-term memory.

Understanding the kinds of cognitive load is vital to designing an environment that prioritizes learning by adjusting the load on the working memory:

Intrinsic Load is imposed by the complexity of the information and the previous expertise of the person in the subject matter. That’s why we break down projects into smaller deliverables since building something by joining simpler parts is more manageable
Extraneous Load is also known as ineffective load. It diverges cognitive resources to irrelevant activities that do not contribute to learning. That’s why we must create workplaces that allow people to focus by diminishing distractions and improving processes
Germane Load is the effective load imposed on the working memory by the process of learning. Transferring information from the working memory to the long-term memory requires effort. This effort is the Germane Load. In the workplace, this happens when discovering how to solve a (business, customer, programming) problem

Extra Intrinsic and Extraneous Load must be eliminated or minimized to optimize learning. Back to our tale, we want more time for the team to learn: did the new feature create value for the customers? What should we optimize based on the gathered feedback? Working on these things is more valuable than fiddling with infrastructure configuration. Providing an engineering platform is critical to increasing Germane Load¹ by working on value-adding activities.

Platform engineering

Platform engineering is an emerging practice that improves the developer experience and productivity by providing a compelling integrated product — an engineering platform² — that will reduce cognitive load by delivering self-service software engineering capabilities.

Put simply, this platform will package everything from developer tools to processes and standards. Imagine, for example, that the team in our introductory tale works in a fintech startup, and their working domain deals with calculating installment payments. What happens when you divide USD 100 into three equal installments? You get three installments, each one valued at USD 33.33. Where’s the missing cent?

No penny must be lost when doing monetary calculations. It may seem silly, but rounding errors cost Uber tens of millions of dollars in 2017 and devalued a Canadian stock index by 50% in 22 months. So, instead of leaving each team in this startup to repeatedly program code that does monetary calculations, a single library must be provided to do it for them.

If monetary calculations seem an obvious example, we can move on to compliance issues. Have you ever had problems complying with GDPR-like privacy laws? Imagine the team from our tale wrongly logging Personal Identifiable Information (PII) like the social security number or credit card for debugging purposes. Now you have in your log history data that shouldn’t be there.

The solution is implementing ways to deal with PII in the engineering platform, preventing them from leaking into the logs. The platform team can redact PII on the application level (in a service chassis) and log stream. This way, the teams are freed from remembering which data they may add to the log. If they mistakenly add anything holding PII data to the log call, they will rest assured that it won’t appear in the records.

In our example, the engineering platform improves the last-mile developer experience of the existent DevOps platform. Remember, in our tale, the team was able to provision infrastructure through self-service tools. The engineering platform in this example would also package utilitarian packages (e.g., ORMs, monetary calculation libraries) and cross-cutting concerns (e.g., logging, security, error handling) in a compelling and standardized service chassis used when bootstrapping new projects in the company.

The team consumers the platform using self-service tools. Some services like the CI/CD, container cluster, and database may be super standardized. However, the team glues everything else, like logging, quality gateway, and authentication.

The team uses a service chassis that exposes the platform’s services and features with opt-in or opt-out toggles. The platform services are standardized and consistently consumed throughout the organization.

With a comprehensive engineering platform like this, the company would evolve its DevOps approach. The teams could focus more on the applications they need to develop and on learning what is driving the business forward and what is solving their customer needs. They can focus on value-adding and learning activities instead of working on extraneous things (reduced cognitive load). It is worth noting that platform engineering improves productivity, leading to more motivated individuals due to better developer experience.

For me, platform engineering is an organically evolutionary step of different industry-wide experiences and adopted approaches like DevOps. Speaking of DevOps, let’s expand the relationship of platform engineering with it and how it leads to higher DevOps maturity.

Higher DevOps maturity

If platform engineering is an evolutionary step of DevOps and other practices³, how can it help to lead to higher maturity? First of all, DevOps practitioners have been creating self-serviced platforms for the last ten years. The DevOps culture evolved into an approach where automation became self-serviced platforms managed as products. From the DevOps Handbook:

Instead of IT Operations doing manual work that comes from work tickets, it enables developer productivity through APIs and self-serviced platforms that create environments, test and deploy code (CI/CD), monitor and display production telemetry, and so forth. By doing this, IT Operations become more like Development (…), engaged in product development, where the product is the platform that developers use to safely, quickly, and securely test, deploy, and run their IT services in production.

Moving from the packaging of operations as a platform to the entire developer experience (i.e., packaging more capabilities in the platform, like logging, security, and error handling) means less cognitive load for the teams working on products and features. Enhanced developer experience will drive the DevOps metrics to higher levels.

Indeed, the Puppet 2021 State of DevOps Report found a high degree of correlation between DevOps evolution and the use of internal platforms: 48% of the highly mature organizations used internal platforms against 25% of the mid-level group, and only 8% of the low-level group.

High performers use more internal platforms compared to mid-level and low performers.

I published N26 Brasil DevOps Metrics on LinkedIn earlier this year. We scored higher than 97% of respondents from the Financial Services industry. Our Software Engineers deployed to production at least once every two days (deployment frequency). Each deployment took 9 minutes on average (lead time). Only 1.2% of the deploys (change failure rate) led to incidents that had a restoration time of 21 hours on average (time to restore). A comprehensive engineering platform was prioritized back in the early days of our operation, contributing to developing a strong DevOps culture.

Platform engineering is an evolutionary step of DevOps. However, it requires expanding the scope of the platform to embrace the entire developer experience. Before diving into the platform’s capabilities, let’s discuss what is needed to create one. Not surprisingly, you’re going to need a platform team.

Organizational design

Platform engineering also has roots in organizational design, specifically after the groundwork set by Team Topologies, an approach to organize business and technology teams for fast flow.

Root to the approach is the acknowledgment that organizational design starts with a team-first mindset and that teams need to minimize the cognitive load on them by establishing good boundaries and clear responsibilities. That’s why one of the most important contributions of the approach was the introduction of its topologies and interaction modes. Discussing organization design with this precise vocabulary is refreshing. The topologies are four:

Stream-aligned team: aligned to a flow of work, which is usually a segment of the business domain
Complicated subsystem team: provides libraries to stream-aligned teams that solve computationally complex problems
Enabling team: helps stream-aligned teams with specialized skills (e.g., testing, Agile practices, database management)
Platform team: provides internal services to reduce the cognitive load, freeing the stream-aligned teams to accelerate their delivery rate

The four topologies.

The stream-aligned teams are responsible for the entire software delivery lifecycle; they’re YBIYRI teams. Engineering platforms are essential because they let teams own the whole lifecycle, eliminating handoffs and waste of manual processes. Platforms minimize the cognitive load while enabling stream-aligned teams to be fully autonomous.

Counter-intuitively, autonomy means restricting collaboration by reducing the dependencies between the teams. The approach defines three modes to understand the interaction between teams:

Collaboration: working together for some time to discover new things (products, APIs, technologies)
X-as-a-Service (XaaS): one team provides something as a service to another team
Facilitation: one team mentors another team

The interaction modes.

Your goal should be to restrict the collaboration interaction mode to specific periods, like new business opportunities that require discovering nouveau solutions. Two or more teams will collaborate closely to define a solution. Once the solution is delivered, the relationship changes to an XaaS in an upstream/downstream perspective.

The evolving organization landscape will trigger interaction mode rearrangements that will guide the relationships between the teams of all the topologies types. But to frame an example closer to our subject, imagine that a stream-aligned team found that they would need to use some new services available in the cloud provider to implement a new feature. At this moment, changing the relationship with the platform team to collaboration mode could be helpful to understand if this new technology needs only a change in the orchestration (i.e., changes in the Infrastructure as Code tool to provision the new service) or if it is something that may be abstracted for later reuse for the other teams.

A team topology with two teams in collaboration mode while consuming services from the Platform team.

In this setting, imagine that the platform team’s first deliverable is the provisioning of the new cloud service, freeing up the stream-aligned team to deal solely with value delivery to the end customer. After this delivery, the platform team may prioritize exposing the aforementioned service as a self-service tool or place it in the roadmap for a future release. Nonetheless, the interaction mode of both teams changes back to an XaaS mode.

After some time, the teams resume their relationship in XaaS mode.

It is worth understanding the approach before designing your teams. Also, the platform team may be composed of multiple teams of other topologies. As your organization grows, you may specialize the team in the different capabilities that compound the platform: numerous teams will work under the “platform team” or “platform area” umbrella.

The Platform team may be composed of multiple teams.

If you're just starting with platform engineering, I suggest having at least one platform team with Cloud Infrastructure and Software Engineers in your organization. You'll be able to plan and build your platform's first version, which may consist of a service chassis, CI/CD pipelines, and an abstracted self-serviceable cloud runtime. In an upcoming article, I will discuss how to plan a platform with an actual example.

Product Mindset

A platform can be a huge endeavor. As such, it is easy to get lost in the details and deliver something that is neither compelling nor minimizes the cognitive load of the stakeholders. Wait a minute? Stakeholders?

Building a platform means having a clear understanding of the platform’s users and their needs. As with any product, you’ll never have the time, money, and people required to build your product vision. So, platforms must be strategically managed like any other product. You’ll need a product vision and management processes like roadmapping and goal setting to align the stakeholders and communicate with them about the upcoming features and priorities.

Product management goes beyond strategic planning. The platform team needs to constantly survey users to see if the provided solutions are solving their problems. Every product must be delivered with complete support, including up-to-date documentation. Simplified onboarding mechanics should be in place to minimize manual tasks, granting a smooth self-service experience.

Starting the platform may be daunting. However, it may not need to be. You only need to discover the minimum viable product for your platform (or the shortest path to value). A handful of agile practices like Impact Mapping and User Story Mapping help you set a goal and prioritize just what is needed to deliver the hypothesized value. In an upcoming article, I will discuss how to plan a platform with an actual example.

Roles

You may have heard the job title “Platform Engineer” or “Product Engineer.” Translating in Team Topologies terms, the Platform Engineer works in a platform team while the second works in a stream-aligned team.

Is the Platform Engineer a different kind of engineer? What’s the difference when comparing with a Product or “regular” Software Engineer? I like to keep things simple. They’re different roles instead of job titles⁴. A Platform Engineer will develop platform-based solutions, which other Software Engineers will use. Product Engineers will develop solutions for end users, which may be other Software Engineers if the developed product is technical (e.g., a payments API). That’s why this distinction is more about the work being done, i.e., the role, than a job title. I don’t mind.

However, job titles aside, people working on platform teams require a different mentality. First, their motivation must be aligned with the fact that a platform team delivers value indirectly to the business. They’ll help to improve the organization’s productivity and also help to lower costs and standardize compliance and security controls.

Secondly, they must know data structures, algorithms, programming languages, operational systems, databases, shell scripting, and so on. It isn’t a different skill set from a “regular” Software Engineer working in a stream-aligned team. However, they must hone their API and software design skills, delivering technical products with excellent developer experience. These products must have well-designed APIs that are intuitive and easy to use. The code must set a high bar of measurable quality and solve well-defined use cases.

People who contributed to open-source projects are great candidates to work in platform teams. The only catch is to create a mindset of solving well-defined use cases. Open-source projects may have too broad APIs anticipating an unbounded number of use cases. But as a platform is for internal use, it is better to solve well-known cases than to anticipate future needs that may never happen. For SOLID practitioners, I'm not against the Open-Closed Principle. The opposite is true: I'm in for sound design, mainly if it reflects a rich domain model.

Capabilities of engineering platforms

Each engineering platform will evolve differently due to the pressing needs of the organization. However, some capabilities are valuable to present in the earliest versions of your product platform. Let’s take a cue from the DevOps Handbook:

By adding the expertise of QA, IT Operations, and Infosec into delivery teams and automated self-service tools and platforms, teams are able to use that expertise in their daily work without being dependent on other teams.

Most internal engineering platforms already package some of the mentioned capabilities in their self-service tools. We generally see some kind of automated infrastructure provisioning that spans a working environment in a cluster like Kubernetes. The same automation typically configures the CI/CD pipeline, running the automated tests for every code commit pushed to the version control system. The same pipeline may run tools like Snyk or Sonarcloud to inspect the code quality and security.

That’s a solid groundwork. However, an engineering platform requires the delivery of an integrated product. We must integrate the entire software delivery lifecycle processes, practices, and components into a compelling product. Software Engineers are constantly working with third-party code to ease dealing with the HTTP stack, database connections, and cross-cutting concerns (e.g., logging, authentication, authorization, caching, etc.) Packaging the frameworks and libraries in a standardized services chassis creates vast benefits.

A service chassis will help you standardize practices and processes while optimizing software dependencies’ governance if you’re using a microservices approach. The teams will also benefit from an easier way to navigate between the different software projects due to the standardization. Internal mobility of Software Engineers will also be improved as the ramp-up to learn a different business context will be smoother as the software development stack is well-known⁵.

Another benefit of this standardization is that platform teams can automate the platform further as they now control part of the application layer. For example, in our introductory tale, the platform team may fix the leaking of PII data to the log by changing the log component to redact a standardized data set. After building up the feature, the CI/CD would trigger a rebuild of every service. In a matter of minutes, the entire production environment would have a new compliance and security policy applied.

Closing remarks

Platform engineering is a trend with great potential for organizations willing to reach higher DevOps maturity levels. Also, it is an approach with the added benefits of improving cross-cutting concerns by packaging in compelling product capabilities like security, compliance, and observability, leading to more productive teams and motivated individuals.

Footnotes

Recent formulations of the Cognitive Load Theory state that Germane Load is not an independent source of cognitive load. John Sweller (2010) explains that Germane Load is a function of working memory resources devoted to the interacting elements that determine Intrinsic Load. Germane Load is merely the quantity of working memory resources available to learn and thus is indirectly affected by external sources of information. ↩
Commonly referred to as Internal Developer Platform (IDP). ↩
Big Tech companies are used to having platform teams and internal platforms. For example, Google developed communication protocols (Protobuf) and CI/CD tools (including its own version control — Piper).

But Big Techs aren’t an exception. Big banks like Goldman Sachs and Bradesco have had platforms since the 2000s (at least). These examples predate DevOps. ↩
Do you remember the old discussions about DevOps being a job title or not? The argument against it was that DevOps was a cultural movement and a set of practices (which is correct). Despite that, the market will always capture a trend and transform it into business opportunities. DevOps certifications, DevOps tools, DevOps jobs.

I prefer to stay as far as I can and use more neutral names because I think they communicate better. A DevOps Engineer is an Infrastructure or Cloud Engineer for me. Trends come and go and now Platform Engineering is on the rise, with many arguing it is not DevOps despite the strong correlation between them. My recommendation is to name things after trends only if you’re a vendor in the related space. ↩
You may be wondering if this kills one of the advantages of a microservices architecture. Indeed, I suggest having a single framework and using a curated set of libraries instead of leaving the teams with open-ended technological choices.

The benefits outlined are amplified with the passing of time due to the constant optimization of this homogeneous set of tools. Also, from the business management perspective, hiring people to work with a smaller set of programming languages is easier, even when not using a mainstream language.

I worked at Nubank in 2019, and they’re huge advocates of Clojure, a functional programming language running on the JVM. While sourcing people with past Clojure experience was practically impossible, people joined the company and were able to develop Clojure after some months. All the learning resources were concentrated on Clojure learning. Also, the service chassis was mature enough to let people use pre-existing building blocks to develop new services. More importantly, the toolset is heavily constrained.

At N26 Brasil, we used a similar strategy (further explained in an upcoming article). We chose Kotlin as our backend programming language and ktor as the base framework for our service chassis. A minority of our Software Engineers had prior experience with the language. But even so, the service chassis helped the development of projects with rich domain models that are easy to navigate due to the sharing of the same package structure. ↩

Building an Engineering culture

Wed, 22 Feb 2023 07:30:00 -0200

Culture is always a much-debated subject. Peter Drucker, the father of modern management theory, allegedly said that “culture eats strategy for breakfast”. When aligned with strategy and leadership, a strong culture leads to positive organizational outcomes.

Ashley Goodall and Marcus Buckingham say, in their book Nine Lies About Work, that culture matters because it contributes to identity, processes, and vision. Besides that, culture is the answer that people give when they are asked “how is it to work there?”. Culture determines how hard people will work and for how long they’ll stay in your company.

Culture matters. However, culture also may be a complicated and confusing subject. Because of that, culture is too often relegated to HR alone and treated secondarily by the business. But it is possible to run a managed process to define the culture and it takes less effort than you imagine.

Before diving into how to do it, it’s worthwhile to understand what is culture.

What is culture?

Culture has some definitions:

It is the knowledge system of a relatively big group of people
It is the nurtured behavior, it is the sum of what a person learned, of the accumulated experience which is socially transmitted – or, to make it short – the behavior through social learning
It is the lifestyle of a group of people – the behaviors, beliefs, values, and symbols that they accept (normally without thinking about them) and that is passed by through communication and imitation from one generation to the next
It is symbolic communication. Some of its symbols include a group’s skills, knowledge, attitudes, values, and motives. The meanings of the symbols are learned and deliberately perpetuated in a society through its institutions

The definitions made it clear that culture is always a collective phenomenon. Culture defines the social game: the cultural norms define what is accepted or not in the group and also direct it toward a shared purpose.

This is the first hint to manage the culture definition process. Values, direction, beliefs, and behaviors are keywords that translate well to strategic management concepts like vision, mission, and values. And it is also worth talking about principles.

Vision, mission, values, and principles

Any leader came across the challenges of goal setting or was questioned regarding the purpose clarity. In strategic management, the tools to work on these issues are the vision, mission, and values. I particularly like the definitions found in the book Product Roadmaps Relaunched:

The vision is the outcome you seek in the long term. The vision tells us where we want to go
The mission is the intent you have right now. It’s your purpose, it’s your guide toward the vision
Values are beliefs and ideas that govern the behavior of a group of people. If the vision tells us where we want to go, the values help us to correct the course toward the vision

At N26 Brasil, we decided to have a set of principles for the Engineering culture. Principles are ideas or basic rules that explain or control how something must work.

Vision, mission, values, and principles. It seems we have enough elements to start working, right?

Define it collaboratively

If culture is a set of shared beliefs and myths, there is no better way to define it than collaboratively. My suggestion is to hold a workshop with clearly defined activities. Product management techniques and Agile facilitation techniques are useful for this. At least, identify the vision and values of your culture.

At N26 Brasil, we run this process on July 2021 when the Engineering team was only five people. We needed to create a strategy for the area and, for that, we needed to know what kind of culture we would like to nurture. Once the cultural elements were set, the roadmapping process got easier to do. Since then, our technical strategy has been aligned with our culture.

If you’re curious to know how to run a culture workshop, check the playbook I created: it is a step-by-step guide to run a process similar to the one we ran at N26 Brasil.

When Netflix shared its culture slide deck, it went viral. You should do the same. The first step is to define the culture. After it is done, create a document and let it easily findable for everyone at your company. To make it more useful, provide a brief description of each identified element of the culture: they will be useful to validate the behaviors and to provide examples to everyone.

However, a shared document is not the same thing as shared knowledge. Present the culture for every hire. Repeat the cultural elements on every occasion: from the All Hands to Daily meetings, explore them to make them shared and assimilated by everybody.

At N26 Brasil, the culture is presented to everyone during the onboarding sessions. Besides that, the culture document reading is incentivized in the 30-60-90 plan that we set for new hires when they join the Engineering team.

Lastly, integrate the cultural elements in your hiring process and use good examples from the day-to-day work to recognize the people’s contribution to the nurturing of the culture. There is no recipe for success, the building of your culture will depend on continuous care and attention.

N26 Brasil Engineering values

I highlight some of the values of our culture:

Professionalism: we are trustworthy and define high standards of behavior and execution. We care about the work, people, customers, and users
Pragmatism: we are pragmatic. We know when we reach a good enough solution preventing over-engineering and cargo cult programming
Humility: no one knows everything. Every question is a learning opportunity. Seniority is no free pass to disrespectful behavior
Collaboration: the N26 Brasil goals are more important than the individual goals. Teamwork triumphs over individual effort
Diversity: we live in a diverse society and we’ll only be able to provide solutions for it with a diverse team

The Engineering culture at N26 Brasil is still being shaped. The team grew by 400% between September 2021 and January 2022. New features are being delivered and legacy systems are being replaced fastly. Even so, it is possible to see the values defined by that small team being said in the interviews, merged in the OKRs and roadmap, used to prioritize tasks, and in the day-to-day communication between people.

Live the culture and be an example

Any person acting as a leader (be it positional or not) must be an example of the organizational culture. As highlighted by David Anderson, the potential of an organization will always be limited by the maturity of its leadership:

An organization’s maturity is limited by its leadership maturity. Business outcomes and aspirations will always be constrained by culture and values.

A mature leader will always work to create alignment sense, unity, and shared purpose. A mature leader manages the culture and its identity. An immature leader is selfish, narcissistic, and manipulative. Mature leadership nurtures the culture and the identity of the organization.

Culture-building work is harsh and requires consistency and repetition. Setting the values and principles but not living them as a person in a leadership role is the first step to creating cynicism in the team.

The same is true for moments of drastic measures when some cultural elements are challenged: make it clear what is happening and why some trade-offs are required. The consistency in the way you act will pay off at these moments.

Did you stop to think about our culture today?

Acknowledgments

I am grateful to everybody who accepted to join the N26 Brasil Tech team (Data, Design, Engineering, Product). They accepted the challenge of changing how people deal with money. Special thanks to the participants of the culture workshop we held in July 2021: Antonio Spinelli, Diógenes Medeiros, Henrique Sloty, and Thiago Costa.

Three highlights from the 2017 State of DevOps Report

Wed, 05 Jul 2017 07:30:00 -0300

The cover of the 2017 State of DevOps Report.

Last month was published the 2017 State of DevOps Report, an annualy published study by Puppet and DORA (DevOps Research & Assessment). This report is a must read for anyone, whatever its maturity in DevOps adoption.

The numbers are astonishing per se. DevOps works, high-performing teams compared to low-performing teams:

Deploy more and faster
- 46 times more frequent code deployments
- 440 times faster lead time from commit to deploy
Fail less and recover faster
- 96 times faster mean time to recover (MTTR) from downtime
- 5 times lower failure rate
Automate more
- 33% more of their configuration management
- 27% more of their testing
- 30% more of their deployments
- 27% more of their change approval processes
Are 2 times more likely to achieve objectives

And here comes my three highlights.

Maturity model and continuous improvement

My first highlight is not about the numbers or a key finding. It’s about their maturity model. They classify low, medium and high performers using a data-driven approach. The classification is a result of the yearly IT performance assessment, which prevents teams and organizations from falling into a “sucess trap”: after achieving the desired performance level, continuous improvement hangs, as the goal was a static one.

This DevOps maturity model is a moving target, not a static one and it’s truly aligned with Lean’s continuous improvement principle. High-peforming teams and innovative organizations knows that this is the key to excellence.

Architecture

The second highlight goes to architecture. From the report:

In teams with strong IT and organizational performance, the architecture of the system is designed so delivery teams can test, deploy and change their systems without depending on other teams for additional work, resources, or approvals, and with less back-and-forth communication. Therefore, we describe both the architecture and the teams as being loosely coupled.

And goes on:

Architectural approaches that enable this strategy include the use of bounded contexts and APIs as a way to decouple large domains, resulting in smaller, more loosely coupled units. The architecture should also enable the use of test doubles and virtualization to test services or components in isolation. Service-oriented architectures are supposed to enable these outcomes, as should any true microservices architecture.

It is not a surprise that Bounded Contexts from Domain-Driven Design (DDD) is listed as an architectural approach that supports these high-performing teams. For me, DDD should be part of your toolset in your DevOps adoption¹.

It also supports naturally a microservices architecture. Sam Newman states, in his Building Microservices book (he is also a reviewer of this report), that Bounded Context is a way to model microservices with high cohesion and low coupling.

Sam Newman’s coverage of Bounded Context is too shallow. I recommend reading one of the Vaughn Vernon’s books for the right treatment on the subject.

Leadership

The characteristics of transformational leadership are highly correlated with IT performance. In fact, we observed significant differences in leadership characteristics between high-, medium- and low-performing IT teams. High-performing teams reported having leaders with the strongest behaviors across all dimensions: vision, inspirational communication, intellectual stimulation, supportive leadership, and personal recognition. In contrast, low-performing teams reported the lowest levels of these leadership characteristics. The differences we found were all at statistically significant levels.

You can’t underestimate how a bad leadership can undermine the performance of a team. I have the principle that everybody wants to do a good job, but managing people is hard. And software development is a techno-human effort, and without the human part, you don’t get the technological one.

What I hear from people are the same horror stories: managers who just use Agile as a moniker for a perpetual rush to throw code to production, disregarding quality, without slack to enable continuous improvement and a complete alienation of the development teams, which are just code monkeys. Unhappy people, crap code and slow delivery cycles.

The last four dimensions can be mapped directly to Daniel Pink’s three elements of motivation. If you aren’t prepared to work on people’s motivation or you don’t mind about the subject, make a favor to everyone: quit your job.

Closing remarks

The report is a quick, simple and insightful reading. DevOps, which started as a cultural movement, is now something bigger, highly publicized and moving on the hype cycle’s path. The report reminds us to focus on the results and emphasizes the technical practices.

As a side note, I really liked the graphic design of the report, which shows an interesting diversity on genre, color and ethnicity. It seems intentional, and it’s great! The infographic is worth the download too.

References

Brown et al, 2017. The 2017 State of DevOps Report (infographic)
Pink, 2009. Drive: The Surprising Truth About What Motivates Us
Vernon, 2016. Domain-Driven Design Distilled (my review)
Wikipedia. Hype cycle

I could coin the DDDevOps term, but for the greater good, I won’t. ↩

Good design is informed and careful

Tue, 20 Jun 2017 07:30:00 -0300

In the last couple weeks, I had the chance to talk with lot of people and teams from different companies. And some common themes appeared, like motivation issues, code quality, odd architecture choices and slow delivery cycles.

I reflected about these issues and a common element seemed obvious between them: the lack of design, or better, bad design. To discuss what is good or bad design, a definition is welcomed:

Good design is design that is informed by specific information, what kind of climate do you have, what kind of building code do you have, what kind of social behavior do you have and which you would like to promote. So that everything you do is actually done for a specific purpose. Because I think good design is careful, bad design is careless (Ingels, 2015)

As you noticed, Bjarke Ingels does not work with software: he talks about the climate, burocracy and social behavior. Ingles is an architect, but he has a good point: good design is informed and careful.

A partial transformation

In my Domain-Driven Design introduction talk, I start discussing how design was put aside even with the Agile transformation that happened in the last 16 years. Sandro Mancuso discuss in his The Software Crafsman book about how this transformation was partial:

The Agile transformations focused mainly on process; empowering people; reducing bureaucracy and waste; prioritization; visibility of work in progress; and improving the information flow. These were real and important problems that needed to be solved (…). However, the principles behind Agile (Manifesto) were forgotten. Process became more important than technical excellence. Technical excellence, which is assumed to be in place by every process-related Agile methodology, is normally ignored by managers and ill-prepared Agile coaches. (Mancuso, 2014)

This partial transformation unfold in the problems I introduced above and in the proverbial “we tried, but it did not worked” and “this won’t work here”. There is no way for methodologies or processes that depends on feedback cycles succeed without technical excellence and without improving the skills from the team’s professionals. Then we come to the following (sad) fact:

Many Agile projects are now, steadily and iteratively, producing crap code. (Mancuso, 2014)

Design is everywhere

The design that manifests in code is the result of the intersection between the organization’s design and the team’s design. The design question is by no way an issue of a single team member.

It is worth noting that most of the time, when I talk about software development, I’m talking about a human-technological endeavor that is done by a team. The majority of softwares are developed by teams.

Behind the Agile methodologies, behind the Lean/Kanban and behind the DevOps, we always find principles or objectives for the creation of highly functional teams. From the Agile Manifesto (Beck et al, 2001), we have the principles 8 and 11, for example:

Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely
The best architectures, requirements, and designs emerge from self-organizing teams

From Lean (Liker, 2003), we have the principles 9 and 10:

Grow leaders who thoroughly understand the work, live the philosophy, and teach it to others
Develop exceptional people and teams who follow your company’s philosophy

And from DevOps (Debois et al, 2016), from The Principles of Continual Learning and Experimentation, we have:

Enabling organizational learning and a safety culture
Institutionalize the improvement of daily work
Leaders reinforce a learning culture

So, the professional software development activity happens in teams, in an organization. And the design of the organization and of the team influences in the code design. Dr. Melvin Conway has some words about this phenomenon, that became so famous that were promoted to what is known today as Conway’s law.

Conway’s law

Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization’s communication structure. (Conway, 1968)

Conway’s law is a sociological observation that can be observed in different kinds of team’s organization. The famous Amazon’s two pizza team (2PT)¹, for example, limits the teams’ size to a maximum of 5-10 people. This constraint has important effects (Debois et al, 2016):

The team has a clear understanding of the system they are working on
It limits the growth rate of the product or service being worked on
It decentralizes power and enables autonomy
Promotes a safety culture, the smaller work scope limits the failures’ consequences

In the same way, it seems that the Spotify’s Squads (Kniberg, 2012) contributes to achieve the same effects.

The difference is that Amazon is a market-oriented organization, optimized to respond quickly to the clients needs and formed by multidisciplinary teams in a flat hierarchy. Spotify is a matrix-oriented organization, which combines elements from functional-oriented organizations (optimized for expertise, labor division and cost reduction) and from market-oriented organizations.

The important thing in these differences is the fact they disclose a concious organization design. If good design is informed, adopting a team organization like Amazon’s 2PT or Spotify’s Squads requires understanding not only the teams’ structure as well as their interdependencies and the impacts of these new dynamics in the organization.

This design, made in a careful way, dictates how the work is performed and the achieved outcomes (Debois et al, 2016).

Teams, individuals and code

The teams’ design is a direct outcome of the organization’s design. Market-oriented teams are, in an extreme, totally responsible not only in software development, but also for testing, securing, deploying and supporting their service in production. They’re DevOps teams (Debois et al, 2016).

The classic Scrum team is also cross-functional, formed by a Product Owner, by the Scrum Master and by the Development Team. However, the goal for this team is to deliver a potentially releasable product increment (Schwaber, 2013). This team’s responsibility is limited to the software development.

The teams’ structure will define their interactions and potential conflicts. Usually the developers defines how they’ll work. However, they seem to not realize their influence in the organization and for the whole society: the world depends more and more on software.

Developers are, generally, highly motivated and curious individuals. But this same curiosity seems to result in an extremelly technology-oriented mindset. It is not uncommon to see teams trying to solve problems with technology instead of doing a careful design. Then, the software model turns into a Big Ball of Mud (Foote; Yoder, 1999), with almost no business’ knowledge coded in an expressive way.

That’s where us, as individuals, have the duty to act as professionals. Being professional requires discipline. The organizations still don’t understand us. Uncle Bob (Martin, 2016) has some good words about how to be a professional:

In particular, the business does not understand our disciplines (…), business doesn’t understand Pair Programming, doesn’t understand Test-Driven Development, doesn’t understand Refactoring, doesn’t understand Simple Design. These technical disciplines are not within the expertise of business.

And they shouldn’t be. They belong to us, its part of our tech expertise. So business can not approve or endorse them. You can’t go to the business and say “Is it ok if we write tests?”. (…) Because when you do that, what you are trying to do is to shed the risk, you are not willing to take the risk, you put it on the business, and the business can not evaluate that and take the risk. (…)

And the risk, frankly, is ours to take. We own that risk. That’s us. We are the ones who know the things that need to be tested, we are the ones who know the things that need to be refactored, we are the ones who know how to get software done and so we have to take the risk as part of our normal professional operation. As professionals. And that’s what professionals does: a professional take the risk on what they know what must be done.

Another professional responsibility we have is to understand the business domain in which we work:

(…) if programmers are not interested in the domain, they learn only what the application should do, not the principles behind it. Useful software can be built that way, but the project will never arrive at a point where powerful new features unfold as corollaries to older features. (Evans, 2003)

Uncle Bob and Eric Evans exposes deep economical implications. Our professionalism can be the difference between the business realizing the return on investment made in software development or having a software that’s just cost.

We can’t run away from our responsibilities: we have the duty, as professionals, as individuals, to never stop learning and to improve in our craft.

Informed and careful design

A software development approach that supports informed and careful design is Domain-Driven Design (DDD). DDD helps to implement the best software design from models that explicitly reflects the organization’s competencies. DDD helps to develop software that gives competitive advantage to the business because it forces the organization to understand where it must excel (Vernon, 2016).

DDD has strategic and tactical design tools to support the design activities in the organization/team intersection. The strategic design is about dividing the work that must be done and about finding what is strategically important to the business. In short, DDD is primarily about modeling a Ubiquitous Language in a Bounded Context (Vernon, 2016).

These tools are DDD’s cornerstones. Teams that adopts DDD speaks in a Ubiquitous Language that is spoken both by developers and by domain experts. This language is implemented the software model (that’s why it’s ubiquitous). The Contexts are used to constrain and clarify the vocabulary of this Language. They’re semantic boundaries. Inside a Context, each term has a specific meaning, which is understood by the team members.

The Contexts are the acknowledgement that the business’ domain is not only big and complex, but best managed by dividing it. The recommendation to assign a Context to a team, with a separate source code repository and database (Vernon, 2016) aligns perfectly with product’s teams organization. It is also a natural way to implement a Microservices architecture (Newman, 2014).

The resulting model from the strategic design is informed: the team acquired the needed information from the business to divide it in sub-domains, finding their Contexts and modelling the Ubiquitous Language.

The tactical design is a pattern catalog that helps to implement the model in code, with the goal to make the software an explicit reflection of the model. If my code is an explicit representation of the model that was created by the collective effort from the developers and domain experts, this code documents and centralizes the business’ knowledge. It’s careful design.

Continuous improvement as a design practice

Organizations and teams inter-relations are complex systems. Software development is a hard activity. Given these challenges, we create strategies to execute tjese design activities (organizational, team and code) to transform the organization.

However, a organizational transformation depends on people. They learn and change the organization (Soares, 2017). In software organizations, it is pretty common see people burned out with too much transformational activities, with multiple process being adopted in parallel.

This not only raise the risk of failure in the adoption and understading of the processes, this overburden causes morale issues in the team. With each new failure, the team desmotivates. Even with its best intentions and efforts, the new process failed. A thick fog of indifference and ressentiment appears.

Aiming to have fully DevOps teams is a nice starting point. DevOps, as a set of practices, has flow visualization amongst its principles. The Lean mindset and the Kanban Method are integrated into DevOps (Debois et al, 2016) and as mentioned previously, the creation of a continuous improvement culture.

The Kanban Method is designed to minimize the initial impact of changes and reduce resistance to adopting change. Adopting Kanban should change the culture of your organization and help it mature. If the adoption is done correcly, the organization will morpth into one that adopts change readily and becomes good at implementing changes and process improvements. (Anderson, 2010)

The strategy doesn’t matter, it is primordial to minimize the initial impact and resistance to changes to adopt a continuous improvement culture.

Design is innevitable

Questions about whether design is necessary or affordable are quite beside the point: design is inevitable. The alternative to good design is bad design, not no design at all. (Vernon, 2016)

Design being inevitable, we have the duty to design the organization, the teams and the code. There is no alternative, we need to do design. Better if informed and careful. What are we waiting to start?

References

Ingels, 2015. Bjarke Ingels: Good design is careful, bad design is careless (video)
Mancuso, 2014. The Software Craftsman: Professionalism, Pragmatism, Pride
Beck et al, 2001. Princípios por trás do Manifesto Ágil
Liker, 2003. The Toyota Way: 14 Management Principles from the World’s Greatest Manufacturer
Debois et al, 2016. The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations
Conway, 1968. How Do Committees Invent?
Kniberg; Ivarsson, 2012. Scaling Agile @ Spotify
Schwaber; Sutherland, 2013. The Scrum Guide
Martin, 2016. The Future of Programming (video)
Foote; Yoder, 1999. Big Ball of Mud
Evans, 2003. Domain-Driven Design: Tackling Complexity in the Heart of Software
Vernon, 2016. Domain-Driven Design Distilled (my review)
Newman, 2015. Building Microservices: Designing Fine-Grained Systems
Soares, 2017. Barriers to organizational learning
Anderson, 2010. Kanban: Successful Evolutionary Change for Your Technology Business

The idea is that the team can be fed with two pizzas. I believe, for now, that this is not an Amazon’s plan to raise the demand for pizzas by increasing the appetite of unsuspecting readers. ↩

TensorFlow is commodity software

Mon, 16 Nov 2015 09:00:00 -0200

As you already know, Alphabet’s Google open sourced TensorFlow, a machine learning library. A lot of news articles were written about this. Some of them got it all wrong, depicting it as a marvelous black box that will miraculously provide your organization with brilliant Artificial Intelligence capabilities, out of thin air.

Others were straight to the point: Google open sourced TensorFlow because it is not what gives them competitive advantage and also because they will benefit from this. TensorFlow is commodity software. I will step back to explain what TensorFlow is and why I see it as the commodity part of Google’s operation.

TensorFlow is part framework and part a distributed computing engine. The framework provides an API to express machine learning algorithms using directed graphs as their representation. The TensorFlow paper highlights that its development was driven by the findings they had with DistBelief (their previous proprietary machine learning library) and that Artificial Neural Networks were one of the focus of the research (Google Research, 2015, pages 1-2).

Artificial Neural Networks (ANNs) are, as the name implies, based on biological neural networks¹. ANNs are formed by layers of interconnected nodes. The connections between these nodes are activated when an input value is strong enough to activate a node. This interaction repeats until a node in the output layer is activated, which is the result that the network thinks it is right for the given input.

For example, a neural network to recognize handwritten digits scanned in 28x28 pixels images is shown in the following image:

A three-layer neural network that recognizes handwritten digits (Nielsen, Michael A., 2015)

The input layer have a node for each pixel of the 28x28 pixel image. Each node is given a value ranging from 0.0 (white) to 1.0 (black) and the range represents the shades of gray. When a node in the input layer is given a value, it tries to activate the hidden layer and so on until a node in the output layer is activated. The activated node in the output layer indicates the digit that the neural network thinks it is.

This capability to infer a digit is possible by training the neural network. By feeding the neural network with a training data set, it is possible to query it later with an image that it was not present in the data set. The neural network then infers the result based on past experience.

Besides the existence of different neural networks models, training them is also hard. The model type and training type and rate matters. You can end with a neural network that overgeneralize the inferences or that don’t learn anything. Building them requires competencies not just in the algorithms as well in math and statistics.

Also important is to know what you want to achieve by building a machine learning model. What kind of capability or enhancement you want to provide for your product or service. Leveraging machine learning to enhance a product or service ressembles domain modeling. It is context-sensitive, specific to a business. It requires you to formulate your hypotheses, collect data, formulate the models and learn from the results². Tweak, rinse and repeat. And for sure, you need data.

And that’s why Google open sourced it. They turned TensorFlow a commodity because it has a supporting role in its activities. As pointed by, Rob Landley, commoditization happens when markets mature:

As any market matures, it commoditizes. Once-proprietary advances become generally understood, are improved upon, and become obsolete. The rapid pace of information technology has accelerated this cycle as well.

This happened before in the IT industry: from the commoditization of computer hardware (IBM PC), operating systems (Unix, Windows, Linux), SQL and NoSQL databases (MySQL/PostgreSQL, MongoDB/Redis), messaging brokers (RabbitMQ, ActiveMQ) to Big Data distributed storage and computing systems (Hadoop, Spark).

In the previous commoditization cycle, the Big Data cycle, Hadoop emerged as its flagship software. Inspired by Google’s File System and MapReduce papers, Hadoop reigned for some years and was pointed by an ex-Google engineer as more sophisticated than the company’s proprietary software³.

Maybe Google did not took advantages from the sustaining innovations introduced in Hadoop but their proprietary software enabled them to build a gigantic infrastructure based on commodity hardware and to scale its systems in an unprecedented way.

And here is where the commoditization of TensorFlow becomes more evident: Google already owns a monopoly on data and has a massive infrastructure. They crunch massive amounts of data in their machine learning models in a scale that no other company does. This poses a great barrier for competitors. As Steve Blank (2003) points out “attacking a competitor with a monopoly in an existing market requires three times the competitor’s spending”. To create services to compete against Google’s core services is then a huge endeavor that only some companies can tackle.

In this scenario, open sourcing TensorFlow⁴ is hugely beneficial for Google. It will bring a stream of sustaining innovations through contributions from different parties, leveraging the Artificial Intelligence community knowledge⁵. For the community and the market, it will raise interest in the subject, not to forget that it “borne from real-world experience in conducting research and deploying more than one hundred machine learning projects throughout a wide range of Google products and services” (Google Research, 2015, page 14). Indeed, it was a huge development effort and it is a great contribution to the ecosystem.

To wrap up, I quote Rob Landley again:

Commoditization does not mean that innovation ceases in that niche. On the contrary, innovation is greatly helped by the modular nature of a commodity marketplace: it’s easier to upgrade to a better mousetrap when your mousetrap didn’t come bundled with the house. Commoditization merely means that consumers have a general idea what they want and multiple suppliers know how to give it to them. A mature market has a usable baseline and a frame of reference. If it was static and boring, the commodities market wouldn’t be so active.

References

Nielsen, Michael A. 2015. Neural Networks and Deep Learning. The book is licensed under a Creative Commons Attribution-NonCommercial 3.0 Unported
Google Research, 2015. TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems (preliminary whitepaper)
Blank, Steve. 2003. The Four Steps to the Epiphany (2nd edition)

I’m not by any means specialist in the Artificial Intelligence subject neither in neural networks. I implemented some machine learning algorithms, specifically clustering (to detect, for example, duplicated contacts in a SaaS CRM) and classifiers (for credit analysis in a Bank’s customer portfolio). I took the care to provide references where anyone can look for authoritative experience. ↩
Yes, I almost described the Build-Measure-Learn loop from the Lean Startup model. It is worth approaching technology adoption in this way: define your experiment, set your expected outcomes, build and the check the results. Start small. Technology adoption alone is meaningless. ↩
The claims can be seen as an ex-employee whining but I like to highlight that Hadoop had massive investments for its development, initially from Yahoo! and then from the various companies that provides services around it (HortonWorks, Cloudera, AWS, IBM) and, from some big users (Facebook, Twitter, LinkedIn). So, it is feasible that the diversity in contributions coming from multiple vendors and users can result in software of superior quality. ↩
The distributed computing part of TensorFlow is not available in the open source release yet. I doubt this is because they want to secure competitive advantage. I believe it is just a matter of time for them to decouple it from Google’s specific code. ↩
There are numerous machine learning frameworks out there like Caffe, Torch and the also recently open sourced DMTK (from Microsoft). This is also a proof of this commoditization trend. ↩

Mapping your domain knowledge

Wed, 04 Nov 2015 16:30:00 -0200

We build software to help people - the users and customers - reach their goals. This is the only way your organization can get what it wants: revenue for for-profit organization or a measurable impact for non-profit organizations.

To build quality in software, we employ a variety of practices as Test-Driven Development and Continuous Integration. These practices are great to limit the number of defects in the code but them solely don’t guarantee that the built software reflects the domain concepts of the business neither that it helps people reach their goals. It is too easy to build the wrong software.

That’s why the Agile methodologies includes practices that improves communication and collaboration. As example, Scrum has events like the Daily Scrum, the Sprint Planning and the Sprint Retrospective. Extreme Programming has Whole Team and System Metaphor. Both methodologies uses Iterative Development, not just to deliver working software frequently, but also to prove the value hypothesis of the delivered software. These practices are based on the Agile Manifesto principles to reduce the risk of building the wrong software.

Delivering working software that makes people happy does not mean that the team is building shared understanding neither that the software design have the concepts of the domain where it operates. And here comes a critic to my fellow developers, one that I hear often: in the urge to develop a requested feature, we skip conversations that helps building shared understanding of the business in favor to technical discussions.

This critic is beneficial because it shows that the business people are interested in building this shared understanding, that they are incorporating the Agile Manifesto principles. But we are jumping straight to implementation, losing an opportunity to learn the domain and to model the software with the concepts of the domain. Eric Evans (2003) goes further:

(…) if programmers are not interested in the domain, they learn only what the application should do, not the principles behind it. Useful software can be built that way, but the project will never arrive at a point where powerful new features unfold as corollaries to older features. (Evans, 2003)

How to motivate developers to be more aware of the business domain? I think User Story Mapping can come to the rescue.

An early User Story Map from Wikidu (wikidu.com)

User Story Mapping

User Story Mapping is a pattern discovered by Jeff Patton that help teams understand a whole product or feature, with the goal of building a better product:

Story mapping keeps us focused on users and their experience, and the result is a better conversation, and ultimately a better product. (Patton, 2014)

The outstanding documentation that Patton created shines a light in how stories¹ are being misused, making teams to fall in one or more traps. I highlight two of these traps:

The flat backlog: stories let teams focus on building small things, losing sight of the big picture. It often results in Frankenstein-like products, where the products seems assembled from mismatched parts
Form over substance: when the focus from writing stories is only to get them written instead of building shared understanding

I’ve included a crash course to better explain how it can help motivate everyone to build shared understanding. You’ll need sticky notes, pens, a tabletop and a wall (or a whiteboard).

The backlog column don’t show the big picture of this product

Crash course

Set a timebox (10 minutes at maximum) and ask people to start writing in silence the tasks that the users will make in your application. These users tasks are things like “Request a ride”, “Rate the book”, “Add product to cart”. Once everyone finished, ask people to read each sticky note and place it in the tabletop. If duplicates are found, remove them. In summary, the steps are: think, write, explain and position.

You’ll find some interesting things during the process:

Most sticky notes will contain short verb phrases
People will organize the tasks intuitively in a flow order, with tasks that must happen first in the left side with later tasks following them
Some sticky notes will contain tasks that happens around the same time as other tasks. These are subtasks. In a software application these can be additional and/or optional steps the user can take

Now that you identified the user tasks, it is time to organize your map. Intuitively some of this work was already done: tasks that must happen first are placed in the left side, with later tasks following them. Stack the subtasks below their main tasks.

To validate if this flow is consistent, start telling the story of your map: point to the first user task saying “First the user does this”, and then pointing to the next saying “and then it does that” until the last task.

This is what Patton defines as the “narrative flow” or the “storytelling order”. Reorder the tasks placement if the narrative seems awkward. You’ll also find missing details because the map helps you to see the big picture. Fill the missing details as you go.

Prove the consistency of your map telling its story

The next step is to distill the map. Step back and pay attention to the narrative flow. You’ll find that there are clusters of tasks that goes together. The ecommerce user tasks, “Add to cart”, “Estimate freight” and “Checkout cart” could be grouped in an “Place an order” activity. For a social network, “Search by name”, “Visit the person profile” and “Add as a connection” could be grouped in an “Connect” or “Friend” activity.

Write the activity name in a sticky note and place it above the cluster of tasks that it defines. Validate if the narrative flow is still consistent, reading the activities in the same manner as you had read the user tasks. The activities forms the backbone of the map.

The backbone of the map is formed by the activities

Adding context to the map

A map can have more than user tasks and activities. You can add any meaningful information to it, helping to build shared understanding:

Opportunities and hypothesis. Why build it? What kind of problem it solves?
Product principles. Examples: fun, easy, accessible (hey, this should be a principle of any product, period), friendly, fast
Customers and users. Who are my users? What are the different users roles? Why they will use the product? What do we think about them? What types we will prioritize now?
Details, options and questions. More details about the tasks and activities can be provided as well different options. If someone is not sure how something will work, it can place sticky notes with doubts. Notes about how other services - competing or not - solved some issue can also be placed in the map.

You can use sticky notes of different colors and place this information where it is meaningful to give more context.

Enrich your conversation adding more context into the map

Building shared understanding is the key

What is really important during the creation of the map is the shared understanding built by better conversations. This is the original idea of stories when introduced by Kent Beck: to turn the focus from shared documents towards shared understanding.

Just writing stories as shared documents don’t help to build shared understanding because when someone reads a story, it can interpret it in a different way than the story’s author. This is only possible when we externalize our ideas. The written story then becomes a mean, not an end.

How can this motivate technology-driven developers?

As a developer, I know a lot of things that can make developers less concerned about the business. One of them is the absence of product vision, the lack of the big picture. Roadmaps are constantly created to provide this big picture but usually they are not visual enough or updated often to radiate clearly where in the journey the team is. And the flat backlog can give an impression that the only work the developers must do is to process new stories.

The developers become alienated from the business goals and thus lose interest in the domain. And as developers like to build things, it is easy to turn the attention to the technology side and solve domain problems with technology (Evans, 2003).

As a visual tool, a map can clearly radiate information about the big picture. It can be used to prioritize the stories to be worked out (Patton explains prioritization techniques in his book). It is a living roadmap. The developers can understand better what and why they are working and how that fits in the big picture.

Asking developers to help create the map (I think the first map should be created with the contribution of the whole team) can provide not just different points of view but also interesting and valuable ideas (developers tend to be early adopters of all kind of technologies, from mobile apps to wearables). And then they start to ask why/what/how: now they’re not just helping to build shared understanding, they are willing to learn about the business.

Creating the map can also build stronger relations between team members and develop values like respect and courage. If the team goes further to try to understand the users/customers better and create personas², it can develop empathy by the users in the developers. Stronger relations between the team, more awareness about the business goals and more empathy for the users can unlock one of the three elements of motivation: purpose (Pink, 2009). Purpose is the yearning to do what we do in the service of something larger than ourselves. People who find purpose in their work unlock the highest level of the motivation game.

Map to help with your DDD adoption and to produce an Ubiquitous Language

Evans’ Domain-Driven Design (DDD) book introduced a new approach to software development that places the focus in implementing a software design based on a model of the domain. It can help reaching a point where the design is exactly how the software works (Vernon, 2013) by giving strategic and tactical modelling tools necessary to design high-quality software that meets core business objectives .

As an approach, DDD is not about technology:

DDD is about discussion, listening, understanding, discovery, and business value, all in effort to centralize knowledge. If you are capable of understanding the business in which your company works, you can at a minimum participate in the software model discovery process to produce a Ubiquitous Language (Vernon, 2013)

To build a domain model and effectively adopt DDD, it is needed to crunch knowledge. “To centralize knowledge” is a synonym of “to build shared understanding”. Evans says that effective domain modelers are knowledge crunchers and emphatize that this is a team activity (and a requirement to practice DDD):

Knowledge crunching is not a solitary activity. A team of developers and domain experts collaborate, typically led by developers. Together they draw in information and crunch it in useful form. The raw material comers from the minds of domain experts, from users of existing systems, from the prior experience of the technical team with a related legacy system or another project in the same domain. It comes in the form of documents written for the project or used in the business, and lots and lots of talk. Early versions or prototypes feed experience back into the team and change interpretations. (Evans, 2003)

From this knowledge crunching, the team (developers and domain experts) must develop an Ubiquitous Language, which is a shared language. The vocabulary of this language is centered in how the business itself thinks and operates (Vernon, 2013) . This shared language prevents misunderstanding between the business people and developers by removing the need of translations between business concepts and the abstractions used in the software design.

The Ubiquitous Language is one of the pillars of DDD alongside Bounded Context (Vernon, 2013) . Its vocabulary includes names of classes and prominent operations and is the the primary carrier of the aspects of design that don’t appear in code (Evans, 2003).

That’s where Story Mapping can help your DDD adoption. It is a great technique to start the knowledge crunching process and to produce the initial glossary of the Ubiquitous Language. Make it a collaborative effort: it will help to form your team and will push them to have the required conversations.

Map to assess your domain complexity

In Implementing Domain-Driven Design, Vernon presents a scorecard³ to help teams to determine if it is worth to adopt DDD in a project. If you get a score of seven or higher, consider the adoption of DDD.

If your project	Points	Supporting thoughts
If your application is completely data centric and truly qualifies for a pure CRUD solution, where every operation is basically a simple database query to Create, Read, Update, or Delete, you don’t need DDD. Your team just needs to put a pretty face on a database table editor. In other words, if you could trust your users to just insert data directly into a table, update it, and sometimes delete it, you wouldn’t even need a user interface. That’s not realistic, but conceptually relevant. If you could even use a simple database development tool to create a solution, don’t waste your company’s time and money on DDD.	0	This seems like a no-brainer, but it’s not usually that easy to determine simple versus complex. It’s not like every application that isn’t pure CRUD deserves the time and effort of using DDD. So maybe we could come up with other metrics to help us draw a line between what is complex and what is not…
If your system requires just 30 or less business operations, it’s probably pretty simple. This would mean that your application would have no more than 30 total user stories or use case flows, with each of those flows having only minimal business logic. If you could quickly and easily develop such an application using Ruby on Rails or Groovy and Grails and not feel the pain of lacking power and control over complexity and change, your system probably doesn’t need to use DDD.	1	To be clear, I am talking 25-30 single business methods, not 25-30 whole service interfaces, each with multiple methods. The latter might be complex.
So let’s say that somewhere in the range of 30 to 40 user stories or use case flows could be creeping toward complexity. Your system might be getting into DDD territory.	2	Caveat Emptor: Very often complexity is not recognized soon enough. We software developers are really, really good at underestimating complexity and level of effort. Just because we might want to code up a Rails or Grails application doesn’t mean we should. In the long run those could hurt more than help.
Even if the application is not going to be complex now, will it grow in complexity? You may not know this for sure until real users start working with it, but there is a step under Supporting thoughts that may help uncover the true situation. Be careful here. If there is any hint at all that the application has even moderate complexity—here’s a good time to be paranoid—it may be sufficient indication that it will actually be more than moderately complex. Lean toward DDD.	3	Here it pays off to walk through the more complex usage scenarios with domain experts and see where it leads. Are domain experts… (1) already asking for more complex features? If so, it’s likely an indication that the application is already or will soon become too complex to use a CRUD approach. (2) so bored with the features that they can hardly bear discussing them? It’s probably not complex.
The application’s features are going to change often over a number of years, and you can’t anticipate that the kinds of changes will be simple.	4	DDD can help you manage the complexity of refactoring your model over time.
You don’t understand the Domain because it’s new. As far as you and your team know, nobody has done this before. That most likely means it’s complex, or at least deserves due diligence with analytical scrutiny to determine the level of complexity.	5	You are going to need to work with domain experts and experiment with models to get it right. You certainly also scored on one or more of the previous, so use DDD.

After finishing your map, you’ll have the big picture of the product you want to build. Do you remember the user tasks? They are the stories your team will need to develop. Each can have n subtasks or alternative flows. Count the user tasks and you’ll know if you scored in the second or third case. Another complexity indicators are the visual density of the map and the quantity of user roles (and the way they influence the user tasks).

Obviously the map alone can not be enough to assess the complexity of your project. For example, many startups explores business opportunities that fall in the last case. And when a startup depends of a software application for its business, you might expect that its features will change over time to better support the growth and continuity of it. Use the map as one of the raw materials for your assessment.

Closing remarks

User Story Mapping is a simple yet insightful pattern that rescues the original idea of Beck’s stories: to have good conversation for the good of shared understanding. As a collaborative task, it helps in team formation and motivation.

Some Agile practitioners may fear Story Mapping as a big up front plan. It is not meant to be. Build the first map to discover the narrative and then use it to prioritize the stories to be developed. As they approach to be prioritized for development, start the knowledge crunching to identify what you need to learn.

Keep it as a living document and update it every time you discover something new. Update the narrative flow while you validate your value hypothesis. Use the map as your backlog, without losing the big picture!

Curiously, some developers used to Agile software development fear that DDD can lead to a big up front design. My personal belief is that this impression is caused by the emphasis on conversations and on diagrams in the literature. You need to discuss because you need to learn the business principles behind a story. The diagrams are just abstractions, they don’t necessarily reflect the future design: a lot will be left to be discovered and that’s where TDD shines.

Last but not least, Story Mapping can be a valuable input in your assessment for DDD’s adoption. If your project is complex and you adopt the DDD approach to software development, the map can help you start the knowledge crunching process and in the production of the initial Ubiquitous Language glossary.

Acknowledgements

Thanks to Nelson Senna and Fernando Ike for the proofreading and to Vaughn Vernon for the positive feedback.

References

I use the noun story instead of user story for convenience. Stories are widely used in Agile methodologies for planning purposes. Introduced by Kent Beck, stories encourage informal conversations for the building of shared understanding. ↩
A persona is a written representation of a specific user which can be used to make decisions about software features. Patton describes a pragmatic way to create user personas. ↩
The full explanation of the DDD scorecard is also available at the InformIT website (content from the rough cut of the book). ↩

Don't try Composer, adopt it

Mon, 22 Dec 2014 07:30:00 -0200

The new edition of the ThoughtWorks Technology Radar listed for the first time a PHP-related project and it was Composer. However, they listed Composer on the Trial quadrant, while I believe it should be listed on the Adopt quadrant. And I want to tell you why.

First, it is important to state that the Technology Radar is not a list of ThoughtWorks’ approved technologies. It is a compilation of technologies made solely based on opinions and experience by senior technologists. It is a very insightful publication, and I recommend you to read.

When I think “ThoughtWorks”, PHP does not come to mind. I don’t remember any publishing by them mentioning the language. With this in mind, I was surprised by the Composer being listed. And maybe the absence of intimacy with the PHP ecosystem is the reason why they listed Composer in the Trial quadrant.

Composer was introduced in late 2011. Back then, installing dependencies in PHP projects were limited by two choices: resorting to version control system features such as SVN externals or Git submodules or, if available, through the PEAR installer.

Yet, Symfony2 was released July that year, pushing the PHP ecossystem. A wave of new libraries and tools were released, renewing the interest in the language. Composer, released months later, was the tool we were needing to easily benefit from these new libraries and tools.

In a few weeks, 227 packages were listed on Packagist (the main Composer package repository) while PEAR had 592 packages. What was astonishing was the fact these packages were for PHP 5.3+ while PEAR had only 192 for PHP 5+. The impact maybe was faster than Jordi Boggiano (one of the lead Composer developers) predicted.

Today, 45 thousand packages are registered on Packagist. Rubygems, which dates back to 2003, have 93 thousand gems. Security breaches have been discussed, fixed and alternatives have been proposed. Additional tools were released by community developers to help guarantee better security. The good documentation provides a safe starting point and there is always someone helping you to use it in the right way.

Last but not least, thousand of organizations are using Composer to manage dependencies. Millions of runs in CI environments happened and another millions of millions in developers’ environments. How not recommending a tool that is the de-facto solution of the ecosystem toolset?

Don’t make people laugh at you in the pub by telling them you don’t use Composer.

A Docker image for multi-version PHP development

Tue, 16 Dec 2014 11:30:00 -0200

I released a Docker image suitable for PHP development with the latest supported PHP versions installled plus latest 5.3 release. The image is aimed to help developers which need to develop and test their projects in different PHP versions (specially free or open source software developers).

Docker is a tool built on top of Linux’s operating system-level virtualization features. It allows applications to be build and shipped as containers, which can mean that you can hook it in your deployment pipeline to generate a Docker container as your shippable binary (if adopting Docker for both development and production) or just use it for development purposes as a replacement to Vagrant, for example. I won’t extend on this, read the Docker user guide and this paragraph will be a lot clearer, believe me!

Using the Docker image

Now that you’re used to Docker and installed it, using the PHP Development image will be straightforward. At the time of writing, it comes with the following PHP versions:

5.6.3
5.5.19
5.4.35
5.3.29

Each PHP version comes with the Zend OPCache and Xdebug extensions installed and some PHP QA tools are installed globally with Composer. The image also ships with Nginx and PHP-FPM. Check the image’s page for a full list of the extensions and libraries installed.

Let’s start a new container. First, pull the image from the Docker Hub:

$ docker pull eriksencosta/php-dev

Now, let’s create a very simple “Hello World” application:

$ mkdir ~/hello-world; echo '<?php echo "Hello world!";' > ~/hello-world/index.php

Start up the container:

$ docker run -p 80:80 -v ~/hello-world:/var/www/hello-world -d "eriksencosta/php-dev:latest"

Open http://localhost/hello-world/ (if you’re using Boot2Docker, replace localhost with the IP address returned from the command boot2docker ip) in your browser and be delighted by your awesome application greeting the whole world. Better than the UN, they say.

If you go to http://localhost/info.php, you’ll see the full information from the running PHP version.

Switching between the PHP versions

To get the real value of the image, you need to switch between the PHP versions. Stop the current running container:

$ docker stop $(docker ps -l | tail -1 | awk '{print $1}')

And start a new container in interactive mode:

$ docker run -t -i -p 80:80 -v ~/hello-world:/var/www/hello-world "eriksencosta/php-dev:latest" /bin/bash

You’ll be assigned with a terminal inside the container. To switch between the different PHP versions, use the phpenv command:

# phpenv versions
  5.3
  5.3.29
  5.4
  5.4.35
  5.5
  5.5.19
  5.6
* 5.6.3 (set by /opt/phpenv/version)

5.6, 5.5, 5.4 and 5.3 are just shortcuts. phpenv let you define the global (system-wide) and local PHP version. To set it globally:

# phpenv global 5.4
# php -v
PHP 5.4.35 (cli) (built: Dec 14 2014 00:35:12)
Copyright (c) 1997-2014 The PHP Group
Zend Engine v2.4.0, Copyright (c) 1998-2014 Zend Technologies
    with Zend OPcache v7.0.3, Copyright (c) 1999-2014, by Zend Technologies
    with Xdebug v2.2.6, Copyright (c) 2002-2014, by Derick Rethans

To set a local version, run:

# cd /var/www/hello-world
# phpenv local 5.3
# php -v
PHP 5.3.29 (cli) (built: Dec 14 2014 00:24:19)
Copyright (c) 1997-2014 The PHP Group
Zend Engine v2.3.0, Copyright (c) 1998-2014 Zend Technologies
    with Zend OPcache v7.0.3, Copyright (c) 1999-2014, by Zend Technologies
    with Xdebug v2.2.6, Copyright (c) 2002-2014, by Derick Rethans

To start Nginx and PHP-FPM, run:

# webserver start
Starting PHP-FPM (PHP version 5.3) server.
Starting Nginx server.
Done.

To disable the Zend OPCache (which is enabled by default):

# opcache disable
The Zend OPCache (PHP version 5.3) was disabled.

You need to restart the webserver for the changes to take effect.
Execute: webserver restart

A real life example

There is nothing wrong with the “Hello World” application but… it does nothing! So let’s move to a better example using a simple Silex application. This example application is an issue tracker that only list issues (that are stored in a MySQL database). Clone the application’s repository:

$ cd ~
$ git clone git://github.com/eriksencosta/silex-docker-example.git

Our application needs a MySQL database. But our PHP container don’t comes with a MySQL server. The power of Docker lies in using lightweight images as building blocks. So instead of creating an image with all application dependencies, you start different Docker containers and consume then as attached network resources. With this in mind, pull the MySQL Docker image:

$ docker pull mysql

Start the MySQL container:

$ docker run --name some-mysql -e MYSQL_ROOT_PASSWORD=root -d mysql

Start the PHP container:

$ docker run -t -i --link some-mysql:mysql -p 80:80 -v ~/silex-docker-example:/var/www/example -P "eriksencosta/php-dev:latest" /bin/bash

Run the needed set up commands:

# cd /var/www/example
# composer install
# php app/console example:configure:environment --variable="EXAMPLE_DEBUG=true"
# php app/console example:database:create
# php app/console example:schema:create

To check everything is working, run the test suite:

# behat

Load the fixtures data to the database:

# php app/console example:fixtures:load --truncate

Configure the Nginx virtual host and start the servers:

# cp dockerfiles/php-dev/default.vhost /etc/nginx/sites-available/default
# webserver start

In your browser, go to http://localhost. That’s all!

Wrapping up

I limited the scope of this post to present the Docker image and not to explain fully what is Docker and how to use it but I hope I’ve raised some interest about it on you.

This Docker image is intended only for development purposes (it is not meant to be used in production!). The image is built using the PHP Development Ansible playbook. The Dockerfile is on GitHub too. The Silex example application README have an extra usage example.

I'm a panelist on Electric Cloud's Continuous Delivery hangout - you're invited!

Tue, 09 Sep 2014 09:35:00 -0300

Update (17th September): check out the official event page.

On October 8th I’ll be joining an online panel of experts in Agile, DevOps and Continuous Delivery hosted by Electric Cloud who have done large Continuous Delivery projects for organizations like SpaceX, Cisco, GE and E*TRADE.

In the online hangout, myself and the other panelists will discuss how to deliver better software continuously, what are the obstacles on Agile adoption, what DevOps mean to us and how is the journey to get to Continuous Delivery, passing through Continuous Integration and deployment automation.

Electric Cloud will share their experience implementing Continuous Delivery at a large scale, and offer quick wins anyone can implement to accelerate software delivery. It will be a “grass roots” discussion of Continuous Delivery and how/to what extent we can implement it.

I am sure the hangout will be amazing. The people at Electric Cloud is organizing this event as craftsmen: with high standards for the discussion quality. The panelists include experts with different backgrounds in a diverse array of industries and from different countries.

Join us in the hangout, and be sure to fill out the short survey on the state of software delivery - we’ll review the results in the panel discussion.

When

October 8 at 10 AM PST (check your time zone)
Fill out the survey on the state of software delivery
Join the hangout: this link will start working minutes before the event. No registration is required. The hangout will happen online through GoToMeeting - in your web browser. Check the GoToMeeting system requirements (if joining using a mobile device, install the iOS or the Android app).

How to install RabbitMQ with the latest Erlang release on Debian

Mon, 06 May 2013 07:00:00 -0300

RabbitMQ is an Erlang application. The language moves fast and every new release has performance optimizations. This is good until you try to install the RabbitMQ DEB package in your Debian-like server and you see that the package manager installed an old version of Erlang.

For Debian Squeeze, the package erlang-nox comes with Erlang R14A (released in June’s 2010). The backports package comes with Erlang R15B01 (released in April’s 2012). For the current Ubuntu LTS (Precise), erlang-nox comes with Erlang R14B04 (released in October’s 2011) while the latest Raring package - like Debian Squeeze backports - comes with Erlang R15B01.

This post shows how to compile the latest Erlang release (R16B) in Debian Squeeze and how to satisfy the erlang-nox dependency of the RabbitMQ DEB package.

Compiling Erlang

The compilation process is very straightforward, we’ll basically follow the steps available in the official Erlang’s documentation. First, install some of its dependencies:

$ sudo aptitude install -y build-essential libssl-dev ncurses-dev m4

Explaining the dependencies:

build-essential: list of packages considered essential to build Debian packages. Some of these packages are needed to compile Erlang like gcc and make
libssl-dev: needed for SSL support
ncurses-dev: for improved terminal support
m4: needed to compile with HiPE support (High Performance Erlang)

Then download and compile Erlang:

$ sudo mkdir -p /opt/src/erlang /opt/erlang
$ cd /opt/src/erlang
$ sudo curl -O http://www.erlang.org/download/otp_src_R16B.tar.gz
$ sudo tar xzvf otp_src_R16B.tar.gz
$ sudo mv otp_src_R16B r16b
$ cd r16b
$ sudo ./configure --prefix=/opt/erlang/r16b --enable-hipe --with-ssl
$ sudo make
$ sudo make install

Finally, symlink the Erlang binaries (if you don’t want to use symbolic links, you can run configure with the --bindir=/usr/bin option):

$ sudo ln -s /opt/erlang/r16b/bin/dialyzer /usr/bin
$ sudo ln -s /opt/erlang/r16b/bin/epmd /usr/bin
$ sudo ln -s /opt/erlang/r16b/bin/erl /usr/bin
$ sudo ln -s /opt/erlang/r16b/bin/erlc /usr/bin
$ sudo ln -s /opt/erlang/r16b/bin/run_erl /usr/bin
$ sudo ln -s /opt/erlang/r16b/bin/run_test /usr/bin
$ sudo ln -s /opt/erlang/r16b/bin/typer /usr/bin

Installing RabbitMQ

Download the RabbitMQ DEB package available on the project’s website:

$ cd /tmp && curl -O http://www.rabbitmq.com/releases/rabbitmq-server/v3.1.0/rabbitmq-server_3.1.0-1_all.deb

As we already discussed, this package depends on the erlang-nox package:

$ dpkg -I rabbitmq-server_3.1.0-1_all.deb
 Package: rabbitmq-server
 Version: 3.1.0-1
 Architecture: all
 Maintainer: RabbitMQ Team <packaging@rabbitmq.com>
 Installed-Size: 4460
 Depends: erlang-nox (>= 1:12.b.3) | esl-erlang, adduser, logrotate
 Section: net
 Priority: extra
 Homepage: http://www.rabbitmq.com/
 Description: AMQP server written in Erlang
  RabbitMQ is an implementation of AMQP, the emerging standard for high
  performance enterprise messaging. The RabbitMQ server is a robust and
  scalable implementation of an AMQP broker.

If you try to install the package using aptitude, the package manager will install an old version of Erlang. The easiest way to prevent this is to use dpkg with the --ignore-depends option:

$ sudo dpkg -i --ignore-depends=erlang-nox
(Reading database ... 42128 files and directories currently installed.)
Preparing to replace rabbitmq-server 3.1.0-1 (using rabbitmq-server_3.1.0-1_all.deb) ...
Unpacking replacement rabbitmq-server ...
Setting up rabbitmq-server (3.1.0-1) ...
Adding group `rabbitmq' (GID 107) ...
Done.
Adding system user `rabbitmq' (UID 105) ...
Adding new user `rabbitmq' (UID 105) with group `rabbitmq' ...
Not creating home directory `/var/lib/rabbitmq'.
Starting message broker: rabbitmq-server.
Processing triggers for man-db ...

You can check that RabbitMQ is running using the rabbitmqctl command:

$ sudo rabbitmqctl status
Status of node rabbit@squeeze64 ...
[{pid,1776},
 {running_applications,[{rabbit,"RabbitMQ","3.1.0"},
                        {mnesia,"MNESIA  CXC 138 12","4.8"},
                        {os_mon,"CPO  CXC 138 46","2.2.11"},
                        {xmerl,"XML parser","1.3.3"},
                        {sasl,"SASL  CXC 138 11","2.3.1"},
                        {stdlib,"ERTS  CXC 138 10","1.19.1"},
                        {kernel,"ERTS  CXC 138 10","2.16.1"}]},
...

Resolving the dependency

What we did when installing RabbitMQ was to simply ask dpkg to ignore the dependency during the installation transaction. The problem is that for the package manager, the dependency is still unfulfilled. If you try to install or upgrade a package, the package manager will suggest to remove the rabbitmq-server package because of its unmet dependency:

$ sudo aptitude install tree
The following NEW packages will be installed:
  tree
0 packages upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 32.4 kB of archives. After unpacking 98.3 kB will be used.
The following packages have unmet dependencies:
  rabbitmq-server: Depends: erlang-nox (>= 1:12.b.3) but it is not going to be installed. or
                            esl-erlang which is a virtual package.
The following actions will resolve these dependencies:

     Remove the following packages:
1)     rabbitmq-server

Accept this solution? [Y/n/q/?]

If you don’t accept the solution, aptitude will suggest the installation of a lot of Erlang-related packages. This is not what we want, we want to have RabbitMQ installed with the Erlang version that we’ve compiled. To resolve this dependency, we’ll create a dummy package using equivs. From the equivs-build documentantion: “equivs-build is a program that creates Debian packages which can be used to inform dpkg about locally installed packages and their dependencies. Also empty packages that just require other packages can be created with equivs. These can be used as “profile” packages which just mark other ones for installation.”

Before installing equivs, uninstall RabbitMQ:

$ sudo aptitude purge -y rabbitmq-server

Then install equivs:

$ sudo aptitude install -y equivs

Create a file with the following contents:

Section: interpreters
Priority: optional
Standards-Version: 3.6.2

Package: erlang-nox
Version: 1:12.b.3
Maintainer: The Maintainer <maintainer@example.com>
Description: Dummy Erlang package.
  This package provides a dummy package for erlang-nox, a dependency of the rabbitmq-server package.

Or just copy and paste the following commands:

$ erlang_nox="Section: interpreters
Priority: optional
Standards-Version: 3.6.2

Package: erlang-nox
Version: 1:12.b.3
Maintainer: The Maintainer <maintainer@example.com>
Description: Dummy Erlang package.
  This package provides a dummy package for erlang-nox, a dependency of rabbitmq-server package from RabbitMQ."
$ IFS='%'; echo $erlang_nox > erlang-nox; unset IFS

Then run the equivs-build command to create the DEB package:

$ equivs-build erlang-nox
...
dpkg-deb: building package `erlang-nox' in `../erlang-nox_12.b.3_all.deb'.

The package has been created.
Attention, the package has been created in the current directory,
not in ".." as indicated by the message above!

Now, install the created DEB package and the RabbitMQ DEB package:

$ sudo dpkg -i erlang-nox_12.b.3_all.deb rabbitmq-server_3.1.0-1_all.deb

You can check that the dummy package was installed:

$ aptitude show erlang-nox
Package: erlang-nox
State: installed
Automatically installed: no
Version: 1:12.b.3
Priority: optional
Section: interpreters
Maintainer: The Maintainer <maintainer@example.com>
Uncompressed Size: 36.9 k
Description: Dummy Erlang package.
 This package provides a dummy package for erlang-nox, a dependency of rabbitmq-server package from RabbitMQ.

Now you don’t need to worry about aptitude complaining about the unmet dependency of the rabbitmq-server package. At least not while trying to install new packages. Hold on…

Marking the package as held

Everything is fine until you try to upgrade the packages from your system and you see something like this:

$ sudo aptitude safe-upgrade
Resolving dependencies...
The following NEW packages will be installed:
  erlang-asn1{a} erlang-base{a} erlang-corba{a} erlang-crypto{a} erlang-docbuilder{a} erlang-edoc{a} erlang-erl-docgen{a} erlang-eunit{a}
  erlang-ic{a} erlang-inets{a} erlang-inviso{a} erlang-mnesia{a} erlang-odbc{a} erlang-os-mon{a} erlang-parsetools{a} erlang-percept{a}
  erlang-public-key{a} erlang-runtime-tools{a} erlang-snmp{a} erlang-ssh{a} erlang-ssl{a} erlang-syntax-tools{a} erlang-tools{a} erlang-webtool{a}
  erlang-xmerl{a} libltdl7{a} libsctp1{a} lksctp-tools{a} odbcinst{a} odbcinst1debian2{a} unixodbc{a}
The following packages will be upgraded:
  erlang-nox
1 packages upgraded, 31 newly installed, 0 to remove and 0 not upgraded.
Need to get 20.8 MB of archives. After unpacking 36.5 MB will be used.
Do you want to continue? [Y/n/?]

Look at the erlang-nox file we created before. It declares it version as 1:12.b.3 while the Debian Squeeze erlang-nox package version is 1:14.a-dfsg-3squeeze1. If we created the dummy package declaring a most recent version, we’d have the same problem if a new version was released.

To resolve this, just mark the package as held with aptitude:

$ sudo aptitude hold erlang-nox

aptitude show will show the package as installed and held:

$ aptitude show erlang-nox
Package: erlang-nox
State: installed [held]
...

Now you are free from the package constraints and use the desired Erlang version for running not just RabbitMQ but any Erlang application!

Creating custom Vagrant boxes with Veewee

Wed, 20 Jun 2012 08:00:00 -0300

I already talked a bit of the handy Vagrant tool and about the DevOps philosophy. In the Vagrant post I showed how to use an available public Ubuntu box and how easy is to use the shell provisioner to setup a basic LAMP box.

Then I said that there are publicly available boxes out there but omitted the Vagrant Boxes site. Why the omission? Simply, I really don’t find them so useful and I think the drawbacks overweights the benefits. Grabbing a box that you don’t know how it was set up can lead to serious security problems. And if the box is just a basic installation, it is almost helpless since you’ll need to make the provisioning of your desired setup everytime you init a VM based in the box.

Why not just create a base box? Here comes Veewee.

Veewee

Veewee is a tool to easily build Vagrant boxes or KVM, VirtualBox and Fusion images. Hands-on, let’s open a terminal. You’ll need to have the RubyGems installed on your machine to install Veewee. Read the Vagrant post to see how to install it. Let’s install the latest available alpha release (note: the following commands were tested on an ol’ but good MacOS X Leopard.)

$ sudo gem install veewee --pre

Veewee comes with a lot of OS templates that you can use to build your Vagrant base box or VM image (and that’s why we used the alpha release since it contains updated templates for the major OS versions.) You can list the available templates using the following command:

$ vagrant basebox templates | sort

There are templates for popular Linux distros (Debian, Ubuntu, CentOS, Scientific Linux), some Unix flavors (Solaris, FreeBSD) and even for the Windows 8 preview release. Let’s build a Debian box. First, let’s define our base box with the name ‘squeeze64-lamp’ using Debian 6.0.4 AMD64.

$ mkdir -p ~/Dev/veewee/squeeze/amd64
$ cd ~/Dev/veewee/squeeze/amd64
$ vagrant basebox define squeeze64-lamp Debian-6.0.4-amd64-netboot
[vagrant] The basebox 'squeeze64-lamp' has been succesfully created from the template 'Debian-6.0.4-amd64-netboot'
[vagrant] You can now edit the definition files stored in definitions/squeeze64-lamp or build the box with:
[vagrant] vagrant basebox build 'squeeze64-lamp'

Looking at the definitions/squeeze64-lamp directory, we can see a bunch of files that are used to install the enough software needed for a Vagrant base box:

$ ls definitions/squeeze64-lamp/
base.sh       cleanup-virtualbox.sh definition.rb      puppet.sh       vagrant.sh        zerodisk.sh
chef.sh       cleanup.sh            preseed.cfg        ruby.sh         virtualbox.sh

The main file here is the definition.rb. It have a hash with some of the VirtualBox VM definitions (like the number of CPUs, the memory and disk sizes), boot and Kickstart definitions that are used to automatize the OS installation and the post-install files. Let’s setup the same LAMP box from the Vagrant post. Download or clone the vagrant-shell-scripts Git repository and change the post install files in definitions.rb using the following snippet:

:postinstall_files => [
  "base.sh",
  "vagrant.sh",
  "virtualbox.sh",
  "ruby.sh",
  "puppet.sh",
  "chef.sh",
  "dotdeb.sh",
  "db-mysql.sh",
  "dev-tools.sh",
  "php5.sh",
  "php5-qa.sh",
  "php5-tools.sh",
  "drush.sh",
  "cleanup-virtualbox.sh",
  "cleanup.sh",
  "zerodisk.sh"
],

Hint: put your post-install files anywhere after the base.sh file and before the cleanup.sh file.

With everything in place, build your box! This can take some time since it will download the Virtual Box Guest Additions and the OS ISO image (but just for the first time.)

$ vagrant basebox build squeeze64-lamp

After creating the box, validate it:

$ vagrant basebox validate squeeze64-lamp

With the tests’ ok, export your box:

$ vagrant basebox export squeeze64-lamp

Now you have a custom Vagrant base box that you can distribute for your IT team. Just add it to Vagrant and start using it!

$ vagrant box add squeeze64-lamp squeeze64-lamp.box
$ vagrant init squeeze64-lamp
$ vagrant up
$ vagrant ssh

Run from the user space: using Vagrant to setup development environments

Fri, 08 Jun 2012 06:50:00 -0300

A Brand New Day, by Thomas Hawk.

In the DevOps post, I said that Vagrant is a helpful tool that eases the creation and configuration of virtualized environments and make the “it works on my machine” an excuse from the past. Surely Vagrant is a tool that helps DevOps teams to achieve environment consistency. But you don’t need to practice DevOps to enjoy the benefits.

You know, as a developer, that is hard to maintain your development environment up to date when switching between different projects. Even tools like rbenv and phpenv (that helps supporting the coexistance of different versions of the same language in a machine), we still have problems when we need to use different versions of databases, application servers and other softwares that compound the dependency stack of the different projects.

Then you start compiling different versions of databases, creating scripts to start/stop services according to a project but with the passing of time you (can) ends with a machine so messy that the maintenance of it starts to become a nightmare.

Other problem is that you starts to feel your user space becoming less responsible in the course of time. The startup of your machine becomes slower, specially when you have a lot of software installed as service. If using an notebook with the power cord unplugged, your battery can be sucked faster.

Then you develop using Mac or Windows. That impressive new open source project? Sometimes it does not have (a good) support for it yet. When it have (using tools like MacPorts, Homebrew or Cygwin) you sometimes get stuck with an older version of a package.

I myself had problems like this. I have a Mac. In the last years I worked with a lot of different PHP-based projects and had a lot of problems to compile PHP with extensions that need libraries like PDFLib, ICU and libxml. Then a year ago I decided to help developing the Symfony2 framework, specifically the Locale component.

The Symfony2’s Locale component needs ICU since it uses the PHP’s ext/intl classes. But Fabien Potencier and the Symfony core developers had decided that the framework would not depends on non-core PHP features to run. So we needed to develop a way to emulate the used behaviors of the ext/intl classes in user land code. Then Igor Wiedler and me decided at the time to create the tests asserting against our implementation and the ext/intl implementation.

Then I needed to install some different versions of PHP and ICU to find which minimum version we would code for (we coded using ICU 4.2). I don’t wanted to compile all those softwares at mine machine (a Macbook with the ol’ but good MacOS X Leopard). I was thinking in switching back for some Debian-based distro as desktop.

Nothing beats a Linux distribution (and its various packages managers) when it comes for using open source projects for software development. But I still prefered Mac for the desktop environment. At the time I was already using VirtualBox for development and then I discovered Vagrant that made my life a lot easier. Since then I create a virtualized environment for every project. This post shows the way I personally use Vagrant.

Talk is cheap. Let’s go to the fun part!

Installation and the basic commands

Since Vagrant builds VirtualBox virtual machines, you’ll obviously need it installed (version 4.0.x or 4.1.x). Download a recent version of Vagrant, it have installers for Windows and MacOS X and packages for Debian, RedHat, and Arch Linux. Or you can install it using Rubygems:

$ sudo gem install vagrant

With Vagrant installed, you’ll have to add a box. A box is a tar package that contains a base virtual machine image. Start adding one of the official available base boxes of Vagrant:

$ vagrant box add lucid32 http://files.vagrantup.com/lucid32.box
$ vagrant box add lucid64 http://files.vagrantup.com/lucid64.box

When executing one of the commands above, the chosen base box will be downloaded from the Vagrant site. The first argument of the vagrant box add command is the name of the box in your Vagrant environment. You’ll use it to specify which box you want to use when creating a virtualized environment. You can list the available boxes at your Vagrant environment with the vagrant list command:

$ vagrant box list
lucid32
lucid64

After finishing the download, create your first virtualized environment using Vagrant:

$ vagrant init lucid32
A `Vagrantfile` has been placed in this directory. You are now
ready to `vagrant up` your first virtual environment! Please read
the comments in the Vagrantfile as well as documentation on
`vagrantup.com` for more information on using Vagrant.

We will look at the Vagrantfile in the next topic. As the message above state, just run the vagrant up command:

$ vagrant up

Then log in to the VM:

$ vagrant ssh
vagrant@lucid32:~$

Vagrant automatically set up a VirtualBox shared folder in the path /vagrant to the directory of the Vagrantfile:

vagrant@lucid32:~$ ls /vagrant/
Vagrantfile

After stopping using the virtualized enviroment, you can suspend (pause) or halt (shut down) it. First, do the log off:

vagrant@lucid32:~$ exit
$ vagrant status
Current VM states:

default                  running

The VM is running. To stop this VM, you can run `vagrant halt` to
shut it down forcefully, or you can run `vagrant suspend` to simply
suspend the virtual machine. In either case, to restart it again,
simply run `vagrant up`.

Then run the vagrant suspend or the vagrant halt command:

$ vagrant suspend
[default] Saving VM state and suspending execution...
$ vagrant status
Current VM states:

default                  saved

To resume this VM, simply run `vagrant up`.
$ vagrant halt
[default] Discarding saved state of VM...
$ vagrant status
Current VM states:

default                  poweroff

The VM is powered off. To restart the VM, simply run `vagrant up`

The Vagrantfile

The Vagrantfile is where all the definitions of a VM are. You can set up any definition of the VirtualBox VM. To set up your VM to use 512 MB of RAM, set the following:

config.vm.customize [
  "modifyvm", :id,
  "--memory", "512",
  "--name",   "My First Vagrant box"
]

If you want to use another base box, change the config.vm.box value to the name of the box:

config.vm.box = "lucid64"

Now imagine you’re developing a web application and that you want to let your folks at the same network to access it. You can set a static IP for this:

config.vm.network :hostonly, "192.168.33.11"

As we saw before, Vagrant configures the VM to use a VirtualBox shared folder automatically at the path /vagrant. While VirtualBox shared folders can be useful, its performance degrades quickly while the number of files in the shared folder increases. I do recommend using NFS and skipping the shared folders altogether (you need set the VM to use a static IP, like the previous example shows):

# Relative path
config.vm.share_folder("v-root", "/vagrant", ".", :nfs => true)

# Absolute path
config.vm.share_folder("v-root", "/vagrant", "/Users/eriksencosta/Dev/my-project", :nfs => true)

After changing your Vagrantfile, you’ll need to restart the virtual machine for the changes to take effect:

$ vagrant halt
$ vagrant up

If using the NFS shared folder, you’ll be prompted for administrator rights, as Vagrant will modifies the /etc/exports file to rightly configure the NFS server on the host machine.

It is the provisioners support that makes Vagrant a really useful tool. At this point, you saw how to launch a new VM and some basic setup but something was missing. For developing an application, we commonly need other softwares installed like programming languages, databases and the like. But the installation process must be repeatable and more importantly, automated.

Vagrant supports provisioning using Chef (Solo and Server), Puppet (Standalone and Server) and Shell. Its Chef and Puppet support is a valuable feature for learning and testing Chef’s Cookbooks and Puppet’s Modules. For simplicity, let’s provision a LAMP environment using the Shell provisioner.

Creating a LAMP environment using the Shell provisioner

Let’s create a LAMP environment. We’ll install Apache, MySQL and PHP packages available at Ubuntu’s repositories, some general development tools (Git, Subversion), PHP tools (Phing, Composer), PHP QA tools (PHPUnit, PDepend) and a Drupal specific tool (Drush). Download or clone the vagrant-shell-scripts Git repository. Then add the following snippet to your Vagrantfile:

Vagrant::Config.run do |config|
  config.vm.provision :shell, :path => "/path/to/vagrant-shell-scripts/db-mysql.sh"
  config.vm.provision :shell, :path => "/path/to/vagrant-shell-scripts/dev-tools.sh"
  config.vm.provision :shell, :path => "/path/to/vagrant-shell-scripts/php5.sh"
  config.vm.provision :shell, :path => "/path/to/vagrant-shell-scripts/php5-qa.sh"
  config.vm.provision :shell, :path => "/path/to/vagrant-shell-scripts/php5-tools.sh"
  config.vm.provision :shell, :path => "/path/to/vagrant-shell-scripts/drush.sh"
end

After saving the file, run the vagrant up command:

$ vagrant up

Vagrant will show you the progress of the provisioning. After finishing, you’ll have a fully configured environment. Log in and try to run the phpunit command!

Note: after halting a machine that have provision definitions, running vagrant up will execute the provisioning again. If you just want to run all the provisioning again, use the option --no-provision:

$ vagrant up --no-provision

The Vagrantfile is what makes the process of creating VM environments repeatable. It’s the infrastructure as code part of Vagrant. Add it to your project’s repository to share it with your team, then every developer will only need to run vagrant up to use the same working enviroment.

And don’t forget to read the excelent documentation available at the Vagrant’s site!

DevOps, bridging development and operations

Thu, 24 May 2012 11:30:00 -0300

Eleven years after the Agile Manifesto and much more years of continuous improvements in the Software Development’s Best Practices had a great impact in our industry. Practices like Test Driven Development (TDD) and Continuous Delivery enabled development teams to develop better software in a more predictably way. But there was still a gap - between operations and developers - and DevOps is the bridge that let us skip the gap.

Enters DevOps

DevOps is, according to Stephen Nelson-Smith, a “movement of people who think it’s time for change in the IT industry - time to stop wasting money, time to start delivering great software, and building systems that scale and last”.

The movement have roots in the Agile Infrastructure and Operations movements that were applying the Agile engineering practices to improve the Continuous Delivery. These movements were a response to a identified problem that was the fact that organizations that were developing software with Agile were fallbacking to a waterfall methodology when it was the time to deploy the developed software.

This boundary with the waterfall process is what Andrew Shafer calls the circle of happiness and wall of confusion. Basically the development team (developers, testers and product owners) sits inside this circle with regular feedback loops between iterations and smooth communication. But communication with people outside this circle is clumsy and this brings a lot of problems that affects the delivery of value.

Stephen also outlines four problems that our software industry faces today that DevOps can help to eliminate or mitigate: fear of change, risky deployments, “it works on my machine!” and siloisation. The fear of change leads to bureaucratic management processes that makes any change to the application (be it the introduction of a new feature or a bug fix) take a long time to be made. Risky deployments specially hurts the development and operations teams since this can lead to deployments at calm times - a lot of times at dawn, which can burn a lot of the teams’ energy - because there is no confidence in the software to be deployed.

“It works on my machine!” is a cliché. Who never experienced a problem that occurs on live and that when it’s reported to the developers, they say: “it works on my machine!”? It is not uncommon to find development teams using different OSes and/or the software’s dependencies environment in different versions from the production environments. This can lead to erroneous assumptions based on the different features available in the different layers of the software stack - which is specially evident when the developers does not know the infrastructure.

And the notorious problem with teams in different silos is the culture of “finger pointing” and the “us and them” mentality. Shafer’s circle of happiness is much about this problem: the different silos are afraid of each other, they collaborate and communicate poorly.

Culture changes

DevOps it is a lot about culture changing since it applies many of the lessons learned in the Agile Software Development practices. One of the lessons is all about communication and collaboration. It reduces the boundaries between development and operations by building a multidisciplinary team with people that, as Stephen points out, “feels comfortable with infrastructure and configuration, but also happy to roll up their sleeves, write tests, debug, and ship features.”

Bringing developers and operations people together means giving opportunities to discover better ways to deliver better software. It propagates the chain of business value that is well known in the circle of happiness to the operations team that starts to be viewed as part of the solution instead of just keeping the servers 24x7. It fills a gap that was missing from the Agile methodologies: it makes operations to be kept in sync with the development process, aligning the development and operations teams to an unified business process.

Surely it’s not easy to go from a black box operations team to a full DevOps team. It’s hard to find people with this multidisciplinary mindset. But DevOps, like Mitchell Hashimoto says, it’s not an absolute, it’s a range. You don’t need to aim having a team where the developers do all the operations, you need to aim your way of doing DevOps. It’s more about identifying people that can be the bridges between development and operations, encouraging and empowering them to do so.

Quoting Stephen again, DevOps “has a tremendous impact on the business. Suddenly the technical team starts trying to pull together as one. An ‘all hands on deck’ mentality emerges, with all technical people feeling empowered, and capable of helping in all areas. The traditionally problematic areas of deployment and maintenance once live become tractable - and the key battlegrounds of developers (‘the sysadmin built an unreliable platform’) versus sysadmins (‘the developers wrote unreliable code’) begins to transform into a cross-disciplinary approach to maximizing reliability in all areas. This, of course, has a positive effect on the bottom line - better reliability and availability, happier clients, faster time to market, and more time to focus the team’s energy on core business rather than wasteful administration and firefighting” (emphasis mine.)

Toolset

Besides the cultural changes, DevOps is loaded with good Agile practices applied in the infrastructure. One big change is the is the concept of Infrastructure in code. In DevOps, everything must be under version control - from network to applications configurations. There is no DevOps without version control since it is the base for all the other practices. Like software, the Infrastructure must be build from source and be equally treated as an application.

Building from source means having a bare metal server to running services with an automated process. The configuration management systems are very helpful here: they put a system into a known state using the configuration that is available in version control and helps achieving consistency in the IT environment. It enables the operations to manage the servers lifecycle, provisioning new boxes by means of their services, in a way similar to applying templates, giving servers responsibilities like “Database server”, “HTTP Cache server” or “Application server”.

There are some open source configuration management system available being Puppet and Chef the most known ones. You can find a lot of interesting use case of both systems. Tumblr (the popular blogging platform), for example, uses Puppet to update the development machines. Travis CI (the hosted continuous integration service) uses Chef to build the CI worker images that are responsible to run the tests for tens of thousands open source projects.

This kind of server management optimizes the operations from manually configuring servers which leads to inconsistency between similar service servers and prevents occurrences of what Shafer calls the “Mistery Machine” - a server that no one knows what runs and that everyone have fear to turn them off. Another important aspect of these systems is idempotence - in this context, the ability to apply the configuration template¹ in the same machine a lot of times without modifying the final result which is the configured machine as described in the template.

Others tools are equally helpful. One of them is Vagrant², which is a tool that eases the creation and configuration of virtualized environments. With Vagrant you can can, for instance, create a virtual machine with an environment as similar as possible from the production environment to be distributed to the team. This is really an important tool for making the “works on my machine” an excuse from the past. Vagrant is also used as a sandbox for testing templates of configuration management systems like Puppet or Chef. You can also just use plain old shell scripts to setup your environment.

The advances of the virtualization tecnologies also plays an important role for achieving better software quality and delivery. Take advantage of the almost ubiquitous cloud computing to quickly provide development, testing, staging and production environments that are as similar as possible for your team.

But also remember that most important than the tools is creating the culture. Create a team with cross-functional skills and involve it in the deployment activities. Make the deploy an activity that both developers and operations can do easily and at any time. Like yours project’s source code, make the infrastructure collectively owned and find your way of doing DevOps in baby steps³.

This is just a simplification of what these systems really provides. Puppet have Modules and Chef have Cookbooks. Both (Puppet Templating, Chef Templates) use templates as a a way of replacing or creating configuration files as described in a module or cookbook. ↩
Like Puppet and Chef, Vagrant is written in Ruby. Another scripting language that is being heavily used for writting this kind of tool is Python. A basic understanding of these languages can be very helpful, specially if you plan to extends functionality. ↩
Context and good sense are always important. I really like the way GitHub works. They don’t use any Agile methodology but is extremely successfull in its Continuous Delivery. ↩

Eriksen Costa

Introducing Kotlin Money

Allocation

Wrapping up

Platform engineering at N26: how we planned and launched it

Culture eats strategy for breakfast

Strategic planning

Mapping the Developer Experience

Planning to build less and the Thinnest Viable Platform

How our platform works

Closing remarks

Acknowledgments

Footnotes

Platform engineering: capabilities, roles, and its benefits for DevOps maturity

You build it, you run it

Cognitive load

Platform engineering

Higher DevOps maturity

Organizational design

Product Mindset

Roles

Capabilities of engineering platforms

Closing remarks

Footnotes

Building an Engineering culture

What is culture?

Vision, mission, values, and principles

Define it collaboratively

Document, share, integrate, and recognize

N26 Brasil Engineering values

Live the culture and be an example

Acknowledgments

Three highlights from the 2017 State of DevOps Report

Maturity model and continuous improvement

Architecture

Leadership

Closing remarks

References

Good design is informed and careful

A partial transformation

Design is everywhere

Conway’s law

Teams, individuals and code

Informed and careful design

Continuous improvement as a design practice

Design is innevitable

References

TensorFlow is commodity software

References

Mapping your domain knowledge

User Story Mapping

Crash course

Adding context to the map

Building shared understanding is the key

How can this motivate technology-driven developers?

Map to help with your DDD adoption and to produce an Ubiquitous Language

Map to assess your domain complexity

Closing remarks

Acknowledgements

References

Don't try Composer, adopt it

A Docker image for multi-version PHP development

Using the Docker image

Switching between the PHP versions

A real life example

Wrapping up

I'm a panelist on Electric Cloud's Continuous Delivery hangout - you're invited!

When

How to install RabbitMQ with the latest Erlang release on Debian

Compiling Erlang

Installing RabbitMQ

Resolving the dependency

Marking the package as held

Creating custom Vagrant boxes with Veewee

Veewee

Run from the user space: using Vagrant to setup development environments

Installation and the basic commands

The Vagrantfile

Creating a LAMP environment using the Shell provisioner

DevOps, bridging development and operations