Simplifying Complex Rails Apps with Operations

I work on several large and mature Rails applications and have recently been feeling a lot of pain as these applications become more and more complex.

I started examining where these issues were occurring in our code bases, taking a hard look at how we got there, and doing lots of research of why these things are they way they are.

It wasn’t until I came across Piotr Solnica’s dry-validation gem and some of the ideas behind Nick Sutterer’s Trailblazer framework that I was able to piece together several seemingly separate problems that ultimately are best solved with Nick’s concept of an “Operation”.

I first want to discuss a few of the pain points with complex Rails applications and ultimately how the idea of an Operation can solve some of them.

Short on time? tl;dr or listen:

User Interaction & Persisted Data Representation

Rails, by default, fosters a coupling between how the user interacts with your system and how data is persisted. When your UI or API starts to interact with more than one ActiveRecord Model in a single action, things quickly start to fall apart and you get stop gaps like the oft-maligned accepts_nested_attributes_for or validate_associated. You start passing attributes for models through other models, none of which necessarily have anything to do with the other.

Actions your user’s can perform and how you store data are not the same thing, so why are our UIs and forms married to our data store schemas? It may start out that way when you scaffold and CRUD your initial models, but as your application and business logic grows more complex, it tends to move away from strict data-entry.

Your UI and business logic should not be coupled with your ActiveRecord Models. ActiveRecord Models are a representation of your persisted data, they are not a place for business logic; they are used by business logic to persist data. Decoupling your business logic and the way you access data allows your business domain to manipulate data without causing a ripple effect across other processes. ActiveRecord models should be entirely devoid of logic except perhaps to enforce some internal consistency, such as associations, a state machine, or data-store-enforced restrictions (not null, unique, etc).

Your models are the building blocks of how your business processes work within your system, and if they do anything other than keep themselves consistent and persistent, they can become hard to reuse in all business contexts.

ActiveRecord and Validation

Reading Piotr Solnica’s blog post Invalid Object Is An Anti-Pattern and some of the points he made got me re-thinking about how validations should work, especially in the context of a large, complex system.

As explained above, ActiveRecord Models are a representation of your persisted data. Being able to represent something invalid as persisted data is a real problem… more so when you’re delegating through many levels of service objects and POROs, where you don’t have any assurances of where the object came from and its internal consistency… and you shouldn’t have to care, but as Piotr says so eloquently:

You can’t treat them [ActiveRecord models] as values as they are mutable. You can’t really rely on their state, because it can be invalid.

As a result, you can’t trust the data that you are passing around your application. How many if or present? statements have you thrown around in your code to deal with incomplete models?

ActiveRecord models should be limited to valid and persisted data, and the act of validating input to create or update this data made in a different context.

Contextual Validation

At first, validation seems pretty simple. Your models always adhere to your rules. It must always have this value present, it must always have an integer greater than 5, and so on. These rules live in your model, to enforce its internal state.

As your application grows more complex, always gives way to usually. Usually it has to be greater than 5, but an admin can set it to anything. Requests from the UI can return a maximum of 10 records, but API requests can return a maximum of 50.

Validation turns out to be very contextual. What is valid can depend a lot on who is doing the changes, where they are doing it, when they did it, and the values of other models at time. We can use if and unless in our ActiveRecord Models to solve some of these problems, but we start to leak knowledge of other models, user roles, and business logic into our model supposedly only used for interacting with persisted data.

There is a difference between what is always enforced for a model and what is enforced from the business domain’s perspective.

When is always true? To me, only things enforced by the data store, such as uniqueness indexes or NOT NULL attributes. These things have to be true, otherwise the model won’t save. Any other value is a business domain decision and its validation can be dependent on the context the data was received.

Domain Processes

The Code Climate article 7 Patterns to Refactor Fat ActiveRecord Models, is an excellent place to start learning how to extract business logic from ActiveRecord models. It helps you build up a layer of PORO objects, implementing all of your business logic and sitting between the UI/API/CLI and your persisted data.

In big, complex systems, it is often difficult to achieve this with consistency and interacting with your business layer to perform actions can require a lot of specific domain knowledge. While individual classes may have intention-revealing names and are beautifully architected, the entire process cannot be effectively communicated this way. This happens when your CTO drops into the CLI to credit a user’s balance and forgets to add an activity entry or kick off a related background worker.

These separate, decoupled but related pieces comprise of a high-level function performed on your system; coordinating many processes into a single, repeatable unit of action. You don’t want to tie balance updates together with creating activity feed entries, but there is a need to orchestrate the two together in the act of crediting an account.

Solving These Problems

The 4 problems we’ve covered are solvable on their own:

• Form Objects, provided by gems like ActiveType, Reform, and Virtus can decouple how user’s interface with your system and how your data is stored, and provide somewhere for contextual validation.

• The dry-validation gem can be used to validate input outside of an ActiveRecord Model instance and build trust in the persisted data you pass around.

• Organizing a top-level layer of service objects to orchestrate the processes and encapsulate the business rules, that then delegate to more specific classes… and the discipline to ensure you use it.

The benefit of an Operation is that it wraps all of these pieces, validation, UI decoupling and a business layer, together into a single, consistent entry point for performing an action on the system; whether that be via a form on the UI, a JSON API call, a process kicked off by cron, or a meddlesome CTO in the Rails console.

What is an Operation?

The Trailblazer framework defines an operation very well:

… an operation embraces and orchestrates all business logic between the controller dispatch and the persistence layer. This ranges from tasks as finding or creating a model, validating incoming data using a form object to persisting application state using model(s) and dispatching post-processing callbacks or even nested operations.

Note that operation is not a monolithic god object, but a composition of many stakeholders.

The last bit is key: an Operation itself does nothing. Everything is delegated but it provides a common, functional-style interface to your application and can orchestrate many separate actions into a cohesive business process.

In essence, you create a sort of DSL for interacting with your business layer that describes what you want to do, hiding the how.

Using Operations

A lot of this discussion has been conceptual and abstract. What does an operation look like in implementation and how does it solve all of these problems?

Treat an operation like a function: you pass a hash of simple input to a class method which runs the operation and returns an immutable instance of itself. This resulting instance can be used to render the UI or serialize a JSON response based on the result.

What the operation does is completely up to you, but it is likely you will have it orchestrate the authorization of the request, validation of the input, execution of the tasks and return the results.

For example, we have a form that contains some extra details about the user when they sign up for our website. The operation might look like this:

Our form's input span several models. Our input doesn't conform to any nested hashing based on how we are storing the data; even through a User may have a has_many relationship with SocialAccounts, our sign up form or JSON API doesn’t care.

The Operation defines a validation contract: what values it will accept and the rules that validate them. A password confirmation and a twitter handle are only required when a User with the customer role is created via our registration form. The User model doesn’t need to know about these rules and adding a form for creating admin users with different validations won’t be affected.

Once authorized and valid, the Operation executes the business logic. This could be directly creating ActiveRecord models, delegating to service objects, external API calls, creating background jobs… whatever is required. The operation just delegates the actual work to other objects but orchestrates the process as a whole.

Now, when you need to implement a CSV bulk customer import, maybe you don’t want to send birthday emails, or the password confirmation isn’t necessary; you can create another Operation that captures the specific process of the CSV import and its specific validation rules without duplicating any of the underlying functionality.

What Benefits Do We Get

By organizing our code this way, there are quite a few benefits.

1. Operations are not tied to models in any way, they are functional business processes. Each operation can validate, authorize and process itself in its own context and returns an immutable and stateless result.

2. Underlying data and models can be refactored without changing the operation’s public interface… or we can add more parameters, processes, and models without affecting what it already does, or other similar operations.

3. Controllers become ultra thin. They care nothing about validation, authorization, loading, creating, or updating. They know nothing about parameters (bye bye strong_parameters). They simply call the operation and render or redirect based on the result.

4. You end up with a folder full of performable actions. Onboarding new developers, understanding what your application can do and interacting with it becomes more clear.

5. Operations are the perfect target for acceptance/integration tests. They outline an entire, repeatable process, tying many underlying pieces together and the only way a user interacts with your entire system.

6. You can get rid of fixtures and/or factory_girl. Setting up your tests by creating (and then maintaining) the underlying data models is not how data is created in production. Your test can use the list of operations to build up a scenario exactly the same way it would occur in production… No maintenance required!

7. Operations are stateless with simple input, making it easy to run them inline or as asynchronous jobs.

Operation Implementations

Nick’s Trailblazer framework encapsulates a lot of this, including the ability to use dry-validation with his reform form object library, along with some other great ideas. You would be well served to take a look at it and understand the reasons behind the choices made in the framework.

He has recently extracted trailblazer-operations into its own separate gem, so you can grab it without needing to dive into the entire Trailblazer Framework.

I have authored a small, low-ceremony gem Operational that tackles the same problem but relies on Rails conventions rather than being framework agnostic, resulting in a powerful but simple library with much less code and no dependencies. It might be what you're looking for.

Do you know of any other implementations of the concept of operations? Comment below!

TL;DR

I don’t necessarily recommend you start your next greenfield project with all sorts of extra layers and throw away some of the quick, boilerplate free benefits Rails gives us. Applications made “the Rails Way” can and do work, but as your business layer gets more complex, Operations may provide a clean way to grow and organize your project.

• Separate interaction with your application and how data is stored. Interaction should focus on business operations, not how data is persisted.
• Validate input without creating invalid objects. Rely on gems like dry-validation to eliminate inconsistency so you can trust your ActiveRecord models.
• Almost all validation is contextual and business domain related. Remove it from your ActiveRecord models and validate it in the context you receive it.
• Simplify and standardize your application’s interface by wrapping up business actions as functional Operations, creating a pseudo API/DSL of your business domain.
• Have many small, decoupled objects to delegate to, but actions, whether its from an API call, Rails console, web form, background worker or a cron task, all start as an Operation.
• Take a look at Operational or Trailblazer to help organize a complex business domain into a consistent and functional interface of immutable, stateless, and repeatable Operations.