I started examining where these issues were occurring in our code bases, taking a hard look at how we got there, and doing lots of research of why these things are they way they are.

It wasn’t until I came across Piotr Solnica’s dry-validation gem and some of the ideas behind Nick Sutterer’s Trailblazer framework that I was able to piece together several seemingly separate problems that ultimately are best solved with Nick’s concept of an “Operation”.

I first want to discuss a few of the pain points with complex Rails applications and ultimately how the idea of an Operation can solve some of them.

Short on time? tl;dr or listen:

User Interaction & Persisted Data Representation

Rails, by default, fosters a coupling between how the user interacts with your system and how data is persisted. When your UI or API starts to interact with more than one ActiveRecord Model in a single action, things quickly start to fall apart and you get stop gaps like the oft-maligned accepts_nested_attributes_for or validate_associated. You start passing attributes for models through other models, none of which necessarily have anything to do with the other.

Actions your user’s can perform and how you store data are not the same thing, so why are our UIs and forms married to our data store schemas? It may start out that way when you scaffold and CRUD your initial models, but as your application and business logic grows more complex, it tends to move away from strict data-entry.

Your UI and business logic should not be coupled with your ActiveRecord Models. ActiveRecord Models are a representation of your persisted data, they are not a place for business logic; they are used by business logic to persist data. Decoupling your business logic and the way you access data allows your business domain to manipulate data without causing a ripple effect across other processes. ActiveRecord models should be entirely devoid of logic except perhaps to enforce some internal consistency, such as associations, a state machine, or data-store-enforced restrictions (not null, unique, etc).

Your models are the building blocks of how your business processes work within your system, and if they do anything other than keep themselves consistent and persistent, they can become hard to reuse in all business contexts.

ActiveRecord and Validation

Reading Piotr Solnica’s blog post Invalid Object Is An Anti-Pattern and some of the points he made got me re-thinking about how validations should work, especially in the context of a large, complex system.

As explained above, ActiveRecord Models are a representation of your persisted data. Being able to represent something invalid as persisted data is a real problem… more so when you’re delegating through many levels of service objects and POROs, where you don’t have any assurances of where the object came from and its internal consistency… and you shouldn’t have to care, but as Piotr says so eloquently:

You can’t treat them [ActiveRecord models] as values as they are mutable. You can’t really rely on their state, because it can be invalid.

As a result, you can’t trust the data that you are passing around your application. How many if or present? statements have you thrown around in your code to deal with incomplete models?

ActiveRecord models should be limited to valid and persisted data, and the act of validating input to create or update this data made in a different context.

Contextual Validation

At first, validation seems pretty simple. Your models always adhere to your rules. It must always have this value present, it must always have an integer greater than 5, and so on. These rules live in your model, to enforce its internal state.

As your application grows more complex, always gives way to usually. Usually it has to be greater than 5, but an admin can set it to anything. Requests from the UI can return a maximum of 10 records, but API requests can return a maximum of 50.

Validation turns out to be very contextual. What is valid can depend a lot on who is doing the changes, where they are doing it, when they did it, and the values of other models at time. We can use if and unless in our ActiveRecord Models to solve some of these problems, but we start to leak knowledge of other models, user roles, and business logic into our model supposedly only used for interacting with persisted data.

There is a difference between what is always enforced for a model and what is enforced from the business domain’s perspective.

When is always true? To me, only things enforced by the data store, such as uniqueness indexes or NOT NULL attributes. These things have to be true, otherwise the model won’t save. Any other value is a business domain decision and its validation can be dependent on the context the data was received.

Domain Processes

The Code Climate article 7 Patterns to Refactor Fat ActiveRecord Models, is an excellent place to start learning how to extract business logic from ActiveRecord models. It helps you build up a layer of PORO objects, implementing all of your business logic and sitting between the UI/API/CLI and your persisted data.

In big, complex systems, it is often difficult to achieve this with consistency and interacting with your business layer to perform actions can require a lot of specific domain knowledge. While individual classes may have intention-revealing names and are beautifully architected, the entire process cannot be effectively communicated this way. This happens when your CTO drops into the CLI to credit a user’s balance and forgets to add an activity entry or kick off a related background worker.

These separate, decoupled but related pieces comprise of a high-level function performed on your system; coordinating many processes into a single, repeatable unit of action. You don’t want to tie balance updates together with creating activity feed entries, but there is a need to orchestrate the two together in the act of crediting an account.

Solving These Problems

The 4 problems we’ve covered are solvable on their own:

• Form Objects, provided by gems like ActiveType, Reform, and Virtus can decouple how user’s interface with your system and how your data is stored, and provide somewhere for contextual validation.

• The dry-validation gem can be used to validate input outside of an ActiveRecord Model instance and build trust in the persisted data you pass around.

• Organizing a top-level layer of service objects to orchestrate the processes and encapsulate the business rules, that then delegate to more specific classes… and the discipline to ensure you use it.

The benefit of an Operation is that it wraps all of these pieces, validation, UI decoupling and a business layer, together into a single, consistent entry point for performing an action on the system; whether that be via a form on the UI, a JSON API call, a process kicked off by cron, or a meddlesome CTO in the Rails console.

What is an Operation?

The Trailblazer framework defines an operation very well:

… an operation embraces and orchestrates all business logic between the controller dispatch and the persistence layer. This ranges from tasks as finding or creating a model, validating incoming data using a form object to persisting application state using model(s) and dispatching post-processing callbacks or even nested operations.

Note that operation is not a monolithic god object, but a composition of many stakeholders.

The last bit is key: an Operation itself does nothing. Everything is delegated but it provides a common, functional-style interface to your application and can orchestrate many separate actions into a cohesive business process.

In essence, you create a sort of DSL for interacting with your business layer that describes what you want to do, hiding the how.

Using Operations

A lot of this discussion has been conceptual and abstract. What does an operation look like in implementation and how does it solve all of these problems?

Treat an operation like a function: you pass a hash of simple input to a class method which runs the operation and returns an immutable instance of itself. This resulting instance can be used to render the UI or serialize a JSON response based on the result.

What the operation does is completely up to you, but it is likely you will have it orchestrate the authorization of the request, validation of the input, execution of the tasks and return the results.

For example, we have a form that contains some extra details about the user when they sign up for our website. The operation might look like this:

Our form's input span several models. Our input doesn't conform to any nested hashing based on how we are storing the data; even through a User may have a has_many relationship with SocialAccounts, our sign up form or JSON API doesn’t care.

The Operation defines a validation contract: what values it will accept and the rules that validate them. A password confirmation and a twitter handle are only required when a User with the customer role is created via our registration form. The User model doesn’t need to know about these rules and adding a form for creating admin users with different validations won’t be affected.

Once authorized and valid, the Operation executes the business logic. This could be directly creating ActiveRecord models, delegating to service objects, external API calls, creating background jobs… whatever is required. The operation just delegates the actual work to other objects but orchestrates the process as a whole.

Now, when you need to implement a CSV bulk customer import, maybe you don’t want to send birthday emails, or the password confirmation isn’t necessary; you can create another Operation that captures the specific process of the CSV import and its specific validation rules without duplicating any of the underlying functionality.

What Benefits Do We Get

By organizing our code this way, there are quite a few benefits.

1. Operations are not tied to models in any way, they are functional business processes. Each operation can validate, authorize and process itself in its own context and returns an immutable and stateless result.

2. Underlying data and models can be refactored without changing the operation’s public interface… or we can add more parameters, processes, and models without affecting what it already does, or other similar operations.

3. Controllers become ultra thin. They care nothing about validation, authorization, loading, creating, or updating. They know nothing about parameters (bye bye strong_parameters). They simply call the operation and render or redirect based on the result.

4. You end up with a folder full of performable actions. Onboarding new developers, understanding what your application can do and interacting with it becomes more clear.

5. Operations are the perfect target for acceptance/integration tests. They outline an entire, repeatable process, tying many underlying pieces together and the only way a user interacts with your entire system.

6. You can get rid of fixtures and/or factory_girl. Setting up your tests by creating (and then maintaining) the underlying data models is not how data is created in production. Your test can use the list of operations to build up a scenario exactly the same way it would occur in production… No maintenance required!

7. Operations are stateless with simple input, making it easy to run them inline or as asynchronous jobs.

Operation Implementations

Nick’s Trailblazer framework encapsulates a lot of this, including the ability to use dry-validation with his reform form object library, along with some other great ideas. You would be well served to take a look at it and understand the reasons behind the choices made in the framework.

He has recently extracted trailblazer-operations into its own separate gem, so you can grab it without needing to dive into the entire Trailblazer Framework.

I have authored a small, low-ceremony gem Operational that tackles the same problem but relies on Rails conventions rather than being framework agnostic, resulting in a powerful but simple library with much less code and no dependencies. It might be what you're looking for.

Do you know of any other implementations of the concept of operations? Comment below!

TL;DR

I don’t necessarily recommend you start your next greenfield project with all sorts of extra layers and throw away some of the quick, boilerplate free benefits Rails gives us. Applications made “the Rails Way” can and do work, but as your business layer gets more complex, Operations may provide a clean way to grow and organize your project.

• Separate interaction with your application and how data is stored. Interaction should focus on business operations, not how data is persisted.
• Validate input without creating invalid objects. Rely on gems like dry-validation to eliminate inconsistency so you can trust your ActiveRecord models.
• Almost all validation is contextual and business domain related. Remove it from your ActiveRecord models and validate it in the context you receive it.
• Simplify and standardize your application’s interface by wrapping up business actions as functional Operations, creating a pseudo API/DSL of your business domain.
• Have many small, decoupled objects to delegate to, but actions, whether its from an API call, Rails console, web form, background worker or a cron task, all start as an Operation.
• Take a look at Operational or Trailblazer to help organize a complex business domain into a consistent and functional interface of immutable, stateless, and repeatable Operations.

When building real-time and telephony communication applications, you will inevitably need to store phone numbers. Whether it's input you get from Freeswitch, Asterisk, or via an API like Tropo or Twilio, phone numbers can be tricky to handle, parse, verify, store, and display in your application.

Why Are Phone Numbers So Hard?

Phone numbers are very difficult to verify as their format can be dramatically different for various countries. Length, allowed starting numbers, reserved blocks, short codes and more make it very difficult to parse and verify a number is valid. When you receive a phone number input, does the number include the country code, an international dialing prefix, a national dialing prefix, an extension number, a special code like 411 or 911, or a special carrier command like *69 or 1157.

Just displaying phone numbers from around the world can be tricky as the groupings of numbers is different, such as in the US: (213) 555-1234 or the UK: (0)20 1234 5678. In addition, some countries have multiple formats! In Spain you can write a number like: 123 456 789 or 123 45 67 89.

Even the "country code" is mislabeled as 20 countries around North America share the same country code (thanks to the North American Numbering Plan).

How are we supposed to handle, verify, query, and work with this diverse pool of numbers that conforms to very few rules?

The E.164 Standard Format

You've probably run into a similar issue when working with date and timezones in your career as a developer. Date input can be just as varied as the phone number system and timezones add a unique wrinkle when outputing and comparing dates. This has (arguably) been solved with a standardized format ISO 8601, which unambiguously organizes dates and times with all the necessary localization information in a easily human readable and machine parseable format.

E.164 does that for phone numbers. It defines a simple format for unambiguously storing phone numbers in an easily readable string. The string starts with a + sign, followed by the country code and the "subscriber" number which is the phone number without any context prefixes such as local dialing codes, international dialing codes or formatting.

Numbers stored as E.164 can easily be parsed, formatted and displayed in the appropriate context... since the context of a phone number can greatly affect its format. So, with a UK number stored as +442012345678 we can easily display it in the appropriate format for the various contexts:

• +44 20 1234 5678 - UK International format.
• (0)20 1234 4567 - UK National format.
• 011 44 20 1234 5678 - Dialing from US to UK.
• 020 1234 5678 - Dialing locally within the UK.

E.164 stores the important parts of the phone number that never change in an easily parseable string that allows us to then format the number depending on the context which we are displaying it.

Now that you want to store your numbers as E.164, how do you parse and format them in your application?

Google's libphonenumber

There are lots of libraries out there that parse and format numbers into E.164, but I think that Google's open source libphonenumber is the best. Google's experience with international number support on their Android platform exposes them to more complete and accurate list of phone numbers around the globe.

With libphonenumber you can parse, verify, and format phone number inputs quite easily, do as you type formatting and even glean extra information about the number, like whether it was a mobile or landline or what state or province it was from.

libphonenumber in its basic form consists of a set of rules and regular expressions in an XML file for breaking down and parsing a number. Google provides a Java, Javascript, and C++ version of the lib, but people have ported it to other languages like Ruby, PHP, and Python.

In addition, it can provide offline reverse geocoding and map numbers to specific carriers if the data is available.

Other Special Cases

E.164 describes a format for internationally routeable numbers. Numbers that are reachable from many countries. Some special numbers do not meet this criteria like nationally specific numbers such as 911. Special numbers, especially emergency numbers like 911 or 112 require specific and often regulated handling, specific to your country. If you have to deal with these numbers, ensure you are meeting any required regulations and handle them as special cases. They are not formattable as E.164 numbers.

Extensions are another common piece of data when storing and collecting phone numbers. Think of extensions as extra information to send once you are connected. Extensions are not dialed when connecting to a phone number but are sent as extra instructions to the end system after you've connected to further direct your call... kind of like telephone NAT. They are not part of an E.164 number but are easily stored in a separate field and appended to any format for creating dialing strings or in the view.

TL;DR

• Parse and store all your phone numbers as E.164. It is easy to compare and unambiguous to parse.
• Use a library like libphonenumber to parse and format a phone number for output.
• Ensure you handle emergency numbers like 911 and 112 as special cases and make sure you are meeting any of your country's regulations.
• Extensions are not a part of a phone number but something to send after you've connected. They should be input and stored separately.

Rails 3 WebDAV Tutorial with Custom Resources, Authentication with Devise, and User Specific Routing

The great gem DAV4Rack and its creator Chris Roberts deserve a huge shout-out.

Note: The tutorial is part of a Wiki and is subject to change.

Update: I built a sample app for this and it is available on Github: github.com/bryanrite/dav4rack-example-devise-subdirectories

I will explain how this happens and some steps you can implement to mitigate it.

Most discussion seems to be focused very much on the obvious security difference: that data in the cookie is stored in plain text (well its stored in Base64, but that's trivially convertible).  Storing sensitive information or stateful data in a session, be it server side or cookie store, is bad practice and for most of us storing simple reference information, like the logged in user's id, this is not usually much of a concern.

Lets use the example: our cookie stores the logged in user's ID or nothing if the user isn't logged in. This is a common scheme used in many of the rails screencasts and guide books.

Instead of storing a session_id which identifies a server side storage record to get the user's ID, we store the user ID in the cookie. Generally this user ID isn't sensitive, its probably in the user's profile URL or within the code somewhere.

If you're following the above rule and only storing simple reference information, your cookie isn't much different than a server-sided store.  The HMAC digest within the Rails CookieStore cookie prevents someone from tampering and changing the cookie, so we cannot change our cookie to a different user ID and be logged in as that user, same way it would be difficult to guess a different session_id and access someone else's session.

Since both implementations rely on a cookie being passed anyways, they have the same security concerns, and are equally susceptible to replay and fixation attacks.

Ok, so what is our concern then?

There is one major difference that never seems to be brought up.  If (and when!) a CookieStore cookie is stolen:

By default, a CookieStore session will never become invalid.

By this, I mean if I steal an authenticated Cookie, I can use it to access the site as that user. This is a common attack called Session Hijacking or Sidejacking but with server-sided storage it can be mitigated.

Generally, with a server side store, you delete the session_id and accompanying data when a user logs out or times out.  Any stolen session_ids are no longer valid because that session_id no longer exists.  If the valid user never logs out and the attacker keeps sending requests, they can keep the session alive, but this can still be detected and stopped.

With the cookie based store, even if the valid user logs out and an expiry date is put on the cookie, an attacker can change the expiry date and replay the cookie at any time.  It will always be valid as there is no session_id to compare against and the cookie expiry is not guarded by the HMAC digest.

The Rails Guides suggest using reset_session to stop hijacking, but this does not help for CookieStore, there is no session identifier to reset!

Really?

Give it a try yourself! On any of your CookieStore based rails app:

• Load up Firefox and install the Live HTTP Headers or Tamper Data plugin, i'll use Live HTTP Headers.
• Log into your app.
• Start Live HTTP Headers
• Go to a page that shows you if you are logged in or not.
• On the Live HTTP Headers modal, select the main request header, its usually the top most one. Pretend you are an attacker and read this off a wireless network.
• Log out of your web app.
• Back on the Live HTTP Headers modal, you are now the attacker logging in with stolen cookie: replay the main request header.
• You will now be logged into your app. No matter what you do, the attacker can save that stolen cookie and replay it any time from anywhere and log back in.

How can we stop it?

Well, stopping it is quite easy, and I'll explain a couple of ways how, but the real reason I bring this up is because it isn't obvious to people using the default session store there is a huge concern here. Everyone talks about how never to store sensitive information in the cookie, but getting an authenticated cookie, by default, gives you life-time pass to that user's account. That seems much worse! Without a security measure, your application has a huge hole, and none of the rails tutorials, documentation, or screencasts seem to mention this.

No matter what authentication library you use: Devise, Authlogic, or if you roll your own, they are all susceptible because they all use whatever session store you decide.

The best and easiest solution is simply to use SSL.  Not just on your login forms and actions, but your entire site, or at least any pages where you have sessions turned on.  With SSL on, the user will not be able to replay your cookies and the entire attack vector is shut down. Rails 3.1 has a handy force_ssl switch you can use, and you can use something like:

:secure => Rails.env.production?

in your config.session_store declaration to ensure the cookie is only served over SSL.

If you don't or can't use SSL, try implementing a timeout and a nonce within the HMAC protected portion of the cookie.

For example, managing a session timeout yourself by updating the expiry date on every request unless the expiry has passed creates a non-editable timeout on the session and will invalidate it after a specific time. Of course the attacker can still touch your app to keep the timeout alive indefinitely, but it helps.

In addition to a timeout, adding a nonce, even something simple, can help invalidate existing cookies. Storing a hash based on the user's last login and/or logout time can invalidate stolen cookies every time the valid user logs in/out. This, coupled with the timeout can mitigate hijacking and puts it on par with server-based session management schemes.

Regardless of the session store mechanism you use, they're all susceptible to attack unless you're using SSL. Unfortunately, by default, Rails' CookieStore gives you a no-fuss lifetime pass instead of a day-pass.

I will explain some useful tips we learned while implementing Internationalization in one of my latest projects.

Understanding Globalization and Localization

Internationalization can be roughly broken down into two sections: Globalization and Localization, and understanding the difference between these is important.

When we talk about internationalization, people mainly think about multiple languages. For example, we want to offer our project in English and Chinese. What we’re really saying is we want to localize our project with two different languages. Essentially replacing the text depending on the user’s preference.

Now say that project is selling a product… are we going to change the currency symbols to match the language? Are all your prices going to be converted to Yuan for your Chinese speaking users? In some cases, yes; in a lot of cases, no, we still want to display currency and date formats in a single culture. This is the difference between Localization, language, and Globalization, culture.

Take for instance the Apple Store.  The Apple Store is Globalized for several different cultures.  When you go to the Chinese store, it will be selling its products to people in China, using the Chinese language and culture (zh-CN) is the right thing to do.  Now say they offered a version of its USA store in Mexican-Spanish.  Apple would likely not display currency in Pesos, it is still the USA store, they just want to make it more friendly to native Spanish speakers.  In this case we would use the US culture (en-US) and Mexican-Spanish language (es-MX): the text would be displayed in Spanish, but the currency symbols would continue to be dollars.

ASP.Net allows us to specify these settings separately:

Thread.CurrentThread.CurrentCulture = CultureInfo.CreateSpecificCulture("en-US");  
Thread.CurrentThread.CurrentUICulture = CultureInfo.CreateSpecificCulture("es-MX");  

What’s so important about the Culture

In my experience, cultural settings are most important in two areas: currency and date format.

Currency can have different currency symbols ($, €, ¥), different thousands separators, different decimal separators.  Dates can be displayed in different formats (m/d/y, d/m/y).

Since these can differ from our localization settings, we need a way to dynamically show them based on the culture (not the language).  ASP.Net has the handy .ToString() method which can take standard identifiers like “C” or “D”.  ASP.Net will automatically output the values based on your current culture, but when you want to customize the output, check out the following libraries:

System.Globalization.CultureInfo.CurrentCulture.NumberFormat  
System.Globalization.CultureInfo.CurrentCulture.DateTimeFormat  

They will have everything you need, from the current cultures date format strings to the currency symbol.  Below are a couple of Extensions I commonly use to display data.  As you can see they take the format from the culture and the language from the ui culture.

public static string ToLongDateNoWeekDayString(this DateTime date, bool abbrieviate)  
{
    string dateFormat = System.Text.RegularExpressions.Regex.Replace(System.Globalization.CultureInfo.CurrentCulture.DateTimeFormat.LongDatePattern, @",?\s*dddd?,?", "");

    if (abbrieviate)
        return date.ToString(System.Text.RegularExpressions.Regex.Replace(dateFormat, @"MMMM", "MMM"), System.Globalization.CultureInfo.CurrentUICulture);
    else
        return date.ToString(dateFormat, System.Globalization.CultureInfo.CurrentUICulture);
}

public static string ToLongDateTimeNoWeekDayString(this DateTime date, bool removeSeconds)  
{
    string pattern = (removeSeconds) ? @"(,?\s*dddd?,?)|(:ss)" : @",?\s*dddd?,?";
    return date.ToString(System.Text.RegularExpressions.Regex.Replace(System.Globalization.CultureInfo.CurrentCulture.DateTimeFormat.FullDateTimePattern, pattern, ""), System.Globalization.CultureInfo.CurrentUICulture);
}

public static string ToCulturalCurrencyWithoutSign(this decimal amount)  
{
    System.Globalization.NumberFormatInfo ni = (System.Globalization.NumberFormatInfo)System.Globalization.CultureInfo.CurrentCulture.NumberFormat.Clone();
    ni.CurrencySymbol = "";
    return amount.ToString("C", ni).Trim();
}

Regional Dialects

With the culture, we always have to specify a region. For example, our site’s culture cannot be English (en), it has to be USA English (en-us), Canadian English (en-ca), UK English (en-gb), etc. Because all of these have slightly different settings, there is no “English” culture.

This is not true with language. We can create generalized resource files as well as specify regional or dialect specific files. ASP.Net will automatically select the best resource file to use. For example, we can create resource files for general english and override it with regional specific one. All we have to do is name the resource files properly:

App_LocalResources  
- Example.aspx.en.resx
- Example.aspx.en-CA.resx
- Example.aspx.es.resx
- Example.aspx.es-DO.resx
- Example.aspx.es-MX.resx

Above we have 5 different resource files:

• en : General English.
• en-CA : Canadian English (we have more u’s!)
• es : Spanish
• es-DO : Dominican Republic Spanish
• es-MX : Mexican Spanish

ASP.Net will select the most specific file based on your UI Culture.

Localization Files and Variables

Another important feature of localization is the use of variables in our localization strings. Languages can be very complex and it is important to remember you cannot just break up a string around a variable.

For a very simple example, “You just added War and Peace to your Shopping Cart.”, is a common type of string you’ll need. The variable War and Peace can be replaced with any item you are adding to your shopping cart. You may be tempted to localize the string as:

1. You just added
2. to your Shopping Cart.

And output it around a variable on your page. Since in different languages, the structure of the sentence can vary greatly, it is highly recommended to use String.Format and numbered variables. This allows the localized versions of this sentence all the flexibility they need. You would want to localize the string like: You just added {0} to your Shopping Cart.

And display it with:

String.Format(GetLocalResourceObject("example").ToString(), item.Name)  

The same idea goes for logical paragraphs or basic formatting.  Instead of breaking a paragraph up into multiple separate sentences, always try to keep logically grouped text together as much as possible, and I personally have no problems allowing basic HTML formatting in localized strings.  This allows for smooth translations, as the more the text is broken up, the choppier the translations will be.

Plurals

The use of plurals is an important and difficult syntax to master.  In English, and some other languages, we only worry about 2 cases, maybe 3:

• You have multiple items.
• You have a single item.
• You have zero items.  (usually the same as multiple, but in some cases it can be awkward)

This can be managed by storing the 2-3 different strings in our resource files and extending GetLocalResourceObject to include an identifier for which string to use based on the count.

In many languages, it can be more complex then that.  In Polish for example, the grammar for 1, 2-4, and 0 or 4+ is different, and changes again after 20.  GNU’s gettext has a pretty good solution to this dynamic type of pluralization, but ASP.Net does not, you will have to come up with your own solution if that need arises in your project.

Using the Drobo, there were a couple of gotchas.  It doesn't have sshd enabled, no bash shell, no rsync (or rsnapshot which would make this easier) and more.

I will describe how I setup an automated, password-less, incremental differential, hard-linked backup using rsync across ssh.

Whats an incremental differential hard-linked backup?

It is essentially the same as Apple's Time Machine system.  We create a backup using only the files and directories that have changed, but still store it as a full backup on the backup media.  In other words, every time we backup, we will only transfer what has changed, but if you look at that backup on the remote computer, all the files will be there.

How do we do this?  Utilizing hard-linking in the filesystem.  Essentially we are creating two "pointers" that reference the same file (inode).  That way each backup folder has its own copy of the file, but we only store the file once.  If you run the command stat on a file or directory, you can see in the output how many links to that inode there are, or run ls -i to see the inode you are pointing to.  New versions of that file won't overwrite the old ones, the new backups will reference a different inode, keeping your historical data.

This helps for several reasons.  When we need to restore the entire filesystem, we can just move all the files from one directory, rather than merging the incrementals into a delta and applying it to the full backup to get our latest system and we only transfer what has changed which dramatically reduces bandwidth, time, and space.

Let's get started...

First things first, setup your Drobo as the instructions specify and be sure to enable DroboApps.  Create a share on the Drobo for your backups.  I'll name mine backup, if you use a different name, edit the below scripts as necessary.

You will then need to install the Dropbear SSH and Rsync apps for the following to work.  [Note: You do not need the Apache or Admin apps, but you might find them useful.]

At this point you should be able to login via SSH to your Drobo using the default SSH credentials, at the time of writing for the DroboFS its root and root.  We'll update the root SSH password using the following command:

/mnt/DroboFS/Shares/DroboApps/dropbear/root_passwd

This will reset and persist our root password between reboots.

[Note: I know that allowing root SSH access is not ideal but I have so far been unable to stop the drobo from allowing it if SSH is enabled... perhaps a commenter will have a solution?]

Now we need to edit the rsync config file:

vi /mnt/DroboFS/Shares/DroboApps/rsync/rsyncd.conf  

You should see one share named [drobofs]. We want to replace it so that the file looks like:

uid = root  
gid = root  
pid file = /mnt/DroboFS/Shares/DroboApps/rsync/rsyncd.pid

[drobofs]
        path = /mnt/DroboFS/Shares/backup
        comment = Backup Share
        read only = false

[Note: if you named your backup share different, you'll need to specify it here.]

Alright, now to provide password-less login.  From the machine that has the data you want backed up, generate your SSH keys.  You will likely not want to use a passphrase and I'll leave you to secure SSH as you see fit (perhaps read up about forced-commands-only).

user@server:~> ssh-keygen -t rsa  
Generating public/private rsa key pair.  
Enter file in which to save the key (/home/user/.ssh/id_rsa):  
Enter passphrase (empty for no passphrase):  
Enter same passphrase again:  
Your identification has been saved in /home/user/.ssh/id_rsa.  
Your public key has been saved in /home/user/.ssh/id_rsa.pub.  
The key fingerprint is:  
f2:13:b7:23:75:da:4e:35:a8:32:61:af:43:e1:a0:53 user@server  

Copy what is contained in id_rsa.pub. Now, on the Drobo we have to create the authorized_keys file to enable us to login with the key we just created. The required directory and files do not exist by default so, from the Drobo:

mkdir ~/.ssh  
vi ~/.ssh/authorized_keys  

Now copy the contents of id_rsa.pub into the authorized_keys file. You should now be able to SSH from the server to the Drobo without needing to enter a password.

Now that our apps are setup on the Drobo and communications will be seamless, we're going to create two scripts, one to rotate and manage the backups on the Drobo and one to do the actual rsync-ing.

The first script will reside on the Drobo in the root of the backup folder.  In our example:

/mnt/DroboFS/Shares/backup

I'll call it rotate_backups.sh and it'll look like the following.  Edit as necessary for your own purposes, and please note, the Drobo doesn't have the bash shell, so we're using good old #!/bin/sh

#!/bin/sh

# How many backups would you like to keep, each time you run
# the backup script, a new one will be created, so if you want:
# Daily for a week, script goes cron daily and enter 7.
# Hourly for 3 days, script goes cron hourly and enter 72 (24 hours x 3 days)
NUMOFBACKUPS=7

# Where are we backing up to?
BACKUPLOC=/mnt/DroboFS/Shares/backup

# Delete the oldest backup
NUMOFBACKUPS=`expr $NUMOFBACKUPS - 1`  
if [ -d $BACKUPLOC/backup.$NUMOFBACKUPS ]; then  
        echo "delete backup.$NUMOFBACKUPS"
        rm -Rf $BACKUPLOC/backup.$NUMOFBACKUPS
fi

# Move each snapshot
while [ $NUMOFBACKUPS -gt 0 ]  
do  
        NUMOFBACKUPS=`expr $NUMOFBACKUPS - 1`
        if [ -d $BACKUPLOC/backup.$NUMOFBACKUPS ] ; then
                NEW=`expr $NUMOFBACKUPS + 1`
                mv $BACKUPLOC/backup.$NUMOFBACKUPS $BACKUPLOC/backup.$NEW
                echo "Move backup.$NUMOFBACKUPS to backup.$NEW"
        fi
done  

This script will delete your oldest backup and move the others down the line... so your latest backup will always be backup.0, 1 run old is backup.1, 2 runs old is backup.2, etc.

Now that our backups are managed properly we will create the actual rsync script.  This will be run on the computer you want to back up from, and I suggest you add it to the crontab so it runs as often as you want it to. Edit BDIR, BSERVER, REMOVEDIR, and EXCLUDES as necessary.

#!/bin/sh

# Directory to backup.
BDIR=/data_to_backup

# Remote server (should match the password-less SSH credentials
# we setup earlier).
BSERVER=root@drobo

# Full path of the directory on the Drobo to backup to.
REMOTEDIR=/mnt/DroboFS/Shares/backup

# A list of files or directories to exclude.
EXCLUDES=/home/user/backup/excludes.txt

# Activate our rotate_backups script on the drobo.
ssh $BSERVER $REMOTEDIR/rotate_backups.sh

OPTS="--force --ignore-errors --delete-excluded --exclude-from=$EXCLUDES --delete -av --rsync-path=/mnt/DroboFS/Shares/DroboApps/rsync/rsync"

# Do the rsync.
rsync $OPTS -e 'ssh -p 22' --link-dest=../backup.1 $BDIR $BSERVER:$REMOTEDIR/backup.0/  

Add that script to your hourly/daily/whatever you want crontab and you should be good. The first time you run it you'll be transferring the entire backup directory so it might take a long time and you'll have an error message about --link-dest backup.1 not existing, but you can ignore that. Subsequent backups should run without a hitch.

/dev/md0:
        Version : 00.90.03
  Creation Time : Sun Nov 16 14:13:20 2007
     Raid Level : raid5
     Array Size : 732587712 (698.65 GiB 750.17 GB)
  Used Dev Size : 244195904 (232.88 GiB 250.06 GB)
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 0  
    Persistence : Superblock is persistent

    Update Time : Wed Dec 31 10:41:15 2008
          State : clean, degraded
 Active Devices : 3
Working Devices : 3  
 Failed Devices : 1
  Spare Devices : 0

...

    Number   Major   Minor   RaidDevice State
       0       0        0        0      removed
       1       8       17        1      active sync   /dev/sdb1
       2       8       33        2      active sync   /dev/sdc1
       3       8       49        3      active sync   /dev/sdd1
       4       8        1        -      faulty spare

One of the drives in my fileserver had died! Time to back up and get that sucker running again.

Please note: The following is only a guide to help you replace a failed disc. I cannot guarantee this will work for you, but it is what I do and has worked every time without any data loss.

As you can see, it is a 4 disc software RAID 5 array with no hot-swap spares. The following should work for most single disc failure situations in RAID 1, 5, or 6.

It appears that sda has bailed on me. First things first, backup the machine. If anything happens, you can rebuild from scratch.

You can see the faulty disc has already been removed from the array, but if yours hasn't been removed yet, the commands:

mdadm --manage /dev/md0 -f /dev/sda1  
mdadm --manage /dev/md0 -r /dev/sda1  

will mark it as failed (so it can be removed) and remove the sda1 partition.

Shutdown the machine and switch out the hard drives, make sure you only replace the faulty drive, don't mess up the order of the drives cause it'll be a pain to get it back in.

Boot up the machine. Your RAID array will be in the same degraded state. We need to partition the new drive exactly the same way we partitioned the drives in the existing array. Luckily this is a one-liner with sfdisk:

sfdisk -d /dev/sdb | sfdisk /dev/sda  

The above code will dump the partition table of sdb (or use any of the functioning drives) and pipe it to sfdisk to partition sda the same way. It should only take a second.

Then we can simply add the new drive to the array:

mdadm --manage /dev/md0 -a /dev/sda1  

If you take a look at cat /proc/mdstat or mdadm --detail /dev/md0 you should see that the array is recovering (with a percentage done).

After the recovery is done, you'll be back to new and clean!

Good luck!

Working with a GridView in an Update Panel is pretty quick and simple for many reasons, but when you require a full postback instead of an asynchronous one you can run into some trouble. If the actual object to cause the full postback has a set ID, no problem simply add:

<asp:UpdatePanel ...>  
   <ContentTemplate>...</ContentTemplate>
   <Triggers>
      <asp:PostBackTrigger ControlID="CONTROL_ID" />
   </Triggers>
</asp:UpdatePanel>  

The PostBackTrigger defined there will issue a full page postback. The problem arises on run-time controls such as a TemplateField or ButtonField within the GridView. There is no (easy) way to assign them as PostBackTrigger as either you don't know the ID (in a ButtonField example) or the ID changes based on any MasterPages or panels. The solution is simple though.

Basically all we have to do is add the control to the Trigger collection in the code behind on the DataBind event of the item. This will work with any type of control you want to put in your GridView. In this example, I'll use a LinkButton in a TemplateField.

... UpdatePanel definition etc. ...
<asp:GridView ...>  
   <Columns>
      <asp:TemplateField>
         <ItemTemplate>
            <asp:LinkButton CommandName="Select" ID="PostBackButton" runat="Server" Text="Do PostBack" OnDataBinding="PostBackBind_DataBinding">
            </asp:LinkButton>
         </ItemTemplate>
     </asp:TemplateField>
     ... Any other Columns ...
   </Columns>
</asp:GridView>  

Then in the codebehind:

protected void PostBackBind_DataBinding(object sender, EventArgs e)  
{
   LinkButton lb = (LinkButton) sender;
   ScriptManager sm = (ScriptManager)Page.Master.FindControl("SM_ID");
   sm.RegisterPostBackControl(lb);
}

The script manager has a handy RegisterPostBackControl method and the link buttons are dynamically set as full postback calls.

Note: If you aren't using a Master Page, you can just get the scriptmanager via its local reference: this.SM_ID.RegisterPostBackControl(lb);

Hope this helps someone out!