engineering@contactually

Upgrading a Production Postgres RDS Instance with Minimal Downtime

Photo by Luca Bravo Scaling a web application requires numerous database upgrades, each potentially requiring downtime. I’m going to tell you a story about Contactually, and the lessons we’ve learned during a few of these upgrades. Before upgrading — practice, practice, practice. Early last year, we switched from Unicorn to Puma, a multi-threaded web server. During our initial rollout, we were seeing intermittent spikes in Request Queueing. During root cause analysis, we zeroed in on spikes in Postgres Queue Depth, which is where read and write queries are queued until Postgres is able to complete them and return a response to the client. Additionally, we wanted to upgrade our RDS instances from Postgres 9.4 to 9.5 to take advantage of new features and improvements. Analyzing the Root Cause When the spike in the Postgres Queue Depth occurs, it ties up Puma threads which are waiting for their answer from the database. When enough of these occur, all threads on all servers are stuck waiting from an under-performing database. This means there are no threads available to handle new HTTP requests that are coming in. These new HTTP requests that can’t be worked on sit in the Puma request queue for so long that they timeout before any of the threads are able to address them. It’s worth mentioning that scaling up the number of web servers would only exacerbate the problem because it would increase the load on an already under-performing database. Additionally, these Postgres Queue Depth spikes cause background jobs to take much longer than ordinary. In some cases, these jobs can timeout depending on application specific logic. The Solution — Increase IOPS Photo by Marcus Spiske The number...
Building Contactually’s iOS App 3.0

Building Contactually’s iOS App 3.0

A few months ago, our team decided to revisit our mobile experience, and to evaluate alternative means of developing the next version. In the end, we decided to build our app in RubyMotion! We did this for a variety of reasons, but they largely boiled down to familiarity with the language and available libraries (both Ruby and Cocoapods). After a few months of busily writing code, we thought we’d take some time to talk about how that decision worked out. To start out: we’ve shipped iOS v3.0! It’s in the App Store, and if you haven’t already grabbed it, go do that now, it should provide some context to the rest of this article. Technical highlights of the new app: Customized table views with “cards” Gestures tied to swipe movements on the cards Progressively-loaded lists Detail views with nested tables 400% faster than the previous version (2.4) iPad-friendly through constraints How We Approached Development Building an iOS app on a “blank slate” was both a blessing and a curse. We had no technical debts from a previous API or architecture holding us back, but we were also starting from nothing and needed to build it all back up again. We started with the API (v2 will be published soon!), and took an outside-in approach to its development. From there, we built out a Ruby client that allowed us to interact with the API as though it were our persistence layer (database). With the foundation of these models, we were able to concentrate on the views and interactions. Everything else was table stakes. Everyone expects an app to be able...
Picking a Mobile App Framework

Picking a Mobile App Framework

Note: This is part two in a series.  Part one is here. There’s no shortage of ways to build a mobile app these days. HTML5 is on the upswing (again), and quasi-native frameworks are popping up almost as often as Javascript libraries. Here at Contactually, we wanted to take the time to re-evaluate all of the current leaders in the mobile app framework space to give us the most bang for our buck. We were looking for the framework that gave us: Longevity – we don’t want to have to write from scratch again Interoperability – we have a dev team of very talented Ruby and web developers that we’d like to use across teams if possible. This also ensures that the mobile devs would be able to troubleshoot anything that goes wrong with our API Hirability – we want to make sure that we don’t pick a language that’s too specialized, which would make it hard to make any experienced hires (or experienced hires prohibitively expensive) Developer Happiness – writing for mobile shouldn’t be a burden; we want to pick a language and framework that makes the most sense to the most developers In this search we looked at several mobile app frameworks, but ended up prototyping with three frameworks based on our initial research: Swift 2.0 – The truly native choice. It’s the only one that we looked at that’s sponsored (and actively promoted) by Apple ReactNative 0.12.0 – Facebook’s newly-published framework that brings React to the native environment, complete with bindings RubyMotion 4.4 – Commercial product that allows you to write in Ruby and compile down to App/Play Store...
Lessons Learned: Four Years of Startup Mobile App Development

Lessons Learned: Four Years of Startup Mobile App Development

Here at Contactually we are reinvesting in our mobile apps.  Four years after our founding, we finally have the ability to build mobile applications that live up to the standard of our core web experience. To make that a reality, we’ve taken a step back to plan out our vision and framework for the future. In doing so, we wanted to document how we got here, and hopefully help any other startups who find themselves pulled in a million directions to develop products across all the platforms available today (just kidding about that last one). In part one, we’re going to give a history of our mobile app development — the choices we made at the time and lessons learned. Lessons Learned from 4 Years of Startup Mobile App Development 2012 Our founders were web developers (Ruby on Rails) who naturally focused on building out our app for the web. We specialized in scaffolding things out quickly and seeing how users reacted. After gaining traction, by 2012 the demand for an iOS app had grown to the level where we knew we had to do something. HTML5 apps were gaining in popularity (though not without some major detractors later that year), and it was a technology we already understood, so we went for it. We built our first iOS application using Appcelerator Titanium, and covered all the basics of Contactually. You could create contacts, bucket them, and make notes about them. Because we understood the core of the technology before building it, it only took a month to go from first commit to initial release. During that development, we also built out the first version of our...

Moving Contactually to Rails 4.2

A brief history of Contactually’s usage of Rails When we first started building Contactually three years ago, we were running Rails 3.2.8 and felt pretty proud to be on the bleeding edge. Through the years, we dutifully upgraded through all of the subsequent point releases, but never made the major version jump to 4.x. Increasingly, the gems we use to run the project stopped supporting 3.x, and we soon realized it was time to get off that sinking ship while we still could. We didn’t want to fall too far behind the major versions of Rails, or our most heavily-used gems. We were losing out on potential performance gains, new features, and community support — and truthfully, we just like playing with the latest and greatest. Most importantly though, the biggest reason for us to upgrade was simply to stay at least one major version away from obsolescence. For those of you who still need to move up to Rails 4.2, here’s what that transition looks like, circa February 2015. The process We dipped our toes in the water by first updating the required version of Rails in our Gemfile. While we were at it, we also upgraded the Mongoid version from 3.1.3 to 4.0.0. After that, it was a simple bundle install to upgrade our Gemfile.lock. Our old configuration files needed to be updated before we could even see how many of our tests were failing, so we started a new, empty Rails 4 project to compare them. The first thing we noticed was that the blank project generally used Rails.application and the only place OUR_APP_NAME was referenced...

JavaScript Test-Driven Development with Rails 4 & Konacha

At Contactually, our app’s back-end is written in Rails 4, while most of the front-end is a Single-Page Application (SPA) written in JavaScript, using Backbone.js, Marionette.js, and a number of JavaScript plugins. On the back-end, we have a suite of RSpec unit specs and Capybara acceptance specs to make sure everything works on the back-end. When I joined Contactually in September 2014, the front-end code didn’t have any unit tests – everything was tested manually by traditional QA processes. In our app, there’s a lot of business logic between different objects. A User has many Contacts and Groupings A Contact can be in many Groupings, but only certain subtypes of Grouping, such as Buckets A Grouping can belong to many Users A Domain can have many Users, and many Groupings Etc. Almost all of this logic is spec’ed out and verified by RSpec and Capybara specs, but a lot of the same business logic on the client-side was not, which led to hard-to-find bugs in the Backbone app. As an example, after creating a new Contact, the UI didn’t update properly despite the new Contact was created successfully on the back-end. I set up a Test-Driven Development (TDD) environment for testing the various Views, Models, Controllers, and Mixins that make up Contactually’s Backbone app. Going forward with new UI features or bug fixes, any developer can set up a JavaScript spec with minimal overhead and keep the TDD ball rolling. Since Contactually is a Rails app, I chose a JS testing framework that’s similar in style to RSpec. I chose Mocha (and Chai) over alternatives such as QUnit or...