engineering@contactually

Tackling Architectural Debt: How We Replaced a Production Elasticsearch Cluster

Photo by Alice Pasqual As the quantity and complexity of application data scales with any burgeoning startup, we all run into performance and scalability issues. Some issues can be addressed with slight adjustments to existing infrastructure, while others require fundamental changes to the system’s architecture. At Contactually, one such instance of this was dealing with an unhealthy Elasticsearch cluster that suffered from availability issues, syncing issues, and fundamental inefficiencies of data flow through our systems. This article will focus primarily on the large architectural change we made at Contactually in June of 2016 to reinforce an unreliable Elasticsearch cluster. Why We Use Elasticsearch at Contactually Elasticsearch is a distributed, RESTful search and analytics engine capable of solving a growing number of use cases. As the heart of the Elastic Stack, it centrally stores your data so you can discover the expected and uncover the unexpected. At Contactually, we use Elasticsearch to allow our users to quickly filter their contacts by numerous search criteria such as name, email address, mailing address, phone number, zip code, and even custom fields the user has created. Speed is key here — many of our users have hundreds of thousands of contacts. We store billions of data points in our normalized, relational data model. I won’t go into the pros and cons of normalization in this article, but they are very important to consider when designing any high-throughput system. Denormalized document storage Imagine trying to find a record in your database by attributes that could be in any of 10 tables. This might require an extremely expensive join, especially if the scale of the data you’re working with is...

5 Aspects of Rapid Feature Validation & Iteration

When Eric Ries published The Lean Startup in 2011, he outlined foundational principles of rapidly testing business hypothesis through product experiments. Startups like Contactually know they should split-test, be data-driven, and “Move fast and break things”, but actually applying these principles is another story. Rapidly validating and iterating on product features requires your team to establish tooling and methodology for doing so, which requires explicit effort and commitment. Contactually product development process has evolved into a fairly effective but simple pattern for running experiments that involves 5 different areas of methodology. These 5 aspects of experiment execution have allowed us to drive powerful business outcomes like improving our onboarding’s user activation rate by 20% Additionally, these 5 areas serve to reasonable balance several types of assessments: Subjective and objective quality of changes Qualitative and quantitative effects of changes Short and long term effects of changes 1. A/B Testing Tools Key to any controlled experiment is tooling that enables you to apply the new functionality to an experimental group of users, while maintaining a control group to compare against. While many A/B testing tools exist for swapping out HTML or front-end components, Contactually wanted the ability to control deeper backend behavior (Ruby on Rails) based on experiment groupings. Things like changing scoring algorithms, triggering alternative background jobs, exposing or not exposing certain ActiveRecord relationships. This led us to write our own Rails service to control A/B experiments. Contactually’s “Feature Flipper” Service Our custom-rolled Feature Flipper service allows us to create and manipulate experiment “Features” via a Rails console, or an internal admin panel for our non-technical employees, and is backed by Redis. Our FeatureFlipper service give us the following...

Engineering Great Design Process

Contactually is a company all about people and relationships. We love our customers — and hope they love us too! For this reason and many others, strong usability and user experience design are critically important to the success of our business. I recently joined Contactually to lead the user experience (UX) team, after having held similar roles in companies such as AddThis, Oracle, and America Online. In this blog post I’ll share some strategies that our designers, product managers, and developers use to successfully collaborate and build great experiences together. Establish Design Standards Frontify Design resources such as style guidelines, design pattern libraries, and corresponding front-end component libraries aren’t just about consistency. These tools work together to help teams design, build, and iterate faster. At Contactually we make style guidelines and brand assets available to the larger organization using Frontify, and are in the process of converting the Contactually web application to React, using Semantic UI and Storybook, a UI component development environment for React. React is a component-based Javascript library for building interfaces, and will allow us to reuse interface components across the Contactually platform. Tools such as these help us create more consistent, customer-friendly experiences more quickly — a big win for a small team. Storybook interface for React Components Balance Building, Iterating, and Fixing Product design and development roadmaps always require trade-offs. There are new features to be explored, existing features to test and iterate, customer questions and concerns to address, and bugs to fix. As an agile design and development team, we plan sprints by distributing a percentage of “points” — which correspond to available developer time — to each of those priorities. We collaborate closely with...

Postgres at Scale: Query Performance and Autovacuuming for Large Tables

Photo by Yu-chuan Hsu There are few large, hard to solve problems that keep your typical software engineer up at night — and a malfunctioning database is absolutely one of them. We’ll walk you through how our team at Contactually discovered the issue of query performance slippage on some of our largest tables (200 mil — 3 billion+ records) and what we did to address it. The Problem: Systematic Degraded Query Performance In early 2016, we noticed query performance on our largest tables were starting to slip in NewRelic. Over the course of a few weeks, queries that were once taking 10ms were now taking upwards of 6 seconds or more. The team considered numerous causes from poorly written queries to expensive background jobs that may be causing systematic performance issues. Eventually, with the aid of NewRelic, Amazon RDS logs, and hundreds of EXPLAIN queries, we came to the conclusion that this was a database level issue, not poorly written queries in our application (a very common cause of poor database performance). The Culprit: Default Autovacuum Settings After days of research, we narrowed in on the cause: the default autovacuum settings for large tables. Photo by Clem Onojeghuo The default settings can cause weeks to elapse without triggering an autovacuuming process to run on a large table. When autovacuuming doesn’t occur, the query planner is using outdated, incorrect data to decide how to most efficiently execute a query. Imagine trying to tell a friend where the milk is in a grocery store using directions from what the store looked like 5 years ago — you’re friend is going to waste a lot of time using your outdated directions. Critical...
Building Contactually’s iOS App 3.0

Building Contactually’s iOS App 3.0

A few months ago, our team decided to revisit our mobile experience, and to evaluate alternative means of developing the next version. In the end, we decided to build our app in RubyMotion! We did this for a variety of reasons, but they largely boiled down to familiarity with the language and available libraries (both Ruby and Cocoapods). After a few months of busily writing code, we thought we’d take some time to talk about how that decision worked out. To start out: we’ve shipped iOS v3.0! It’s in the App Store, and if you haven’t already grabbed it, go do that now, it should provide some context to the rest of this article. Technical highlights of the new app: Customized table views with “cards” Gestures tied to swipe movements on the cards Progressively-loaded lists Detail views with nested tables 400% faster than the previous version (2.4) iPad-friendly through constraints How We Approached Development Building an iOS app on a “blank slate” was both a blessing and a curse. We had no technical debts from a previous API or architecture holding us back, but we were also starting from nothing and needed to build it all back up again. We started with the API (v2 will be published soon!), and took an outside-in approach to its development. From there, we built out a Ruby client that allowed us to interact with the API as though it were our persistence layer (database). With the foundation of these models, we were able to concentrate on the views and interactions. Everything else was table stakes. Everyone expects an app to be able...
Picking a Mobile App Framework

Picking a Mobile App Framework

Note: This is part two in a series.  Part one is here. There’s no shortage of ways to build a mobile app these days. HTML5 is on the upswing (again), and quasi-native frameworks are popping up almost as often as Javascript libraries. Here at Contactually, we wanted to take the time to re-evaluate all of the current leaders in the mobile app framework space to give us the most bang for our buck. We were looking for the framework that gave us: Longevity – we don’t want to have to write from scratch again Interoperability – we have a dev team of very talented Ruby and web developers that we’d like to use across teams if possible. This also ensures that the mobile devs would be able to troubleshoot anything that goes wrong with our API Hirability – we want to make sure that we don’t pick a language that’s too specialized, which would make it hard to make any experienced hires (or experienced hires prohibitively expensive) Developer Happiness – writing for mobile shouldn’t be a burden; we want to pick a language and framework that makes the most sense to the most developers In this search we looked at several mobile app frameworks, but ended up prototyping with three frameworks based on our initial research: Swift 2.0 – The truly native choice. It’s the only one that we looked at that’s sponsored (and actively promoted) by Apple ReactNative 0.12.0 – Facebook’s newly-published framework that brings React to the native environment, complete with bindings RubyMotion 4.4 – Commercial product that allows you to write in Ruby and compile down to App/Play Store...