Incrementally migrating a legacy app to Phoenix

Over the years we have done lots of projects where we migrated an application from one platform to another. We might do this to solve performance issues or to switch to a better technology stack. This can be a challenge when you have a big app that is in production, and you need to do it incrementally.

It depends, of course, on the specific technologies and application functionality, but the process is overall quite similar:

Analyze your existing system and prioritize the work

Before starting, analyze the logs from your existing system. That will give you information about the parts which are having failures and performance problems. You may be surprised about what kind of traffic you are getting, e.g. bots or broken clients.

You can add the $request_time variable to your Nginx logs to get information about how long it took to handle each request, identifying bottlenecks. Look at the slow query logs in your database to see problematic queries, you will need to deal with them in the new system as well.

There may be performance problems caused by bad HTTP/HTML practices on the legacy system, e.g. lack of JS/CSS asset consolidation. Use the network analysis tool in Chrome to see what assets the page is loading. Spending a bit of time making sure that static assets are served from Nginx or a CDN can help with performance during the transition with low risk.

Implementing production monitoring. That will help you stay on top of errors while your are transitioning and make sure that things are working properly. It will show you 404 errors from broken links, etc, as you roll out the new system.

The result is a better understanding of how your legacy system works and a list of priorities for things to fix.

Put the app behind a reverse proxy such as Nginx

Put both apps behind a proxy and use routes to direct traffic to one app or the other, allowing the two apps co-exist. You might have the API on api.example.com, or you might direct certain URL prefixes to Phoenix while keeping the rest in the legacy app. A good example is splitting off the API requests on /api for performance while keeping the user registration and/or admin pages in the old app.

Write the new or replacement features in Elixir/Phoenix

If the app is REST based, then you can generate standard REST controllers/routes. You can also take advantage of plugs to implement common logic like authentication, avoiding repetition.

Create Ecto database schemas for your legacy tables. After you switch to Phoenix, you will want to have database migrations, but during the transition period it's probably enough to just have a db snapshot that you can talk to.

Write tests

Tests give you the confidence to make changes and know that things are working properly.

If you are making an exact replacement, then you have an opportunity to verify that the old code and new code behaves the same. This is particularly easy for API endpoints, because there is no UI to get in the way. Just collect inputs and expected outputs from your legacy system and turn them into ExUnit test cases. A logging or caching HTTP proxy can help with this.

Share authentication between the apps

This allows users log in on one system and access pages on the other.

For an API, it's generally straightforward, e.g. we just validate an API key, though it might involve something like OAuth2. In any case, the new system doesn't have to interact much with the legacy system.

For a web app, it can mean getting into the guts of the session mechanism on your legacy system. Most commonly, that involves getting a session id from a cookie and looking it up in a db, Redis or Memcached. Then you may need to parse the data stored in the session, e.g. to extract the user id and use it to authenticate the user.

This can get ugly if the legacy system is using a language-specific serialization format for its convenience, e.g. PHP serialization. We wrote a parser for the PHP serialization format in Erlang years ago... if there is interest, I will share it.

If you control the source system, then you may be able to modify it to make your life easier. For example, you could switch to a standard format for session data like JSON. If the legacy system is using a proprietary session store, you can likely switch it to use a database (e.g. MySQL or Memcached), allowing both systems to talk to it.

Another option is to set a new cookie on login just for interop, e.g. after the user logs in, write a JWT which has the user id. Then the Elixir side can grab the data easily using a library like Joken or PlugJwt.

Things may be more urgent if you find that the legacy application was not using secure password practices, e.g. it is storing passwords in clear text. In that case, it might make sense to first migrate the login process to Phoenix and fix the security issues, while creating sessions which are compatible with the legacy system. As you transition, you may need to expire sessions, forcing people to log in again. This lets you e.g. update from MD5 hashes to bcrypt, capturing their password, generating the new hash and updating the database.

In an enterprise environment, where you have a lot of systems that may need to interop for a long time, then you can use a single-sign-on system like SAML.

Convert the page layout and navigation

For web apps, you may want to have some pages in the new app and some in the legacy app, allowing users to seamlessly work between both apps. The only thing the user will notice is that the Phoenix pages are 10x faster :-).

To do that you need to take the page layout and convert it to Phoenix format, including compatible navigation links.

That may be straightforward, but probably you want to update the style on the new system, and it doesn't make sense to spend too much time on the old design. It can take a surprisingly large amount of time to incrementally update the graphical design on an existing site. It's faster and more predictable to implement a new template and graphical design, so we need a transition plan that spends the minimum amount of work that will be wasted when we update the design later.

For one legacy site written in Symfony, the original templates were a mess, so we just did "File | Save Page As" in the browser to create the HTML template in all of its horrible glory. We ran it through tidy to make it valid XHTML, then roughly cut into templates with header, body and footer.

There are Elixir implementations of the template syntax of many different systems. For example you could use Django templates to convert templates, integrating with Phoenix using PhoenixDtl. Long term you are probably better off just converting templates to EEx, though.