Mina: Deploy Spree Commerce with Sidekiq, Unicorn and Nginx

2015/01/30

Recently, I had a great time trying out Spree Commerce for the first time as a side project - an online store for my brother - which was also my crash course into a (somewhat serious) Rails project.

Most of the steps are referenced from John McCartie’s blog post, with additional Spree-specific tweaks. I am using Spree 2.4 in this post. The idea is to keep the whole process copy/paste friendly as much as possible.

If you are looking for a Mina tutorial for generic Rails app, that article is the best one you can find on the Internet.

Displaimer: Many shameless copy/paste step explainations from McCartie’s post..

Setup deploy user

First, ssh into your freshly created server and create a new user called “deployer” - which will be used by mina.

ssh root@your_server_ip
sudo adduser deployer

Now, modify the sudoers file to give deployer permission to run as sudo.

I’m using vim throughout the post, but feel free to use your prefered text editor.

sudo vim /etc/sudoers

Find the line root ALL=(ALL:ALL) ALL and add the line deployer ALL=(ALL:ALL) ALL after it. The result should look like this:

root    ALL=(ALL:ALL) ALL
deployer ALL=(ALL:ALL) ALL

Save and close the file.

Switch to deployer user

su deployer
cd

Setup swapfile and install required packages and ruby

We will create a 4G swapfile as backup in case the server runs out of memory:

sudo fallocate -l 4G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

Now check if swap space is setup successfully:

sudo swapon -s

If things go well, you should be able to see that the space allocated for swapfile:

Filename       Type     Size      Used   Priority
/swapfile      file     4194300   0      -1

Time to make the swapfile permanent so the server will automatically enable swapfile after reboot.

Edit fstab file:

sudo vim /etc/fstab

Add this line at the end of the file:

/swapfile   none    swap    sw    0   0

Save and close the file.

Change swappiness to 10, which is the recommended value for a VPS:

sudo sysctl vm.swappiness=10

Again, in order to keep the swappiness value after reboot, edit:

sudo vim /etc/sysctl.conf

Add this line at the bottom of the file and then save it:

vm.swappiness=10

If you want to findout more about swapfile and its configurations, see this article.

Install the required packages and ruby

Install the required packages:

sudo apt-get update
sudo apt-get install curl git-core build-essential zlib1g-dev libssl-dev libpq-dev libreadline-dev libyaml-dev libsqlite3-dev sqlite3 libcurl4-openssl-dev libxml2-dev libxslt1-dev python-software-properties nodejs

Now, we’ll install rbenv into your home directory and add some commands to your .bashrc for completion and shims.

git clone https://github.com/sstephenson/rbenv.git ~/.rbenv
echo 'export PATH="$HOME/.rbenv/bin:$PATH"' >> ~/.bashrc
echo 'eval "$(rbenv init -)"' >> ~/.bashrc

Now, let’s restart the shell and make sure Rbenv is install:

exec $SHELL
type rbenv
#=> "rbenv is a function"

Time to install Ruby!

git clone https://github.com/sstephenson/ruby-build.git ~/.rbenv/plugins/ruby-build
rbenv install 2.1.2
rbenv global 2.1.2

Your ruby will take a while to complete.

Ok, done?

Let’s make sure all is well…

ruby -v
#=> ruby 2.1.2p95 (2014-05-08 revision 45877) [x86_64-linux]

One last step that always trips people up: install Bundler real quick.

gem install bundler

Install PostgreSQL

(If you’re not using PostgreSQL, this step is not relevant.)

Install PostgreSQL:

sudo apt-get install postgresql postgresql-contrib

Login to psql console:

sudo -u postgres psql
#=> postgres=#

We will be using the default user postgres to connect to the database, but first, we need to change postgres password:

alter user postgres password 'YOUR_PASSWORD';

Now, prepare the database that will be used for your Spree store:

create database yourapp_production owner postgres;

Exit the psql console:

\q

Install Memcache, and Redis

Memcache

sudo apt-get install memcached

And Redis (for Sidekiq and other fun stuff)

sudo apt-get install redis-server

Boot ‘er up and make sure you can get to the Redis console:

sudo redis-server /etc/redis/redis.conf
redis-cli

Install and configure Nginx to work with unicorn and SSL:

SSL certificate

If you already registered your own SSL certificate, this step can be ignored.

Otherwise, generate one from the server and use it for our sandbox:

sudo mkdir /etc/ssl-certs
cd /etc/ssl-certs
sudo openssl req -new -x509 -nodes -out your_domain.crt -keyout your_domain.key

After following the promp instructions, your certificate files will be genrated at:

/etc/ssl-certs/your_domain.crt
/etc/ssl-certs/your_domain.key

The certificate location will be used later in your nginx configurations.

Install Nginx:

sudo apt-get install nginx
sudo service nginx start

Configure nginx to point to your Rails app:

First, backup the default nginx.conf file:

sudo mv /etc/nginx/nginx.conf /etc/nginx/nginx.conf.bk

Then create the new nginx.conf file:

sudo vim /etc/nginx/nginx.conf

And use these configurations for the file content. Change the worker_processes value according to your server’s hardware.

user www-data;
worker_processes 4;
pid /run/nginx.pid;

events {
  worker_connections 768;
}

http {

  ##
  # Basic Settings
  ##

  sendfile on;
  tcp_nopush on;
  tcp_nodelay on;
  keepalive_timeout 65;
  types_hash_max_size 2048;

  server_name_in_redirect off;

  include /etc/nginx/mime.types;
  default_type application/octet-stream;

  ##
  # Logging Settings
  ##

  access_log /var/log/nginx/access.log;
  error_log /var/log/nginx/error.log;

  ##
  # Gzip Settings
  ##

  gzip on;
  gzip_disable "msie6";

  ##
  # Virtual Host Configs
  ##

  include /etc/nginx/conf.d/*.conf;
  include /etc/nginx/sites-enabled/*;
}

Start creating your Rails site’s custom configurations:

sudo vim /etc/nginx/sites-available/yourapp

And add in this content - modify “your_app”, “your_domain” and the ssl certificate location according to your own Spree store. In case you are only using IP address for the test deployment, modify the configurations with your server’s IP.

Since I’m using SSL for the Spree store - and you should, too - there are 2 Nginx server blocks. “listen 80” takes non-http request and redirects to the SSL (listen 443) server block.

upstream app {
  # Path to Unicorn SOCK file, as defined previously
  server unix:/home/deployer/your_app/shared/sockets/unicorn.sock fail_timeout=0;
}

server {
  listen 80;
  # always use https
  return 301 https://$host$request_uri;
}

server {
  # always add www in front of the domain
  if ($host !~* ^www\.){
    rewrite ^(.*)$ https://www.your_domain.com$1;
  }

  listen 443;

  # Application root, as defined previously
  root /home/deployer/your_app/current/public;

  ssl on;
  # Change the location to your ssl certificates on the server
  ssl_certificate_key /etc/ssl-certs/your_ssl_cert.key;
  ssl_certificate /etc/ssl-certs/your_ssl_cert.crt;

  server_name www.your_domain.com your_domain.com;

  try_files $uri/index.html $uri @app;

  access_log /var/log/nginx/your_app_access.log combined;
  error_log /var/log/nginx/your_app_error.log;

  location @app {
    proxy_set_header X-Forwarded-For $remote_addr;
    proxy_set_header Host $http_host;
    proxy_redirect off;
    proxy_pass http://app;
    proxy_set_header   X-Forwarded-Proto https;
  }

  error_page 500 502 503 504 /500.html;
  client_max_body_size 4G;
  keepalive_timeout 10;
}

Symlink the nginx configurations to sites-enabled folder, and remove the default settings:

sudo ln -s /etc/nginx/sites-available/yourapp /etc/nginx/sites-enabled
sudo rm /etc/nginx/sites-enabled/default

Restart nginx:

sudo service nginx restart

Unicorn

Make sure you have unicorn and sidekiq added to your Gemfile and run “bundle install” on your local Spree app.

group :production do
gem 'sidekiq'
gem 'unicorn'
end

“This was another one that took me awhile, so I’m excited to share this setup with someone. I’m going to assume you know what Unicorn is and why you should use it. Here’s my unicorn.rb Rails initializer (config/unicorn.rb).”

# Set your full path to application.
app_dir = File.expand_path('../../', __FILE__)
shared_dir = File.expand_path('../../../shared/', __FILE__)

# Set unicorn options
worker_processes 2
preload_app true
timeout 30

# Fill path to your app
working_directory app_dir

# Set up socket location
listen "#{shared_dir}/sockets/unicorn.sock", :backlog => 64

# Loging
stderr_path "#{shared_dir}/log/unicorn.stderr.log"
stdout_path "#{shared_dir}/log/unicorn.stdout.log"

# Set master PID location
pid "#{shared_dir}/pids/unicorn.pid"

before_fork do |server, worker|
  defined?(ActiveRecord::Base) and ActiveRecord::Base.connection.disconnect!
  old_pid = "#{server.config[:pid]}.oldbin"
  if File.exists?(old_pid) && server.pid != old_pid
    begin
      sig = (worker.nr + 1) >= server.worker_processes ? :QUIT : :TTOU
      Process.kill(sig, File.read(old_pid).to_i)
    rescue Errno::ENOENT, Errno::ESRCH
      # someone else did our job for us
    end
  end
end

after_fork do |server, worker|
defined?(ActiveRecord::Base) and ActiveRecord::Base.establish_connection
end

before_exec do |server|
ENV['BUNDLE_GEMFILE'] = "#{app_dir}/Gemfile"
end

Mina

Now it’s time to add your local machine public key to the remote server’s authorized_keys so you can ssh without entering your user password.

Start a new local terminal window.

scp ~/.ssh/id_rsa.pub deployer@your_server_ip:./local_key.pub
ssh deployer@your_server_ip
touch .ssh/authorized_keys
cat local_key.pub >> .ssh/authorized_keys

Now exit the remote server and then try ssh into it again.

exit
ssh deployer@your_server_ip

If things go well, the remote server won’t promp for your password again.

In order for Mina to clone the git repo, you will first need to add your VPS ssh key to your git hosting service if you are using github, git bucket or gitlab.

Generate the ssh key pair on your VPS. (make sure you already ssh into the remote server as “deployer” user):

ssh-keygen -t rsa -C "your_email@example.com"
Now add your ssh key in '~/.ssh/id_rsa.pub' into your git hosting service as deployment key. More info on this step for github users.

Add Mina and Mina gems for sidekiq and unicorn to your Gemfile:

gem 'mina'
gem 'mina-sidekiq', :require => false
gem 'mina-unicorn', :require => false

And then

bundle install

Time to run Mina’s rake tasks:

mina init

This will generate the “config/deploy.rb” file.

Since we are using “deployer” user for Mina, you need to add all of your project’s folder inside the “/home/deployer/your_app” folder.

Modify ‘config/deploy.rb’ with this content:

require 'mina/bundler'
require 'mina/rails'
require 'mina/git'
require 'mina/rbenv'
require 'mina_sidekiq/tasks'
require 'mina/unicorn'

# Basic settings:
#   domain       - The hostname to SSH to.
#   deploy_to    - Path to deploy into.
#   repository   - Git repo to clone from. (needed by mina/git)
#   branch       - Branch name to deploy. (needed by mina/git)

set :domain, 'your_domain_or_server_IP'
set :deploy_to, '/home/deployer/your_app/'
set :repository, 'git@your_repo.git'
set :branch, 'master'
set :user, 'deployer'
set :forward_agent, true
set :port, '22'
set :unicorn_pid, "#{deploy_to}/shared/pids/unicorn.pid"

# Manually create these paths in shared/ (eg: shared/config/database.yml) in your server.
# They will be linked in the 'deploy:link_shared_paths' step.
set :shared_paths, ['config/database.yml', 'log', 'config/secrets.yml']


# This task is the environment that is loaded for most commands, such as
# `mina deploy` or `mina rake`.
task :environment do
  queue %{
  echo "-----> Loading environment"
  #{echo_cmd %[source ~/.bashrc]}
}
invoke :'rbenv:load'
  # If you're using rbenv, use this to load the rbenv environment.
  # Be sure to commit your .rbenv-version to your repository.
end

# Put any custom mkdir's in here for when `mina setup` is ran.
# For Rails apps, we'll make some of the shared paths that are shared between
# all releases.
task :setup => :environment do
  queue! %[mkdir -p "#{deploy_to}/shared/sockets"]
  queue! %[chmod g+rx,u+rwx "#{deploy_to}/shared/sockets"]

  queue! %[mkdir -p "#{deploy_to}/shared/log"]
  queue! %[chmod g+rx,u+rwx "#{deploy_to}/shared/log"]

  queue! %[mkdir -p "#{deploy_to}/shared/config"]
  queue! %[chmod g+rx,u+rwx "#{deploy_to}/shared/config"]

  queue! %[touch "#{deploy_to}/shared/config/database.yml"]
  queue  %[echo "-----> Be sure to edit 'shared/config/database.yml'."]

  queue! %[touch "#{deploy_to}/shared/config/secrets.yml"]
  queue %[echo "-----> Be sure to edit 'shared/config/secrets.yml'."]

  # sidekiq needs a place to store its pid file and log file
  queue! %[mkdir -p "#{deploy_to}/shared/pids/"]
  queue! %[chmod g+rx,u+rwx "#{deploy_to}/shared/pids"]
end

desc "Deploys the current version to the server."
  task :deploy => :environment do
  deploy do

    # stop accepting new workers
    invoke :'sidekiq:quiet'

    invoke :'git:clone'
    invoke :'deploy:link_shared_paths'
    invoke :'bundle:install'
    invoke :'rails:db_migrate'
    invoke :'rails:assets_precompile'

    to :launch do
      invoke :'sidekiq:restart'
      invoke :'unicorn:restart'
    end
  end
end

desc "Spree Commerce database seeding."
task :seed => :environment do
  queue "cd #{deploy_to}/current"
  queue "bundle exec rake db:seed RAILS_ENV=production"
end

With all that set, from your local Spree’s root directory, run:

mina setup

This command runs the one-time “task :setup” in “config/deploy.rb” file to create all the appropriate folders and files specific to your production/staging environment - which will be symlinked later during deployment. Look how simple and self-explainatory the code is!

If “mina setup” runs successfully, you should get a notice similar to this, idicating the folders and files were created on your remote server.

-----> Done.
-----> Be sure to edit 'shared/config/database.yml'.
-----> Be sure to edit 'shared/config/secrets.yml'.

ssh into remote server and start modifying “database.yml” and “secret.yml” file:

ssh deployer@your_server_ip

Add your database connection to database.yml:

vim /home/deployer/your_app/shared/config/database.yml

Edit the file with these configurations. Change database and password to the one created at PostgreSQL step.

production:
adapter: postgresql
encoding: unicode
database: your_app_production
username: postgres
password: DB_PASSWORD
host: localhost

Then, edit “secrets.yml” file:

vim /home/deployer/your_app/shared/config/secrets.yml

Insert this content:

production:
secret_key_base: RUN `rake secret` TO GENERATE A KEY

Deploying

Almost there! Keep calm and deploy!

mina deploy

If your deployment fails the first time - check the trouble shooting section below and Google - or even better - DuckDuckGo! Remember to run “mina seed” after the first successful deploy.

After successfully deploy the first time, we still need to run Spree Commerce data seed to setup the initial required locales, admin email/passwords, etc… - See “task :seed” in “deploy.rb” file - it is super straight-forward!

mina seed

Troubleshooting

If you get a git error during deploy:

fatal: Could not read from remote repository.

ssh into the remote server and do a manual clone of the git repo so that the git server added to your known_hosts file.

ssh deployer@your_server_ip
git clone git@your_git_repo.git dummy_repo

After following the promp, you should get this notice:

#Permanently added 'your_git_server_address' (ECDSA) to the list of known hosts.

At this step, if you are unable to manually clone the repo, make sure your remote server ssh key is added to the git service deployment key and have the permission to clone the repo. (See the git step above).

Do some cleanup:

rm -rf dummy_repo

Go back to your spree local root location and try deploying again:

mina deploy

Error during Installing gem dependencies using Bundler

If you are getting errors during “Installing gem dependencies using Bundler”, run “bundle install” from your rails local first to update “Gemfile.lock” file:

bundle install

Commit and push the changes:

git add -u
git commit -m "update Gemfile.lock"
git push -u origin master #or the branch you are using for production

Deploy, again!

mina deploy