Deploying A Crystal Application to AWS Elastic Beanstalk With Docker

more beer

I recently had to deploy an application which uses Crystal and Kemal, and wanted to document my experience in case it can benefit others. I generally prefer utilizing Amazon AWS when deploying services, Elastic Beanstalk makes deploying scalable web applications especially easy. I’ve created an open source application with some of the code here. I’ve also picked up a 12er of New Belgium VooDoo Ranger and a Mikes Harder strawberry lemonade, and I suggest you do the same. Judge me… I don’t care.

I initially had a single Dockerfile that would build and serve the application, but I wanted to run on a smaller t2.micro instance size and ran into out of memory errors when compiling the application. What I ended up with was a two step build process: One Docker container only run locally to build the executables and the Dockerfile deployed to AWS, its only job is to serve the already compiled application. I actually like this better because building locally doesn’t take long, and re-deploying on elastic beanstalk literally takes seconds. You can cross compile crystal but I went this route instead, because Docker is cool yo.

BuildDockerfile

The BuildDockerfile installs some system libraries, adds the necessary files to compile the application and compiles. Running the BuildDockerfile just copies the compiled application to an attached volume on the host computer.

You might notice some funny business going on in there. I’m greedy and I wanted to run multiple application processes per container, handled by server.cr and cluster.cr. Crystal’s HTTP::Server#listen has an optional argument reuse_port which makes use of the linux kernels SO_REUSEPORT socket option, allowing you to bind multiple application instances to the same port. So this hack is just to change the kemal.cr server.listen to pass true for the reuse_port option before compiling.

Dockerfile and deployment

The Dockerfile just copies the built executable and runs it, nothing fancy going on there. I created a helper script to do all of this, and spit out a zip file which can be uploaded to Elastic Beanstalk. It builds the BuildDockerfile then runs it binding the ./build/ folder as a volume. Then zips all the require files into a single archive.

Deployment is simple. Create an Elastic Beanstalk application and web server environment, choose Generic Docker platform and upload the build zip file, and thats it!

I hope this helps others who might be trying to accomplish some of the same problems. As always feedback or harsh criticism is accepted and welcome.

DRINK UP!

Authentication From Devise To A Django Database

Because of the awesomeness of the Rails Admin gem I recently had to connect a rails app using Devise to an existing Django application database. Django comes with a barebones admin much like padrino, and I’m sure there are Python libraries to extend the functionality of it. But I already know how to use Rails Admin and the process of creating a new rails app, getting the rails admin gem in and deploying on an ec2 instance through elastic beanstalk takes literally 5 minutes.

Obligatory beer pic. (this stuff is my jam lately, and comes in a 15 pack)

beer again

I should specify I’m using Rails 5.0, the Django application is 1.8.4

My first instinct was to reverse-engineer the Django authentication method to figure out the hashing scheme, then replicate it in Rails. Fortunately enough, after some hellacious googling I came across this tasty little gem pbkdf2_password_hasher. aherve had already done the heaving lifting for me! Cheers bro.

Here’s what my User model looks like:

class User < ApplicationRecord
	self.table_name = 'auth_user'

	devise :database_authenticatable, :registerable,
		:recoverable, :rememberable, :trackable, :validatable

	attr_accessor :encrypted_password, :current_sign_in_at, :remember_created_at, :last_sign_in_at,
		:current_sign_in_ip, :last_sign_in_ip, :sign_in_count

	def valid_password?(pwd)
		Pbkdf2PasswordHasher.check_password(pwd, self[:password])
	end

	def encrypted_password
		self[:password]
	end

	def encrypted_password=(pwd);end
end

Booyakasha.

booyakasha

Any fields that Devise might be trying to access that don’t exist, I simply added a attr_accessor for, except encrypted_password which I had to map to the existing hashed password field, in our case password.

I had to override Devise valid_password? method to return the result of the pbkdf2_password_hasher Pbkdf2PasswordHasher.check_password method.

Hope this helps somebody.

Fantasy Football Genetic Algorithm in Crystal

This blog post fueled by Watch Man IPA

Inspired by a recent talk at Nebraska.code() conference - Artificial Intelligence: A Crash Course from Josh Durham over at Beyond the Scores, I set out to try some AI / Machine learning of my own.

Perhaps one of the more interesting topics in the field, IMO, is the Genetic Algorithm - emulating biological evolution over a data set using natural selection, mutation and breeding. I’m not going to pretend to be an expert on the topic, to the contrary I am a complete noob and suggestions on how to improve my code are very welcome.

And now the hardest part, finding a suitable application for testing and creating the algorithm.

Last year I started playing fantasy football, using a Rails app I created that allows me to track my team and make efficient recuitments / trades based on the data from the Fantasy Football Nerd API. I also tried my hand at FanDuel and wrote some brute-force functions (not really knowing much about linear algebra) to try to build the best team with the highest expected points while staying under the salary cap. But thats boring and took a long time, a reeeeally long time if I used the entire data set - billions of possible combinations.

The fantasy football binary knapsack problem.

This idea isn’t unique or novel in any way, a quick search returns dozens of others that have applied some kind of genetic algorithm to the fantasy football knapsack problem. The one thing that does make this unique, is that its written in Crystal ;)

My genetic population is a list of randomly generated teams, each containing 9 players (quarterback, two running backs, three wide receivers, a tight end, kicker and defence). Links to Team and Player classes.

The Team class, or - the chromosome, contains several important methods:

  • The fitness method returns the total expected points for the team.
  • The mutate method, takes a random position on the team and replaces it with another random player of the same position.
  • Also breed and create_child methods, which takes traits from the 2 parents to produce child teams.

The main run loop (here) creates a population of 10,000 teams and evolves it a total of 80 times.

In the evolve function:

  • The population is sorted by fitness (highest first) and the top %65 of teams will continue live in the population, the remaining will be killed off.
  • Teams have a small chance of being mutated during the loop (0.005).
  • We repopulate our population by breeding two random from the surviving teams.

Check out the beast in action: http://recordit.co/lu8ZEV916D

The salaries right now are randomly generated, as I don’t have actual data to use since we’re not in football season. And I haven’t yet done much tuning of the parameters: changing the population size, number of itterations, mutation percent, etc.

Again, feel free to leave feedback on how this can be improved. I will probably continue to tweak and modify the algorithm so its ready come football season.

Crystal and Kemal

I’ve had my eye on the crystal programming language for a while now, and I have to say I’m pretty excited about it. Ruby syntax combined with the speed of a statically typed compiled language, common now. It’s compiler is now bootstrapped and benchmarks performances are impressive (yeah I know, benchmark scores don’t matter). I would suggest watching the video on the project’s own Bountysource page for a more in-depth look into the language itself. Here I plan on playing around mostly with the Kemal framework (seriously those benchmarks though).

benchmarks

There is already a decent collection of frameworks and libraries over at Awesome Crystal. Database drivers, a CSFML wrapper, 3rd party api libraries, etc, and amongst those is Kemal.

Most of my experince is with Ruby, specifically Ruby on Rails. I’ve also used both Sinatra and Padrino, and I enjoyed working with both of those. I’m going to try and build in a bit of structure to the project: models in a models folder, controllers in a controllers folder, etc. This will also be a single page application using AngularJS and Kemal as the JSON api.

beer

Before we get started, I just want to throw this out there. I’m double fisting New Belgium Ranger and a Mikes Harder Strawberry Lemonade because thats how I roll, gotta hit that Balmer Peak





I’m assuming you already have Crystal installed, if not visit their installation page follow the instructions per your platform. I’m using Crystal 0.13.0.

If you want to check out the source for this project, i’ve stuck it in this repo fridgerator/kemal_test.

Generating the project was easy enough, crystal has a built in project generator which creates a basic project structure for us crystal init app kemal_test. Then add the require dependencies to the shards.yml file (kemal, active_record, and postgres_adapter). Shards is a dependency manager similar to ruby gems and the shard.yml file looks like a cross between a Gemfile and a package.json file. Pretty slick, the crystal project is in alpha stage still and already 13x easier to set up than any node.js application.

For data / domain models, I’m using active_record.cr. I didn’t find anything akin to rails DB rake tasks or migrations, so I created the database and tables manually in the psql interface.

class Post < ActiveRecord::Model
  adapter postgres

  primary id        : Int
  field title       : String
  field body        : String

  def to_h
    {
      id: id,
      title: title,
      body: body
    }
  end
end

I didn’t use anything special for controllers, just Classes containing route definitions.

class KemalTest::Controllers::PostsController
  before_all "/posts" do |env|
    env.response.content_type = "application/json"
  end

  get "/posts" do |env|
    posts = Post.all.map(&.to_h)
    posts.to_json
  end

  ...
end

Kemal serves up static files out of /public, so I threw my main.js, main.css and angular template files in that directory.

So far I have everything I set out for: simple rails-like structure, ruby syntax, insane server response times (most were in the hundreds of microseconds). I’m 100% satisfied and will definitely continue to dink around with Crystal/Kemal in the future.

Goodbye Dropbox and Spotify, Hello ownCloud

I already pay $20 a month for two Spotify accounts (one for me and my wife) and now my dropbox is getting full.  Instead of dishing out the $30 a month I decided to check out other options first.  I came across this article at lifehacker.com which sparked my attention.  I did some more digging around and read quite a bit about ownCloud that sounded right up my alley other than the fact that its written in php... boooooo.

I also, had been previously looking into getting a VPS but could never think of a good enough use for one to justify the cost.  Backupsy.com is a VPS configured for higher storage options (base level is 250GB) AND you can get it for $5 a month.  For me this sounded like a couple of hours of work for a $25 a month savings and WAY more customization and options. 

Visit backupsy.com and sign up for their 250GB storage, 512mb ram 1,000GB bandwidth plan using coupon code "BIGDADDY" to get all this for $5 a month (50% off).  I opted for the automatic OS install but apparently their automatic OS install wasn't working correctly so I installed the OS myself (Ubuntu 12.04).

Once all this is finished, log into the box using the web console.  I had to use firefox because the Java applet needed to do this doesn't work in chrome.  Log in as the user you created during set up and install an ssh server with "sudo apt-get install openssh-server".  You might want to read up on securing an ssh server, and configure at least disabling root login via ssh.

Next lets get lamp, php and ownCloud installed following the instructions here.  When you're done you should have a working ownCloud install and an admin user set up.

I wanted to be able to sync my current ~/Dropbox folder, and also upload all of my digital music without syncing the music back to my computer.  Luckily ownCloud supports the option to only sync specific folders with remote ones.

Install the ownCloud sync client, and create a folder that you want to sync, I created an ownCloud folder in my home folder.  And instead of syncing data with the remote root folder, choose and create a new remote folder to sync with.  If you sync with the remote root folder everything will sync (music, photos) and you will only be able to create one sync.

You also might want to check out the ownCloud apps that are available to use.  I will probably end up creating an account for my wife on my ownCloud and installing the internal messages app for us to use.  And giving her an account will allow her to have her own storage and music service as well.

I've copied my entire dropbox folder into my ownCloud folder and I'm waiting for that to sync.  After that I'll begin uploading my music and testing out the streaming both to my laptop and my android.  As long as this all works out ok I'll be dropping both Spotify and my dropbox account for this.  Next I'll also be looking into enabling SSL and maybe developing my own addons / apps.