Use the Unix: One Command to „Activate“ Any…

In my day-to-day work as a software developer, I find myself moving into a bunch of different directories and running some repetitive commands to “activate” or “launch” the project in that directory.

Ideally I’d like to be able to type something like x-activate after I cd into a directory and have it just do The Right Thing™. Now, if the commands were the same each time I could just make a shell alias like (fictional example for illustration):

alias x-activate="export MODE=DEV && nvm use && grunt server"

… but alas, some projects are the same, some are different. But, with cleverness and determination, there is a nice hacky way we can get what we want…

A Contrived Example

Let’s say we have three projects in our ~/code directory, ringo, paul and george

$ ls ~/code
# => ringo paul george

We usually work on these projects all at once, by opening three separate terminal windows and cding into each directory, then running some command which starts the program.

ringo : nvm use && grunt server
paul : export MODE=DEV && make run
george : make compile && ./bin/run --port 9020

Awful right? Imagine how much worse this would be if our job involved over a dozen microservices, all like this? It would be so much nicer if we could just cd into each directory and type one command (in our example x-activate) and have The Right Thing™ just happen.

In this example we name our new command x-activate, with a leading x-, for the sake of differentiating it from existing commands and programs on the system. For example, python virtualenv tool uses a command called activate. It may (or may not) be a good idea to prefix your own hacky tools with a consistent prefix in general.

Step One: A Script in Every Folder

Let’s go into each of our project directories and create a file called .x-activate, containing the commands we usually run to start that project:

$ cd ~/code
$ cd ringo
$ echo 'nvm use && grunt server' > .x-activate
$ cd ../paul
$ echo 'export MODE=DEV && make run' > .x-activate
$ cd ../george
$ echo 'make compile && ./bin/run --port 9020'

Next, and presuming we use git for version control on our projects, we should add .x-activate to our global git-ignore file, because we don’t want any of these files being committed to our git repositories.

Adding `.x-activate` to the Global Git-ignore file

First, let’s check if we’ve already configured a global gitignore file:

$ git config --global --list | grep 'exclude'
# => core.excludesfile=/Users/shanek/.gitignore_global

Here we can see that on this machine, I’ve already done the following:

created a file called .gitignore_global in my home directory
told git to use that file for its core.excludesfile option

If the last command we ran produces a similar output for you, then you should open that file in your editor of choice and add the following line:

.x-activate

Otherwise, if the command produced no output, you should create the .gitignore_global file and tell git to use it:

$ echo '.x-activate' >> ~/.gitignore_global
$ git config --global core.excludesfile '~/.gitignore_global'

Easy, and now we don’t have to worry about polluting shared code repositories with our own convenient little hack.

Step Two: A Tiny Shell Alias

Now that each project has a little lump of shell-code tucked away in a file, we could start all of our projects by going to each project directory and using the source command to load the .x-activate file as if it were shell source-code:

$ source .x-activate

This is better than trying to remember the right commands for each project, but we can go one step further and create an alias which will shorten source .x-activate to just x-activate…

Let’s presume you’re using the bash shell, like most people these days. In most cases, bash will read in either, or both of the files called .bashrc and .bash_profile located in your home directory (if you’re not sure of the difference between these files, check out this StackOverflow question). Let’s presume that ~/.bashrc will be loaded whenever you open the shell. Open ~/.bashrc in your editor of choice and add the following line:

alias x-activate="source ./.x-activate"

Step Three: Shell Domination

Let’s open a new terminal window or tab, and try it out:

$ cd ~/code/ringo
$ x-activate
# => ... a bunch of output here ...

Nice, that’s exactly what we wanted. Let’s recap on what’s going on here

The shell receives the command x-activate
The shell recognises this as an alias for source ./.x-activate
The shell runs source ./.x-activate
The source command loads the contents of .x-activate and executes it as shell code
Things run, stuff happens

What Have We Learned?

Shell aliases are cool
We can put shell-code which is specific to a certain directory, in that directory
We can use unix-foo to save us time and keystrokes while hacking on our awesome projects

Let’s generate a super-secure random password (let’s say, for our tumblr account), using only the command line and a few basic unix tools.

First, we’ll read 10 bytes of random data out for /dev/random:

$ head -c 10 /dev/random
# -> �u#�ko�%

The output looks kinda shitty huh?

Ok, let’s encode this data in base64 format:

$ head -c 10 /dev/random | base64
# -> 9W0MVZQ+SC27VA==

Better, but those trailing ’=’ characters aren’t really useful to us, and that ’+’ in there reminds me that we should prefer to generate ‘url-safe’ base64 text.

Let’s use tr (translate) to delete (-d) the equals-signs:

$ head -c 10 /dev/random | base64 | tr -d '='
# -> PHCSXH7w3TZgHg

And let’s use tr again to change ’+’ into ’-’ and ’/’ into ’_’:

$ head -c 10 /dev/random | base64 | tr -d '=' | tr '+/' '-_'
# -> XE_TRFKrfv-nwA

Much better, but how many characters are in this password we are generating?

$ _my_password=$(
  head -c 10 /dev/random | base64 |
  tr -d '='| tr '+/' '-_'
)
$ echo -n "$_my_password" | wc -c
# -> 14

(note how we passed -n to echo, asking it to not print a trailing new-line)

Fourteen characters isn’t bad, but we can always get more by increasing the value of the -c parameter to head and get a longer password:

$ head -c 16 /dev/random | base64 | tr -d '=' | tr '+/' '-_'
# -> 94xKa4qk2tpclnL-OjV6Wg
$ head -c 22 /dev/random | base64 | tr -d '=' | tr '+/' '-_'
# -> L8V3Ee3TxyvEl88cOaIJ-SUWB3YCqg

Now we can just copy-paste this delicious new password into our browser and our account is secure again!

While reading up on BEM and other modern CSS methodologies I came across several articles which advised keeping nested CSS rules to a minimum, for the sake of minimizing specificity and also for the sanity of the developers.

I then got to thinking about CSS pre-processors such as LESS and SASS (both of which I’ve used), and that if you don’t use nesting with those pre-processors then the only (commonly used) features you’re left with are variables, mixins, file inclusion and convenience functions.

So, I thought, could skip LESS/SASS and just use the simple (and ancient) macro processor known as GNU m4?

Why m4?

GNU m4 is a macro processor, it’s whole job is to process text files with macros defined in them. A few advantages of m4 are:

it’s available on basically every Unix system, without installing any extra dependencies
it’s not specific to css, or any other language, it could be used on any old text file.
if you’re not already sold on something like LESS or SASS, then m4 could be a really powerful tool to help make your CSS cleaner without adding another heavy dependency to your build-pipeline.

And some of the disadvantages:

being an older unix tool, the syntax is pretty clunky
~~m4 basically doesn’t work with Unicode, or any variable-width text encodings~~ This turns out to be wrong, I’ve tried using m4 with a bunch of non-ascii text in UTF-8 and it appears to work just fine.
because m4 isn’t tied to any particular language, it doesn’t have any language-specific features, such as SASS’s ability to process nested CSS declarations.

Quick intro to m4 language

GNU m4 is all about defining macros in a text file, which can then be used later in the file to generate more text. We call m4 like so:

$ m4 some-file

In this case m4 will read in some-file and process any m4-specific directives found in the text, emiting the processed text to standard-out. In our example we’re going to do some preprocessing on a CSS file called style.css.m4, so we’re going to call m4 like so:

$ m4 -P style.css.m4 > style.css

The -P (or --prefix) flag instructs m4 to prefix it’s own built-in functions with m4_. This is useful because it makes collisions between m4 and our own text less likely. Without -P, we would use the define() function to define a macro, but with -P we use m4_define(). You can see why the latter would be preferable. From here on I’m going to presume we’re using the -P flag to m4.

How to define macros

In the simple case we can just define a simple text-substitution like so:

m4_define(FOO, bar)

We can then use the FOO macro later in the file, to splice in the text bar at that location:

I want to go to the FOO and have an apple-juice.

When m4 is run on this file, the resulting text will be:

I want to go to the bar and have an apple-juice.

We can also define macros which take parameters at call-time, but we’ll get to those later.

m4’s weird quoting

Probably the weirdest part of m4 is it’s quoting rules. Basically, instead of using ordinary ‘quotes’ and “double-quotes”, m4 uses the backtick (`) as the opening quote character, and the single-quote (‚) as the closing quote character. So, We could write the FOO macro example like this:

m4_define(`FOO', `bar')

Kinda weird, but whatever.

Anyway, that’s enough for us to get started with marco-ising our CSS, so let’s get on with it.

Setup

You’ll need the m4 program installed. Mac OSX ships with version 1.4.6 currently, while version 1.4.17 is available in homebrew. For our purposes the pre-installed version will do just fine. On ubuntu m4 can be installed with apt-get install m4. If you’re on Windows, IDK, it’s probably possible to install m4 somehow.

Let’s make a new directory and create a file in there called style.css.m4:

$ mkdir m4-example
$ cd m4-example
$ touch style.css.m4

The .m4 extension isn’t required, but I think it looks nice. Let’s invoke m4 on this (empty) file:

$ m4 -P style.css.m4

Basic Variable Substitution

This invocation of m4 won’t output anything useful, because there’s nothing useful in the file. Let’s fix that by opening our favourite editor and entering the following:

m4_changecom(`@@##')
m4_define(BLACK, #000)
m4_define(GREY,  #ccc)
m4_define(WHITE, #fff)

body {
  background-color: WHITE;
  color: BLACK;
}

On the first line, we change the m4 comment character to be @@###, rather than #. This is a good idea because # is used all over the place in CSS, so we’d rather not have it be interpreted as an m4 comment, and @@## is a suitably obscure alternative.

The next three lines define three macros, BLACK, GREY, and WHITE. From now on, any time those words occur in the file they will be replaced with the appropriate color hash values. We are using UPPER_CASE identifiers for our macros, but bear in mind you should alwoys choose some kind of naming scheme which is not going to conflict with legitimate content in the files you are processing. Use your head.

Of course, we aren’t limited to defining macros for basic color values, we can use any text we want, but for most web developers the most useful values to put in these macros will be color and numeric values. Note also that we didn’t bother to quote either the macro names, nor their values, but we could easily have wrapped these in m4 quotes likes so:

m4_define(`BLACK', `#000')

If we run m4 again, the output should be something like:



body {
  background-color: #fff;
  color: #000;
}

Yes, there’s a bunch of whitespace in there, but the text we care about (the CSS declarations) have been processed and the correct color values have been spliced into the text.

If the extra whitespace is annoying, you can add the m4_dnl directive to the end of the m4_define... lines, which will delete the extra whitespace up to the next new-line, like this:

m4_define(BLACK, #000)m4_dnl

In this tutorial we won’t bother with that, for the sake of clarity. Plus, if you’re running the resulting CSS through a minifier the extra whitespace shouldn’t be an issue.

Including other files

So, now that we can define variables in our CSS code, the next feature we probably care about is the ability to split our CSS over multiple files and then import those files into our main stylesheet. In m4 we can do this with the m4_include directive, like so:

m4_include(./other_file)

Let’s imagine we want to keep all the styles related to our site footer in a separate file, say footer.css.m4:

m4_define(FOOTER_TEXT_COLOR, #222)

.footer {
  color: FOOTER_TEXT_COLOR;
}

We can then include that file in our main file with:

m4_include(./footer.css.m4)

… and the contents of footer.css.m4 will be processed by m4 and spliced into the output text stream. Our example file style.css.m4 now looks something like this:

m4_changecom(`@@##')
m4_define(BLACK, #000)
m4_define(GREY,  #ccc)
m4_define(WHITE, #fff)

body {
  background-color: WHITE;
  color: BLACK;
}

m4_include(./footer.css.m4)

Simulating mixins

Both SASS and LESS support “mixins”, which essentially allow you to create a block of CSS code which can be included in some other block of CSS code with a simple one-liner. In m4 we can achieve the same effect with good-old macros:

m4_define(ANGRY_TEXT, `
    color: red;
    font-weight: bold;')

p.angry {
  overflow: auto;
  ANGRY_TEXT
}

But we can go one step further and define a macro which accepts parameters at the call-site, allowing you to re-use blocks of code almost like functions in a real programming language. Let’s define a macro which will handle all the weirdness of adding a border-radius to a DOM element:

m4_define(BORDER_RADIUS,
  `-webkit-border-radius: $1;
  -moz-border-radius: $1;
  -ms-border-radius: $1;
  border-radius: $1;')

In this example, the $1 stands for the first parameter to the macro. If we use the macro like this: BORDER_RADIUS(6), then the number 6 will be bound to the $1 and processing will continue as you’d expect it to. If we had more than one parameter, then $2, $3 etc would be available to use in the macro.

Let’s add both of these examples to our style.css.m4 file:

m4_changecom(`@@##')
m4_define(BLACK, #000)
m4_define(GREY,  #ccc)
m4_define(WHITE, #fff)

m4_define(ANGRY_TEXT, `
    color: red;
    font-weight: bold;')

m4_define(BORDER_RADIUS,
  `-webkit-border-radius: $1;
  -moz-border-radius: $1;
  -ms-border-radius: $1;
  border-radius: $1;')

body {
  background-color: WHITE;
  color: BLACK;
}

p.angry {
  overflow: auto;
  ANGRY_TEXT
}

pre.formatted-code {
    font-family: monospace;
    background-color: GREY;
    BORDER_RADIUS(6)
}

m4_include(./footer.css.m4)

Conditionals

GNU m4 has a m4_ifdef directive, which allows you to conditionally emit some text:

m4_ifdef(SOME_TEST, `yes')

I think this would only be useful in conjunction with the -D (or --define) flag to the m4 program. Usind -D you can create a definition when the program runs, which is basically equivalent to passing variable definitions into m4.

Consider the following example:

$ m4 -P -D DEBUG=true some_file.m4

In this case you could use m4_ifdef(DEBUG, some_text) to only include some_text when you’re compiling in “debug mode”.

Doing math

One more cool feature: m4 can do basic math by using the m4_eval() directive. For example:

m4_eval(2 + 6)

… will emit “8”.

If we wanted to do some math on our border-radius declarations, we could do something like this:

pre.formatted-code {
  background-color: GREY;
  BORDER_RADIUS(m4_eval(22 / 2));
}

In fact, we could save ourselves some effort and define a macro for our main border-radius number, and then do math using m4_eval whenever we want to use a smaller or larger radius. Here’s a contrived example:

m4_define(BORDER_RADIUS_AMOUNT, 22)

.normal-radius {
  BORDER_RADIUS(BORDER_RADIUS_AMOUNT)
}

.smaller-radius {
  BORDER_RADIUS(m4_eval(BORDER_RADIUS_AMOUNT / 2))
}

.larger-radius {
  BORDER_RADIUS(m4_eval(BORDER_RADIUS_AMOUNT * 2))
}

Why manuel?

Manuel began as something of a joke, a rather snarky jab at gruntjs, which had been getting on my nerves for some time. Without getting too far into it; I was beginning to feel that grunt was an awkward and possibly un-necessary layer of indirection over the basic notion of scripting simple tasks.

At work we use grunt extensively to automate our web-app projects, building our coffee-script and sass assets, starting our development servers and so on. After a particularly aggravating session of trying to debug one of our grunt tasks I came to the realization that grunt was really just giving us a JavaScript interface over the shell, with several disadvantages:

Grunt requires an exceedingly flaky external dependency (npm) to work at all
When something goes wrong in a grunt task it can be difficult (or near impossible) to debug
If the thing you want to do is not already covered by one of the (admittidly rather good) grunt plugins; then you end up needing to shell-out anyways

With all this in mind, I wondered if it might be possible to recreate at least some of the good parts of grunt using nothing but plain old bash and maybe solve some of these problems.

After a few days of learning about bash and thrashing out code I had discovered that yes, it is possible to implement all the useful aspects of grunt in bash, and that the resulting tool may actually be useful!

Results

Essentially, manuel is a small program which provides some of the key functionality from grunt:

loading “tasks” from a file in the current directory
automatically loading tasks from “plugins”
running those “tasks” on demand

All of this can be achieved in just over 100 lines of bash. Loading the contents of a manuelfile is trivial, and the same goes for loading plugins. Beyond that it’s just a matter of writing bash functions and stringing them together to get things done.

We can get a surprising amount of value from even this small amount of code.

With that done, the next natural step was to write some plugins to replicate the functionality I would be missing from grunt. The most obvious candidates were grunt-contrib-watch and grunt-contrib-concurrent, which I slavishly replicated as manuel-contrib-watch and manuel-contrib-concurrent. While these plugins actually took a lot more time to develop than manuel itself, they also proved to be the most enlightening part of the whole process, highlighting the good, the bad and the downright hideous of working with bash as a programming language.

The Good

Bash is great at being a seamless interface to the system. If a developer knows how to achieve something by typing it into the terminal, they are already most of the way towards writing a manuel task.
Job control is built right into the shell, so running several tasks in parallel and waiting for them to finish is pretty easy.
Bash is a surprisingly complete programming language, good enough to be useful for most (though certainly not all) cases where one would use grunt.
Being able to write reasonably well structured scripts without introducing a tonne of new dependecies is a bonus.

The Bad

Bash functions actually suck pretty hard.
Bash has a major problem with data-structures. Arrays are one-dimensional and cannot be passed around as first-class values. Maps (associative arrays) only appeared in version 4 and suffer from the same limitations.
Mac OSX ships with a positively ancient version of bash. Seeing as bash 4 was realeased about six years ago, this feels particularly bad. In practice this means that if your team (or audience) is a mix of OSX and Linux users, then you either need to advise the Mac users to update to bash 4, or completely avoid using new features like associative arrays.

Conclusions

Honestly, I am still torn on this. On the one hand, manuel could be a genuinely useful tool for many developers. On the other hand, the limitations of bash’s data model quickly leave one wishing for something closer to a “real” programming language.

However, despite the problems with bash, I find it to be remarkably powerful, so much so that I would recommend all developers to make an honest attempt at learning the shell, instead of just opting for the safe default of “but bash is ugly/hard/confusing!”.

I’ve certainly learned much more by challenging my own assumptions than by sticking to them.

Hammer version 4.0.0 has been released.

Hammer is a rate-limiter for Elixir with pluggable backends.

The main feature of this version is a new worker-pool implementation for backends (thanks to poolboy), which improves throughput, and allows for multiple instances of the same backend type.

In local stress tests, throughput improved by about 20%. Regardless, the new architecture no longer pushes all limit-checks through a single process, which is strictly A Good Thing™.

The API is the same as before, but the changes to the Backend api necessitate a full version bump.

To get started with Hammer, read the Tutorial, or check out the project on Github.

I enjoyed Douglas Crockfords talk on “Javascript, the Better Parts”, particularly the middle section where Douglas describes all the things he has stopped doing in Javascript, and gives an insight into the style he uses day to day.

This-less programming

In particular, I found his avoidance of the this keyword interesting. Crockford advocates for using constructor functions which return frozen objects, forming a closure over all the methods in the object. In the spirit of the classic contrived OOP examples, lets consider a construcor function for a Dog object:

var Dog = function(spec) {

    var sound = spec['sound'];

    var bark = function() {
        return sound + "!!";
    };

    return Object.freeze({
        bark: bark
    });
};

Notably, this style of OOP completely avoids using the this keyword to refer to any other part of the object, everything is closed over by the functions on the resulting frozen object.

One nice side-effect of this is that an instance of an object built by this constructor cannot be interfered with:

var pup = Dog({sound: "woof"});

pup.bark()  
// => "woof!!"

pup.bark = function() { return "meow"; };

pup.bark()  
// => "woof!!"

In Clojure

Being much more fond of Clojure than Javascript, I opened a lein repl and figured out the equivalent Clojure code:

(defn Dog [spec]
  (let [sound (:sound spec)
        bark (fn [] (str sound "!!"))]
    {:bark bark})

(def pup (Dog {:sound "woof"}))

(:bark pup)
;; => "woof!""

At this point I realized that this pattern seemed kind of like what I had previously read about Clojure Protocols, a topic which I had not fully understood before. I lookup up clojuredocs.org to read up on Protocols, and came up with the following solution:

(defprotocol Barking
  (bark [self]))

(defrecord Dog [sound]
  Barking
  (bark [self] (str (:sound self) "!!"))

Once the Barking protocol and the Dog record are defined, they can be used like so:

(def pup (->Dog "woof"))

(bark pup)
;; => "woof!!"

Which I found to be most pleasing.

Conclusion

Lets compare the two solutions, in Javascript:

var Dog = function(spec) {

    var sound = spec['sound'];

    var bark = function() {
        return sound + "!!!";
    };

    return Object.freeze({
        bark: bark
    });
};

And in Clojure:

(defprotocol Barking
  (bark [self]))

(defrecord Dog [sound]
  Barking
  (bark [self] (str (:sound self) "!!"))

The Clojure solution is obviously quite a bit shorter than the Javascript solution. It also more clearly indicates what is happening, there is a Barkingprotocol, consisting of a single function/method, called bark, and the Dogtype implements this function, using its own sound property.

So Clojure solutions can be more concise and readable than the equivalent Javascript? Sure, we knew that aleady, but that’s not what I found interesting. For me personally this was the light-bulb moment which demonstrated why I would ever want to think about using a Protocol over a plain old data-generating function in Clojure.

Most web-apps in production probably need some kind of rate-limiting, to prevent rough clients from taking down the app by flooding it with expensive http requests.

In this article we’ll see how to add rate-limiting to an Elixir/Phoenix application, using the Hammer package. We will also see how to switch the hammer backend so that it stores it’s data in Redis.

As an example, we’ll use a simple Phoenix application, with one http route (/timestamp), which will serve up a json response containing the current time. We will then demonstrate how easy it is to add rate-limiting to protect the application from greedy clients.

Why Hammer?

Hammer is a rate-limiter for Elixir, with a pluggable backend system, so the rate-limiting data can be stored wherever you want. Currently Hammer supports backends for ETS (in-memory) and Redis, with more in development.

While ETS is fine for small deployments, or when you’re only using one server anyway, it makes sense to move to Redis when you have more than one application server, so that the rate-limiting counters don’t get reset if a machine goes down.

The Phoenix project

Let’s create a new Phonenix project, called timely:

mix phx.new --no-ecto timely

Once that’s all set up, we’ll open the router.ex file and add a route for /timestamp:

scope "/", Timely.Web do
  pipe_through :browser

  get "/", PageController, :index
  get "/timestamp", PageController, :get_timestamp
end

Now, let’s add the get_timestamp function to page_controller.ex:

defmodule Timely.Web.PageController do
  use Timely.Web, :controller
  import Logger

  def index(conn, _params) do
    render conn, "index.html"
  end

  def get_timestamp(conn, _params) do
    Logger.log(:info, "Generating timestamp")
    now = DateTime.utc_now()
    conn |> json(%{timestamp: "#{now}"})
  end
end

If we start the server with mix phx.server, we should be able to hit that endpoint and get the current time:

curl http://localhost:4000/timestamp
# => {"timestamp":"2017-10-07 17:08:46.546596Z"}

Now let’s add the Hammer package to the dependency list in mix.exs:

defp deps do
  # ...
  {:hammer, "~> 2.0.0"},
  # ...
end

And run mix deps.get to install the new dependency.

Once that’s done, we can update page_controller.ex to add rate-limiting to the get_timestamp route:

defmodule Timely.Web.PageController do
  use Timely.Web, :controller
  import Logger

  def index(conn, _params) do
    render conn, "index.html"
  end

  def get_timestamp(conn, _params) do
    # Get a (semi) unique identifier for this user
    ip = conn.remote_ip
      |> Tuple.to_list
      |> Enum.join(".")
    # Do the rate-limit check, permit five hits per minute
    case Hammer.check_rate("get_timestamp:#{ip}", 60_000, 5) do
      # Action is allowed
      {:allow, _count} ->
        Logger.log(:info, "Rate-Limit ok, generating timestamp")
        now = DateTime.utc_now()
        conn |> json(%{timestamp: "#{now}"})
      # Action is denied, because the rate-limit has been exceeded
      # for this time-period
      {:deny, _} ->
        Logger.log(:info, "Rate-Limit exceeded, denying request")
        conn |> send_resp(429, "Too many requests")
    end
  end
end

Here we use Hammer.check_rate/3 to check if this ip address has gone over the limit (five hits per minute). If they’re under the limit then we generate the timestamp and send the response as before, but if they’re over the limit then we send back a 429 response instead.

We can test this out in the shell by hitting the /timestamp route a bunch of times in quick succession:

for i in {1..10}
do
  curl -s http://localhost:4000/timestamp && echo ''
done

# => {"timestamp":"2017-10-07 17:19:20.943411Z"}
# => {"timestamp":"2017-10-07 17:19:20.992869Z"}
# => {"timestamp":"2017-10-07 17:19:21.036328Z"}
# => {"timestamp":"2017-10-07 17:19:21.088144Z"}
# => {"timestamp":"2017-10-07 17:19:21.136055Z"}
# => Too many requests
# => Too many requests
# => Too many requests
# => Too many requests
# => Too many requests

This is exactly what we want to happen. If we wait for a minute before trying again we’ll see another batch of timestamps come through.

Switching to Redis

Because of Hammer’s pluggable backend system, switching to Redis is easy. We’ll presume you have a Redis server running locally on the usual port.

First, let’s add the hammer_backend_redis package to mix.exs:

defp deps do
  # ...
  {:hammer, "~> 2.0.0"},
  {:hammer_backend_redis, "~> 2.0.0"},
  # ...
end

And run mix deps.get again.

Now, open config/dev.exs, and add the following lines:

config :hammer,
  backend: {Hammer.Backend.Redis, [expiry_ms: 60_000 * 10,
                                   redix_config: []]}

This tells hammer to use the Redis backend (which we just installed), with a specific expiry time configured, and default options for the redis connection.

Don’t worry too much about those for now, you can always read the documentation later.

Now, restart the server, you should see a log line that says:

[info] Starting Hammer with backend 'Elixir.Hammer.Backend.Redis'

… which indicates we’re using Redis for storage.

Try hitting the /timestamp route again, and everything should still work.

curl http://localhost:4000/timestamp
# => {"timestamp":"2017-10-07 17:08:46.546596Z"}

To verify that our data is being stored in redis, use redis-cli to list all the keys starting with Hammer:

redis-cli keys 'Hammer*'
1) "Hammer:Redis:Buckets:get_timestamp:127.0.0.1"
2) "Hammer:Redis:get_timestamp:127.0.0.1:25123300"

And that’s it! We’ve now got fully functional rate-limiting in our Phoenix application.

Query Operators

In BedquiltDB v1, queries could be expressed in terms of JSON “query documents”, which would be used to match against the contents of a collection. For example, the following query would return all articles whose authorId is "abcd" and with upvotes equal to 4:

articles.find({'upvotes': 4, 'authorId': 'abcd'})

Which is fine, but what if we want to get all documents with more than 4 upvotes, or where the status is not equal to "open"? In short, those queries were not possible without dropping down to the SQL layer.

In BedquiltDB v2 however, these queries are expressible with the use of Advanced Query Operators. These queries generally take the form:

{"someField": {: }}

… where operator is a string beginning with $, and the argument is some value that has meaning to the operator.

The following operators are supported:

$eq: asserts that some field is equal to some value
$noteq: asserts that some field is not equal to some value
$in: asserts that the fields value is a member of the supplied list of values
$notin: asserts that the fields value is not a member of the supplied list of values
$gt: asserts that the field is greater than some numeric value
$gte:asserts that the field is greater than or equal to some numeric value
$lt: asserts that the field is less than some numeric value
$lte: asserts that the field is less than or equal to some numeric value
$exists: if the supplied values is truthy, asserts that the field exists, otherwise asserts that the field does not exist
$type: asserts the type of a field value, valid arguments are “object”, “string”, “boolean”, “number”, “array” and “null”
$like: asserts that the field value is “like” the supplied match string, following the semantics of PostgreSQL LIKE operation.
$regex: asserts that the field value matches the supplied regex string, following the semantics of PostgreSQL ~ operation.

These operators can be mixed and matched freely within a query document, and the query will do The Right Thing™.

Examples:

collection.find({
    'upvotes': {
        '$gte': 4,
        '$lt':  10
    },
    'authorId': 'abcd',
    'metadata': {
        '$exists': True
    }
})

collection.find({
    "title": {
        "$regex": "^.*Elixir.*$"
    }
})

collection.find({
    "city": {
        "$notin": ["London", "Glasgow"]
    }
})

See the “Advanced” Query Operators section of the BedquiltDB Spec for full documentation.

Other Cool Stuff

You can now sort by the collections $created and $updated times:

# returns documents in order of the time they were created
collection.find({...}, sort=[{'$created': 1}])

# returns documents in order of the time they were updated
collection.find({...}, sort=[{'$updated': 1}])

The find_one operation now accepts sort and skip parameters, which behave the same as those for find:

collection.find_one({...}, sort=[{'upvotes': -1}], skip=2)

And there’s a delightful new remove_many_by_ids operation:

collection.remove_many_by_ids(['one', 'four', 'nine'])

BedquiltDB v2 is available to install right now, see the Installation section of the BedquiltDB Guide for full insturctions.

And of course, the source-code can be found, forked and contributed-to on Github.

Happy Hacking!

See this in the app Show more

Use the Unix: One Command to „Activate“ Any…