00null — Manuel: How and Why

00null — Manuel: How and Why

Problem: sometimes you want to make some document or resource available for download (ex: a whitepaper, brochure, report), but only after the downloader has provided their email address, read some T&Cs and clicked “Accept”.

Setting this up could be a pain if you’re part of a non-technical organisation. Ideally you’d want a service to handle it for you, which you can link to from your website.

Solution: a web app which allows a user to upload a document or other file, and “clickwrap” it. A visitor can then be directed to the URL of the clickwrap’d file where they will be shown the “agreement” text, prompted to fill in their email and click “accept” before being given the file.

Monetisation:

  • provide custom CSS and branding on premium accounts
  • better, deeper metrics on premium accounts
  • maybe restrict to a two-month free trial on free accounts

This past month I’ve been getting re-acquainted with PostgreSQL. After about a year of learning and working with MongoDB, playing with PostgreSQL has been a refreshing change and a chance to brush up on old skills.

One thing I stumbled was how to generate random (or at least random looking) strings, suitable for use as obfuscated unique identifiers. Of course in MongoDB-land one would just use whatever pseudo-random functions are available in the software layer to generate the identifier or slug before sending it to the database. While this would certainly work with PostgreSQL, it doesn’t seem to fit the RDBMS mindset, something about it just feels wrong.

Now Postgres does have the usual auto-incrementing serial integer fields, but I feel that in some cases an incrementing number is not the best choice. After a bit of googling around I saw a few answers on Stack Overflow to similar questions. The solutions seemed quite convoluted, selecting from an array of strings denoting the alpha-numeric character set and so forth. All the solutions I saw just seemed clunky and too much work for a simple request.

Then I found the pgcrypto extension, which offers some nice functions like genrandombytes() and crypt(). The first solution I arrived at looked something like this:

select encode(gen_random_bytes(8), 'hex');

Which returns nice hex strings like “7d95786f7e8038b5”, however I would prefer to use the full alpha-numeric character range, to maximise the amount of information per character in the string. This is pretty important when you want to avoid collisions between strings and still have them be short enough to type out by hand.

Switching the algorith to blowfish yielded some better results:

select crypt(cast(gen_random_bytes(32) as text), 'bf');

Now the resulting strings looked something like “bfRhoBBOZWG/Y”, Which is an improvement, but the ’/’ character is not good for including in urls. the “base64” type also produces similar problems.

The solution I eventually settled on involves replacing the problematic characters with ’-’ before producing the result:

SELECT CAST(  
  regexp_replace(
    encode(gen_random_bytes(6), 'base64'),
    '[/=+]',
    '-', 'g'
  ) AS text
);

And the same query wrapped up in a nice function:

CREATE OR REPLACE FUNCTION random_slug()  
RETURNS text  
AS  
$$ SELECT CAST(
     regexp_replace(
       encode(
         gen_random_bytes(6), 'base64'),
         '[/=+]',
         '-', 'g'
   ) AS text);$$
LANGUAGE SQL

This solution generates some very nice strings:

"LFN0H5k4"  
"dZCAfwU5"  
"9z1g-k-a"  
"QJ1vJO-j"  
...

I have no doubt this solution is not perfect but it’s good enough for those times when you want to generate a random-ish sequence of characters for non-sequential ids.

I’ve been hacking around in Clojure for a few years now, and while I like the language very much there are a few quirks of the implementation which I still find irritating, namely the glacial startup time and general un-suitability to writing small programs.

So this week I’ve been looking at Common Lisp, in search of another nice lisp dialect to add to my tool-box.

Some general impressions:

  • roswell is pretty nice, making it easy to get sbcl, quicklisp and asdf installed
  • Practical Common Lisp is a good book, very easy to read and gets right to the point without trying to explain what programming is from the ground up
  • SBCL seems fine, I’ve not had any trouble with it yet
  • The ecosystem seems to be plenty diverse, with robust implementations of all the things I’d care about
  • The language can be a bit weird:
  • To a beginner it’s not clear why getf, setq and co are named the way they are
  • Using higher-order functions can feel a bit clunky, using #'foo and (funcall foo x y z) feels much less clean than the equivalent code in Clojure. More than once I’ve stumbled on what to do when returning a lambda from a function, binding it to a var and then passing it along to another function
  • Common Lisp doesn’t seem to have a fundamental sequence abstraction like Clojure, so it’s a lottery trying to figure out which functions will work on whichever data-type you’ve got in hand.
  • Verbosity seems like it’s going to be a problem, function names are usually pretty long and there are no data-literals in the style of Clojures {}, [] and #{}
  • The Common Lisp Wiki looks like a nice source of up-to-date information on Common Lisp
  • Common Lisp makes uses the empty list ((), or nil) to denote boolean false, which I could see causing all sorts of problems when parsing formats like JSON. Actually, now that I think about it, this makes working with any external data basically untenable, the language simply doesn’t have a concept of empty-list, null and false being different things.

I think I’ll keep going with this one, despite some flaws Common Lisp does look like a promising language and a good candidate for small, compiled programs and little network services.

2015 was a great year for me. In those twelve months I:

  • Developed BedquiltDB into a viable project
  • Saw NightChamber grow into a small but vibrant community
  • Joined the team at ShareLaTeX
  • Learned a bunch of stuff
  • Most importantly of all, continued to foster wonderful personal relationships
  • Perl
  • ClojureScript
  • Elixir
  • And a bunch of others

Let’s hope 2016 is just as good.

Inspired by a post on Hacker News: create a new competitor to GitHub, but with more focus on the social aspect of “Social Coding”.

  • Organisations
  • Projects/Repos
  • Reddit-like discussion threads attached to things like issues, branches, commits and pull-requests
  • Easy hyperlinking between discussions, issues, commits, etc
  • Obviate the need for OSS projects to host their discussions/community elsewhere.
  • Also addresses the fact that some projects try to use Github issues as if they were forum topics
  • Ideally the implementation should be open-source, and with the business providing a hosted solution

Why? Someone raised a point that Github, despite it’s motto of “Social Coding”, has been more focused on chasing Enterprise contracts than on developing a truly social platform for code. The potential for a code hosting platform with real social features is huge.

Today I released v0.4.0 of BedquiltDB. Changes include:

  • New skip, limit and sort options to find queries
  • node-bedquilt now uses ES6 features, and thus requires node >= 4.0.0
  • Improved documentation, including a new BedquiltDB Guide

A lot has happened for me in the last few months.
Some highlights:

New Job

In August I joined the team at sharelatex.com. We’re working on a new product called DataJoy. It’s awesome.

Lots of other stuff

Which I can’t really remember, or don’t feel like talking about online.

This morning I pushed a set of significant changes to the various projects which make up BedquiltDB. With the addition of schema constraints I feel this is the right time to formally announce the projects existence.

What is BedquiltDB?

BedquiltDB is a json document store built on top of PostgreSQL. If you’ve ever wanted a nice wrapper around PostgreSQL that makes it feel like MongoDB but with better write guarantees, then BedquiltDB is what you’re looking for.

How does it work?

There are two major components to BedquiltDB:

  • The bedquilt postgres extension
  • The client libraries, written in your favourite language

The Getting Started Guide describes the process for getting these two parts of the puzzle in place.

In short, the bedquilt postgres extension contains all the “smarts” of the system, exposing a set of SQL functions as its API, while the client libraries provide a thin wrapper around those SQL functions.

For example, the following python code:

people.insert(  
    {'name': 'Sarah',
     'age': 45})

Would translate to the following SQL call:

SELECT bq_insert(  
    'people',
    '{"name": "Sarah", "age": 45}'::jsonb
);

This means that all the clever stuff like automatically generating _id fields, checking schema constraints and so on can happen in the database, where it belongs. It also means that writing new client drivers for BedquiltDB is pretty straightforward, as there is no real logic on the client-side.

What’s next?

Now that BedquiltDB has a decent feature set and at least two different client implementations, my next goal is to introduce more advanced query operators. At the moment the client can only query on basic field matches, like so:

result = people.find({  
    'location': 'Edinburgh',
    'age': 24})

It would be much more useful to be able to run queries like this (borrowing a little from the MongoDB API):

result = people.find({  
    'location': 'Edinburgh',
    'age': {'$gt': 20, '$lt': 30})

What’s with the name?

Go play Kentucky Route Zero, all will become clear.

(Reposted from the main Nightchamber blog)

Just over a week ago I set up the nightchamber.com server, posted my first thought and stared at the screen. “Well, it’s live” I thought to myself.

There is some satisfaction to be had in bringing an idea to fruition and seeing it actually work on screen. But nightchamber was supposed to be a place full of thoughts, not just my own, but other peoples thoughts. I knew I had to show it to someone.

I posted the link on Hacker News (https://news.ycombinator.com/item?id=8881622) late on Tuesday evening. I kept an eye on the process stats on the live server and waited.

After a few minutes the database process blipped slightly. I queried the “users” collection and the number that came back was one higher than it had been before. I ran it again, now it was ten higher, ten again, then over twenty.

As the evening rolled on thousands of people followed the link to look at my site. A few even left comments on the Hacker News thread. Some liked it, some didn’t, but that was fine, I feel like I don’t really get it either yet. I stayed up late chatting with a friend about how it was strange to have 16000 people interact with something you’ve made.

Two days in the torrent of activity slowed to a steady stream of visitors and new thoughts. It’s fun to look at some of the thoughts as they sit in the database. Some people write poetry, others comment on the state of the world. Someone posted an recipe for eggnog, which I have yet to try.

I laughed at quite a few of these strangers thoughts. One of them broke my heart to read. I wish I could reach out to that person.

Wherever this goes, I’m glad to have felt the presence of these people, faceless and distant as they are. If the site brings even one person any measure of joy or comfort then it will be a shining success.