JSON Schema and CouchDB bulk import performance 15 May 2012
Originally published on
05 April 2011 on the
DevelopmentSeed intranet.
This project launched successfully, check out the release announcement for more info.
I have revised and double checked this article before publishing, but I am able to re-run the benchmarks.
During the last week I have been responsible for fixing a number of critical issues on the FEBP project to get it ready for the client to be able to work with it. One of the areas I ended up needing to spend a lot of time on was improving performance related to data import and validation.
Background
On the FEBP site, the administrator is able to upload a CSV file describing the data for each of the data sets, which then changes how we interpret and display this data, and they can upload new versions of the data itself. Each dataset and schema is versioned, and the site has the concept of an “active version”.
When the admin uploads a new version of the data, once it is imported they are required to validate it against a schema, and are able to preview the new...
Read more →
Dynamic Form Generation with JSON Schema. 06 May 2012
Originally published on
05 December 2010 on the
DevelopmentSeed intranet.
I have taken the time to revise and double check the information contained within it.
Foreword Added on 06 May 2012
This is one of the earliest posts I wrote while learning to use Node.JS. It was written during a phase where I was still trying to turn Node into Drupal. It is also one of the first times I realized that trying to do so was a mistake.
You simply do not need to have a system that automagically generates forms based on a declaritive control structures. In my Drupal days, one of my largest contributions was the Drupal Forms API that worked on similar principles, so this was a very difficult lesson for me to learn.
While I dont think what I was trying to accomplish in this article is the correct approach, the technology I was researching to help me solve it is actually really powerful and useful.
Other people have also realized that schemas could be used to generate forms:
Whenever you find yourself having to come up with a format to declare anything in JSON Schema, wether it be how a config file is structured or some other problem, I urge you to take a look to see if there isn’t already an agreed upon way...
Read more →
Elasticsearch - 5 minute search integration 01 May 2012
Originally published on
15 March 2011 on the
DevelopmentSeed intranet.
I have refactored the example code and made it less client specfic, and supporting the latest versions of node and ES.
One of the lingering tickets on the FEBP project is building full text search for the documents database, and this is the simplest way I have discovered to implement something like this.
Elastisearch is to Solr what CouchDB is to MySQL. It is marvelously simple to get going. You download a file, run a command and then you can index JSON with it via Curl.
Another great thing is that every time it starts up it seems to give itself a new name. The first time I played with it, it dubbed itself Algrim the Strong. It’s not often I run across software that makes me smile just from using it.
Read more →
Building my first Jekyll site. 23 April 2012
Originally published on
20 October 2011 on the
DevelopmentSeed intranet.
I have taken the time to revise and double check the information contained within it.
For the last 2 weeks I have been working on the new site for Global Adaptation Index, to bring it in line with the index data site we built. We chose to use jekyll and github to built the main brunt of the site, which mostly involved ‘about’ pages and team profiles.
The experience has been interesting, and relatively painless. It has been a while since I have had to build something so straight forward and from scratch. I can’t even remember the last time I had to build all the markup and styles for a site, as usually I only become involved once the structure is complete and we just need to wire up the functionality to it.
Read more →
