Articles // Developing With Style

Dynamic Form and Dynamic Errors for Rails 3

Article posted by Joel Moss on 09 Aug 2010 |

Rails 3 is awesome! BUT...

For some unknown reason, the powers that be decided to remove the error_message_on and error_messages_for form helpers. I'm not sure about every one else, but I use these in pretty much every single form that I create, so I just have no idea why it was removed.

Fortunately, all is not lost. The code for these helpers was extracted out into a plugin called dynamic_form. But unfortunately it doesn't seem to have been kept up to date, as the last commit was back on June 12th. Running the tests failed all over the place, simply because the HTML returned has changed slightly in the release candidate. So I took it upon myself to fork the code and managed to get the tests passing with Rails 3 RC. I then also packaged it up into a Gem which you can grab now and install in your Rails 3 app.

My next task was to get the custom-err-msg plugin working with Rails 3. But rather than fix and refactor that as it is, I decided to simply integrate it into the dynamic_form plugin, as it's functionality is closely related.

So the dynamic_form plugin has been enhanced a little, so please install and let me know what you think. I've put in a pull request to the official dynamic_form repo, so hopefully they will accept that soon. But in the meantime, you can still use my version by simply running gem install dynamic_form or adding to your Gemfile (if you are using Bundler) gem "dynamic_form".

Handling Dates in MongoDB

Article posted by Joel Moss on 09 Jul 2010 |

I've been using MongoDB quite a bit recently in several different projects, and it kicks ass. But the hardest thing to learn and to get used to with Mongo, is it's schemaless structure. With SQL databases like MySQL, you always have a defined and very rigid structure, which is of course the schema for each table you create. You create a table with a few columns; each column has a set type, a length, and a whole host of other variables.

It's this rigid structure that most of us have gotten used to for years, and that is the hardest thing to shake off when using NoSQL databases such as MongoDB.

But that is not to say that Mongo has no defined structure. In fact, that is far from the truth. Mongo and all schemaless databases have defined structure, but the difference is that that structure is defined by you. And that is what makes Mongo so cool. No need to actually create any tables and define each column - Just insert some data in your own defined way.

But there is a side effect of this schemaless idea. Even though you no longer have to define a schema in the traditional sense, you still have to think about the structure of each collection, and how that will look. In fact, there is a lot more to think about, as there are a lot more ways to structure your data with Mongo and other schemaless databases.

It's somewhat of a steep learning curve, but well worth sticking with. I'm still leaning the best ways to structure certain data types, and dates is one that I think I've nailed.

Nearly all your collections will no doubt contain some sort of date or time stamp. While Mongo does have it's own native Date() object, it's only really useful within the Mongo shell. I need to use it within PHP and Ruby and other languages. Up until recently, I've been saving datetime as "2010-07-09 13:56:31". This works fine most of the time, but when you start needing to gather your data in more advanced ways, it starts to fall apart.

For example, I'm currently using Mongo and the equally awesome Redis for logging data for ShermansTravel.com. I needed to log a bunch of data each time a user clicks one of our paying ads. I ended up with collection records like this: (in PHP, as that is what this project calls for unfortunately)


  {
    "_id" : ObjectId("4bfea7246c6151d127f80100"),
    "button_rank" : "5",
    "category" : "flights",
    "class" : "Economy",
    "cpc" : "0.17",
    "destination_place_id" : "197868",
    "from_code" : "ORD",
    "from_code_include_nearby" : false,
    "travelers" : "1",
    "created" : "2010-03-29 20:15:34"
  }

Notice the the created timestamp on the last line there. Nothing special there, and I can find by the created date without any issue at all. But I need to now count all clicks grouped by day, not time. This is where the above falls apart.

This is what I tried:

  $keys = new MongoCode('function(doc) { return { created: doc.created.split(" ")[0] }; }');
  $reduce = new MongoCode('function(doc, prev) { prev.count++; }');
  $clicks = $this->group($keys, array('count' => 0), $reduce);

Again, this works... almost! This week, the clicks collection ended up with 900,000 records, and using the above group query kept timing out. The problem is the second line where instead of simply specifying which key(s) to group by, I have to pass a PHP MongoCode object containing a Javascript function that creates and returns a custom key. In this case, I am simply grabbing the created key of each record, and splitting the string, so I have the date and the time. I then return the date only, and can now group by date. This means that Mongo is iterating over every record and applying that function to each one, which obviously can take some time on larger collections, and completely ignores any indexes you may have created.

So I quickly determined from this that I was storing the date and time in the wrong way. I need to split up the date and time. I now have this:


  {
    "_id" : ObjectId("4bfea7246c6151d127f80100"),
    "button_rank" : "5",
    "category" : "flights",
    "class" : "Economy",
    "cpc" : "0.17"
    "destination_place_id" : "197868",
    "from_code" : "ORD",
    "from_code_include_nearby" : false,
    "travelers" : "1",
    "created" : { "d" : "2010-03-29", "t" : "20:15:34" }
  }

The last line still shows the created date and time, but instead stores each in an embedded object or hash. This means that I can still find by created date and time with just a little change, but more importantly, it means I can group much easier and much, much faster:

  $key = array('created.d' => 1);
  $reduce = new MongoCode('function(doc, prev) { prev.count++; }');
  $clicks = $this->group($key, array('count' => 0), $reduce);

As you can from the second line in the above, I am no longer passing a function to build the key I want to group by - which was what was causing the problem. I now simply pass the key I want to group by: created.d. The code is leaner and the query is loads faster, and my indexes are respected.

Converting HTML Entities to Characters with Javascript

Article posted by Joel Moss on 29 Jun 2010 |

So I use Ajax quite a bit in many of my projects. And why not? It kicks ass! I have one project which dynamically updates specific elements of an HTML page via Ajax. Something like this:

$.get('/travel_guide/Paris', function(data){
  $('#advice').text(data);
});

The problem I have, is that sometimes that data can include HTML entities like & and >, and for reasons I don't want to bore you with, I cannot do anything about it on the server side. When inserting a block of text that contain HTML entities into the DOM, those entities are not parsed as such, and get printed exactly as they are.

What I need is a Javascript function that can take a string and parse it for any HTML entities, then replace each one with their HTML character. And this is what I came up with...

function entityToHtml(string) {
	for (var i in entity_table) {
		if (i != 38) {
			string = string.replace(new RegExp(entity_table[i], "g"), String.fromCharCode(i));
		}
	}
	string = string.replace(new RegExp("&#(x?)(\\d+);", "g"), String.fromCharCode(((p1 == 'x') ? parseInt(p2, 16) : p2)));
	string = string.replace(new RegExp(entity_table[38], "g"), String.fromCharCode(38));
	return string;
}

var entity_table = {
  //	34: "&quot;",		// Quotation mark. Not required
  38: "&amp;",		// Ampersand. Applied before everything else in the application
  60: "&lt;",		// Less-than sign
  62: "&gt;",		// Greater-than sign
  //	63: "&#63;",		// Question mark
  //	111: "&#111;",		// Latin small letter o
  160: "&nbsp;",		// Non-breaking space
  161: "&iexcl;",		// Inverted exclamation mark
  162: "&cent;",		// Cent sign
  163: "&pound;",		// Pound sign
  164: "&curren;",	// Currency sign
  165: "&yen;",		// Yen sign
  166: "&brvbar;",	// Broken vertical bar
  167: "&sect;",		// Section sign
  168: "&uml;",		// Diaeresis
  169: "&copy;",		// Copyright sign
  170: "&ordf;",		// Feminine ordinal indicator
  171: "&laquo;",		// Left-pointing double angle quotation mark
  172: "&not;",		// Not sign
  173: "&shy;",		// Soft hyphen
  174: "&reg;",		// Registered sign
  175: "&macr;",		// Macron
  176: "&deg;",		// Degree sign
  177: "&plusmn;",	// Plus-minus sign
  178: "&sup2;",		// Superscript two
  179: "&sup3;",		// Superscript three
  180: "&acute;",		// Acute accent
  181: "&micro;",		// Micro sign
  182: "&para;",		// Pilcrow sign
  183: "&middot;",	// Middle dot
  184: "&cedil;",		// Cedilla
  185: "&sup1;",		// Superscript one
  186: "&ordm;",		// Masculine ordinal indicator
  187: "&raquo;",		// Right-pointing double angle quotation mark
  188: "&frac14;",	// Vulgar fraction one-quarter
  189: "&frac12;",	// Vulgar fraction one-half
  190: "&frac34;",	// Vulgar fraction three-quarters
  191: "&iquest;",	// Inverted question mark
  192: "&Agrave;",	// A with grave
  193: "&Aacute;",	// A with acute
  194: "&Acirc;",		// A with circumflex
  195: "&Atilde;",	// A with tilde
  196: "&Auml;",		// A with diaeresis
  197: "&Aring;",		// A with ring above
  198: "&AElig;",		// AE
  199: "&Ccedil;",	// C with cedilla
  200: "&Egrave;",	// E with grave
  201: "&Eacute;",	// E with acute
  202: "&Ecirc;",		// E with circumflex
  203: "&Euml;",		// E with diaeresis
  204: "&Igrave;",	// I with grave
  205: "&Iacute;",	// I with acute
  206: "&Icirc;",		// I with circumflex
  207: "&Iuml;",		// I with diaeresis
  208: "&ETH;",		// Eth
  209: "&Ntilde;",	// N with tilde
  210: "&Ograve;",	// O with grave
  211: "&Oacute;",	// O with acute
  212: "&Ocirc;",		// O with circumflex
  213: "&Otilde;",	// O with tilde
  214: "&Ouml;",		// O with diaeresis
  215: "&times;",		// Multiplication sign
  216: "&Oslash;",	// O with stroke
  217: "&Ugrave;",	// U with grave
  218: "&Uacute;",	// U with acute
  219: "&Ucirc;",		// U with circumflex
  220: "&Uuml;",		// U with diaeresis
  221: "&Yacute;",	// Y with acute
  222: "&THORN;",		// Thorn
  223: "&szlig;",		// Sharp s. Also known as ess-zed
  224: "&agrave;",	// a with grave
  225: "&aacute;",	// a with acute
  226: "&acirc;",		// a with circumflex
  227: "&atilde;",	// a with tilde
  228: "&auml;",		// a with diaeresis
  229: "&aring;",		// a with ring above
  230: "&aelig;",		// ae. Also known as ligature ae
  231: "&ccedil;",	// c with cedilla
  232: "&egrave;",	// e with grave
  233: "&eacute;",	// e with acute
  234: "&ecirc;",		// e with circumflex
  235: "&euml;",		// e with diaeresis
  236: "&igrave;",	// i with grave
  237: "&iacute;",	// i with acute
  238: "&icirc;",		// i with circumflex
  239: "&iuml;",		// i with diaeresis
  240: "&eth;",		// eth
  241: "&ntilde;",	// n with tilde
  242: "&ograve;",	// o with grave
  243: "&oacute;",	// o with acute
  244: "&ocirc;",		// o with circumflex
  245: "&otilde;",	// o with tilde
  246: "&ouml;",		// o with diaeresis
  247: "&divide;",	// Division sign
  248: "&oslash;",	// o with stroke. Also known as o with slash
  249: "&ugrave;",	// u with grave
  250: "&uacute;",	// u with acute
  251: "&ucirc;",		// u with circumflex
  252: "&uuml;",		// u with diaeresis
  253: "&yacute;",	// y with acute
  254: "&thorn;",		// thorn
  255: "&yuml;",		// y with diaeresis
  264: "&#264;",		// Latin capital letter C with circumflex
  265: "&#265;",		// Latin small letter c with circumflex
  338: "&OElig;",		// Latin capital ligature OE
  339: "&oelig;",		// Latin small ligature oe
  352: "&Scaron;",	// Latin capital letter S with caron
  353: "&scaron;",	// Latin small letter s with caron
  372: "&#372;",		// Latin capital letter W with circumflex
  373: "&#373;",		// Latin small letter w with circumflex
  374: "&#374;",		// Latin capital letter Y with circumflex
  375: "&#375;",		// Latin small letter y with circumflex
  376: "&Yuml;",		// Latin capital letter Y with diaeresis
  402: "&fnof;",		// Latin small f with hook, function, florin
  710: "&circ;",		// Modifier letter circumflex accent
  732: "&tilde;",		// Small tilde
  913: "&Alpha;",		// Alpha
  914: "&Beta;",		// Beta
  915: "&Gamma;",		// Gamma
  916: "&Delta;",		// Delta
  917: "&Epsilon;",	// Epsilon
  918: "&Zeta;",		// Zeta
  919: "&Eta;",		// Eta
  920: "&Theta;",		// Theta
  921: "&Iota;",		// Iota
  922: "&Kappa;",		// Kappa
  923: "&Lambda;",	// Lambda
  924: "&Mu;",		// Mu
  925: "&Nu;",		// Nu
  926: "&Xi;",		// Xi
  927: "&Omicron;",	// Omicron
  928: "&Pi;",		// Pi
  929: "&Rho;",		// Rho
  931: "&Sigma;",		// Sigma
  932: "&Tau;",		// Tau
  933: "&Upsilon;",	// Upsilon
  934: "&Phi;",		// Phi
  935: "&Chi;",		// Chi
  936: "&Psi;",		// Psi
  937: "&Omega;",		// Omega
  945: "&alpha;",		// alpha
  946: "&beta;",		// beta
  947: "&gamma;",		// gamma
  948: "&delta;",		// delta
  949: "&epsilon;",	// epsilon
  950: "&zeta;",		// zeta
  951: "&eta;",		// eta
  952: "&theta;",		// theta
  953: "&iota;",		// iota
  954: "&kappa;",		// kappa
  955: "&lambda;",	// lambda
  956: "&mu;",		// mu
  957: "&nu;",		// nu
  958: "&xi;",		// xi
  959: "&omicron;",	// omicron
  960: "&pi;",		// pi
  961: "&rho;",		// rho
  962: "&sigmaf;",	// sigmaf
  963: "&sigma;",		// sigma
  964: "&tau;",		// tau
  965: "&upsilon;",	// upsilon
  966: "&phi;",		// phi
  967: "&chi;",		// chi
  968: "&psi;",		// psi
  969: "&omega;",		// omega
  977: "&thetasym;",	// Theta symbol
  978: "&upsih;",		// Greek upsilon with hook symbol
  982: "&piv;",		// Pi symbol
  8194: "&ensp;",		// En space
  8195: "&emsp;",		// Em space
  8201: "&thinsp;",	// Thin space
  8204: "&zwnj;",		// Zero width non-joiner
  8205: "&zwj;",		// Zero width joiner
  8206: "&lrm;",		// Left-to-right mark
  8207: "&rlm;",		// Right-to-left mark
  8211: "&ndash;",	// En dash
  8212: "&mdash;",	// Em dash
  8216: "&lsquo;",	// Left single quotation mark
  8217: "&rsquo;",	// Right single quotation mark
  8218: "&sbquo;",	// Single low-9 quotation mark
  8220: "&ldquo;",	// Left double quotation mark
  8221: "&rdquo;",	// Right double quotation mark
  8222: "&bdquo;",	// Double low-9 quotation mark
  8224: "&dagger;",	// Dagger
  8225: "&Dagger;",	// Double dagger
  8226: "&bull;",		// Bullet
  8230: "&hellip;",	// Horizontal ellipsis
  8240: "&permil;",	// Per mille sign
  8242: "&prime;",	// Prime
  8243: "&Prime;",	// Double Prime
  8249: "&lsaquo;",	// Single left-pointing angle quotation
  8250: "&rsaquo;",	// Single right-pointing angle quotation
  8254: "&oline;",	// Overline
  8260: "&frasl;",	// Fraction Slash
  8364: "&euro;",		// Euro sign
  8472: "&weierp;",	// Script capital
  8465: "&image;",	// Blackletter capital I
  8476: "&real;",		// Blackletter capital R
  8482: "&trade;",	// Trade mark sign
  8501: "&alefsym;",	// Alef symbol
  8592: "&larr;",		// Leftward arrow
  8593: "&uarr;",		// Upward arrow
  8594: "&rarr;",		// Rightward arrow
  8595: "&darr;",		// Downward arrow
  8596: "&harr;",		// Left right arrow
  8629: "&crarr;",	// Downward arrow with corner leftward. Also known as carriage return
  8656: "&lArr;",		// Leftward double arrow. ISO 10646 does not say that lArr is the same as the 'is implied by' arrow but also does not have any other character for that function. So ? lArr can be used for 'is implied by' as ISOtech suggests
  8657: "&uArr;",		// Upward double arrow
  8658: "&rArr;",		// Rightward double arrow. ISO 10646 does not say this is the 'implies' character but does not have another character with this function so ? rArr can be used for 'implies' as ISOtech suggests
  8659: "&dArr;",		// Downward double arrow
  8660: "&hArr;",		// Left-right double arrow
  // Mathematical Operators
  8704: "&forall;",	// For all
  8706: "&part;",		// Partial differential
  8707: "&exist;",	// There exists
  8709: "&empty;",	// Empty set. Also known as null set and diameter
  8711: "&nabla;",	// Nabla. Also known as backward difference
  8712: "&isin;",		// Element of
  8713: "&notin;",	// Not an element of
  8715: "&ni;",		// Contains as member
  8719: "&prod;",		// N-ary product. Also known as product sign. Prod is not the same character as U+03A0 'greek capital letter pi' though the same glyph might be used for both
  8721: "&sum;",		// N-ary summation. Sum is not the same character as U+03A3 'greek capital letter sigma' though the same glyph might be used for both
  8722: "&minus;",	// Minus sign
  8727: "&lowast;",	// Asterisk operator
  8729: "&#8729;",	// Bullet operator
  8730: "&radic;",	// Square root. Also known as radical sign
  8733: "&prop;",		// Proportional to
  8734: "&infin;",	// Infinity
  8736: "&ang;",		// Angle
  8743: "&and;",		// Logical and. Also known as wedge
  8744: "&or;",		// Logical or. Also known as vee
  8745: "&cap;",		// Intersection. Also known as cap
  8746: "&cup;",		// Union. Also known as cup
  8747: "&int;",		// Integral
  8756: "&there4;",	// Therefore
  8764: "&sim;",		// tilde operator. Also known as varies with and similar to. The tilde operator is not the same character as the tilde, U+007E, although the same glyph might be used to represent both
  8773: "&cong;",		// Approximately equal to
  8776: "&asymp;",	// Almost equal to. Also known as asymptotic to
  8800: "&ne;",		// Not equal to
  8801: "&equiv;",	// Identical to
  8804: "&le;",		// Less-than or equal to
  8805: "&ge;",		// Greater-than or equal to
  8834: "&sub;",		// Subset of
  8835: "&sup;",		// Superset of. Note that nsup, 'not a superset of, U+2283' is not covered by the Symbol font encoding and is not included.
  8836: "&nsub;",		// Not a subset of
  8838: "&sube;",		// Subset of or equal to
  8839: "&supe;",		// Superset of or equal to
  8853: "&oplus;",	// Circled plus. Also known as direct sum
  8855: "&otimes;",	// Circled times. Also known as vector product
  8869: "&perp;",		// Up tack. Also known as orthogonal to and perpendicular
  8901: "&sdot;",		// Dot operator. The dot operator is not the same character as U+00B7 middle dot
  // Miscellaneous Technical
  8968: "&lceil;",	// Left ceiling. Also known as an APL upstile
  8969: "&rceil;",	// Right ceiling
  8970: "&lfloor;",	// left floor. Also known as APL downstile
  8971: "&rfloor;",	// Right floor
  9001: "&lang;",		// Left-pointing angle bracket. Also known as bra. Lang is not the same character as U+003C 'less than'or U+2039 'single left-pointing angle quotation mark'
  9002: "&rang;",		// Right-pointing angle bracket. Also known as ket. Rang is not the same character as U+003E 'greater than' or U+203A 'single right-pointing angle quotation mark'
  // Geometric Shapes
  9642: "&#9642;",	// Black small square
  9643: "&#9643;",	// White small square
  9674: "&loz;",		// Lozenge
  // Miscellaneous Symbols
  9702: "&#9702;",	// White bullet
  9824: "&spades;",	// Black (filled) spade suit
  9827: "&clubs;",	// Black (filled) club suit. Also known as shamrock
  9829: "&hearts;",	// Black (filled) heart suit. Also known as shamrock
  9830: "&diams;"   // Black (filled) diamond suit
}

Unfortunately, there is no way to convert these entities without a big long list to look through. But it works nicely like so:

entityToHtml(string_with_entities);

Making my code now looking like this:

$.get('/travel_guide/Paris', function(data){
  $('#advice').text(entityToHtml(data));
});

jQuery UI Autocomplete and Caching Query Results

Article posted by Joel Moss on 14 May 2010 |

I've never managed to find a really good jQuery based autocomplete plugins, especially one that is flexible and easily extensible. So when the jQuery UI team released 1.8 back in March, I was intrigued to find that they included - amongst other new widgets - an autocomplete widget. And it turns out to be pretty damn good, and flexible to boot.

So when I had a need to add the ability for a local data source for the autocomplete plugin that we use on ShermansTravel QuickSearch, I gave the new jQuery UI widget a go.

The jQuery UI Autocomplete widget can load in data in three ways... (examples taken straight from the jQuery UI docs)

In an array with local data:

Pass it an array with a list of strings, and it will use present a list of matching values based on your entered string.

$(function() {
	$("#tags").autocomplete({
		source: ["php", "javascript", "asp", "ruby", "python", "c", "scala", "groovy", "haskell", "perl"]
	});
});

A string, specifying a URL to get the data:

Pass it a string of a relative URL, and it will make an ajax call to that URL, passing it the users entered string.

$(function() {
	$("#birds").autocomplete({
		source: "search.php",
		minLength: 3
	});
});

Via a callback (function):

Pass it a callback and you can do pretty much anything you want. This example uses such a technique and caches the query results.

var cache = {};
$("#birds").autocomplete({
	source: function(request, response) {
		if (cache.term == request.term && cache.content) {
			response(cache.content);
			return;
		}
		if (new RegExp(cache.term).test(request.term) && cache.content && cache.content.length < 13) {
			response($.ui.autocomplete.filter(cache.content, request.term));
			return;
		}
		$.ajax({
			url: "search.php",
			dataType: "json",
			data: request,
			success: function(data) {
				cache.term = request.term;
				cache.content = data;
				response(data);
			}
		});
	},
	minLength: 2
});

A Caching Autocomplete for Remote Data

What I really needed was a way to cache every single query when loading data in via an ajax call. The Autocomplete widget does not cache your returned query data at all, which means that an ajax call is made on every single key press. Even if you set the minLength to something like 3 - which will only trigger the autocomplete once the user has entered at least 3 characters - every character entered after that will trigger an ajax call.

So I came up with a somewhat simple, but effective caching strategy using the callback method for loading data.

var cache = {};
$("#birds").autocomplete({
	source: function(request, response) {
	  var term          = request.term.toLowerCase(),
	      element       = this.element,
	      cache         = this.element.data('autocompleteCache') || {},
	      foundInCache  = false;

	  $.each(cache, function(key, data){
	    if (term.indexOf(key) === 0 && data.length > 0) {
	      response(data);
	      foundInCache = true;
	      return;
	    }
	  });

		if (foundInCache) return;
		
		$.ajax({
			url: 'search.php',
			dataType: "json",
			data: request,
			success: function(data) {
				cache[term] = data;
				element.data('autocompleteCache', cache);
				response(data);
			}
		});
	},
	minLength: 2
});

Here we are saving the cache as a hash in the corresponding element using jQuery's data function. And we are doing this for every query performed. We first loop through each element of the cache, and check if any of the keys begin with the current search term. If we find something, then we call the passed response callback function, and return from the each loop.

If no cache is found for the current term, then we call the ajax as usual. But if we do find a matching key in the cache, then we don't call the ajax, and instead return the cached data.

Simple, but works nicely.

« previous page

In an array with local data:

A string, specifying a URL to get the data:

Via a callback (function):

A Caching Autocomplete for Remote Data

Recent Comments