Chris Adams feed

Channel Description:

Latest Articles in this Channel:

03/29/10--07:58: Why should we fear a national ID card? (chan 1353137)

I see the usual concerns about the government collecting data about citizens are making the rounds again, with a recent Wired post about national ID cards and a concerned editorial in the CS Monitor about Census data. This sort of thing comes up regularly in both the generally left-leaning privacy circles and the right-leaning small-government crowd and I find it interesting that the discussion almost inevitably revolves around strategies for preventing the government from collecting data.

What I find surprising is that the discussion always seems to be based on the misunderstanding that this is a future threat, as opposed to something which has been routine for decades. I'm not referring to just SSNs, although they're closed enough to a national ID to make these discussions obsolete, but simply the fact that there are already many large databases containing various bits of personal data and since the invention of the digital computer (or at least the relational database) there's been no privacy advantage to be gained by avoiding a number since anyone can trivially cluster data on characteristics like name, age, address, etc. Academic research has demonstrated that much of our sense of privacy is illusory - something as simple as the name, zipcode and birthdate or approximate home and work locations suffices to uniquely identify most people. At this point, campaigning against a national ID card devolves into the case that abuse will be stopped because the suspect agencies aren't capable of basic database use.   

Since we know that large scale, effective data-mining is already going on - not every agency shares TSA's reliance on WWI-era technology in excel and not all of the data mining is being done by public agencies - the better question is what a reasonable expectation of privacy is in the post-Google era and how we can deter abuse while taking advantage of the benefits offered by modern computing¹. Since data is so easy to collect and mine, we really should be discussing acceptable use of data and the penalties for abuse, covering both government and corporate use. If we had European-style privacy laws, a national ID card could be discussed as the obvious good idea it is rather than the proxy for a litany of tangentially-related fears.

1. Does anyone really want to make the case that government shouldn't use effective tools? Instead of fighting the Census we should be trying to figure out how it could be updated yearly or better so policy can be adjusted for trends on less than a decadal timescale.




07/15/10--14:08: How to work in Git and push changes to Mercurial using hg-git (chan 1353137)

I often need to work with both Git and Mercurial repositories. I've previously used hg-git to work in Hg and push changes to Git but have found Mercurial to be less comfortable than Git (no flames, please: this is the newer vi-Emacs debate. Use the one you like) and was hoping to work in the opposite manner: local changes in Git pushed to an Hg repo on BitBucket or Google Code.

Travis Cline posted some instructions for working in Git and pushing changes to Hg which I've updated with a few stylistic tweaks:

• Install hg-git (e.g. ``pip install hg-git``)
• Make sure you've enabled the Hg bookmark extension in your ``.hgrc``
• Add this to your .hgrc::

      [git]
      intree=1
  

• Clone your Mercurial repo::

      $ hg clone https://acdha@bitbucket.org/ned/coveragepy
  

• Change into the repo::

      $ cd coveragepy
  

• Create a local bookmark tracking your Mercurial default branch - this is what will be exported to Git::

     
      $ hg bookmark hg/default -r default
  

• Export to the git repo::

      $ hg gexport
  

• Configure Hg to ignore the Git repo::

      $ echo ".git" >> .hg/hgignore
  

• Configure Git to ignore the Hg repo::

      $ echo ".hg*" >> .git/info/exclude
  

• Configure Git to ignore the same things as Mercurial::

      $ git config core.excludesfile `pwd`/.hg/hgignore
  

• Have your master branch track the exported Hg default branch::

      $ git branch --track hg/default master
      $ git reset --hard
  

• Do stuff in Git and make commits
• Export your changes to Hg::

      $ hg gimport
      
  

• Push them out to the world::

      $ hg push
  




07/23/10--05:03: Easy JavaScript error collection with Arecibo (chan 1353137)

If you work on sites with complex JavaScript you've probably wanted a way to know about the errors reported by users' browsers: even with a rigorous test process it's likely that there's some permutation of browser version and settings which you don't test, particularly when you consider external factors like JavaScript from third-party sources or the many ways in which anti-virus software, corporate proxies and policies can interact in darkly obscure ways.

There's now a really easy way to collect JavaScript errors thanks to Andy McKay and Clearwind Consulting: Arecibo. It's available as a commercial service for people who need support but it's also a completely open-source project on Github. Recently I've been working on an improved JavaScript client which has now merged into the official codebase, making it really easy to setup a personal error aggregation service up for all of your projects:

1. Set up an Arecibo service on AppEngine following the installation guide
2. Add this JavaScript fragment to your HTML templates:

   <script src="http://your-arecibo.appspot.com/lib/error.js" type="text/javascript" charset="utf-8"></script>
   <script type="text/javascript" charset="utf-8">
       arecibo.account = 'YOUR PUBLIC API KEY';
       arecibo.registerGlobalHandler();
   </script>
           

   It's often desirable to defer loading Arecibo until after the rest of your page has displayed which can be done using something like this example with jQuery:

   <script type="text/javascript" charset="utf-8">
       jQuery(function($) {
           $.getScript("http://your-arecibo.appspot.com/lib/error.js", function () {
               arecibo.account = 'YOUR PUBLIC API KEY';
               arecibo.registerGlobalHandler();
           });
       });
   </script>
           

The reporting interface looks like this:

There are two general caveats here: this service can't collect data when JavaScript is completely disabled or when the problem is caused by internet connectivity issues. Unfortunately browser error handling is also not standardized and WebKit browsers like Safari and Chrome currently don't have a way to capture unhandled exceptions; similarly, attempts to collect detailed stack traces varies from browser to browser so you'll find richer error reports from Firefox than Internet Explorer but in most cases simply getting the report is enough to start working on a fix or at least a more exhaustive test.




08/30/10--08:56: Django performance monitoring using Chrome's SpeedTracer (chan 1353137)

The modern web browsers have increasingly good tools for monitoring client-side performance - perhaps the best being the WebKit Timeline but there's only so much you can improve when most of the processing happens on the server side.

For those of us writing Django apps, there are fairly good tools for monitoring database performance by tracking slow queries (e.g. django-debug-toolbar or django-devserver) but there's not much else out there for measuring any of the other sources of delay (filesystem or network I/O, inefficient Python code, etc.) except the lower-level profile module. Most of these tools have the problem that getting useful data out requires a bit of work manipulating data, filtering out noise and assigning blame to the correct place (e.g. it's more likely that your code is calling an API poorly rather than that underlying API's implementation) - simply sorting a two-dimensional list is not enough for a large application.

One of the challenges for building better tools is building an interface which can interact with potentially very large amounts of trace data. Fortunately, Google's Chrome team added server-side trace support to SpeedTracer, making it possible for us to focus entirely on the problem of getting that data and simply reuse the work which the Chrome team has already done building the SpeedTracer UI and making it fast. Best of all, it uses only HTTP headers so there's no way for it to interfere with your sites' generated HTML and it works for any type of resource as long as it's served by Django.


Getting Started

1. Download and install Speed Tracer
2. Download and install my django-sugar fork - e.g. pip install git://github.com/acdha/django-sugar.git
3. Add "sugar" to your INSTALLED_APPS
4. Add "sugar.middleware.speedtracer. SpeedTracerMiddleware" to your MIDDLEWARE_CLASSES
5. Load your page inside Chrome with SpeedTracer enabled
6. Open SpeedTracer and expand the "Server Trace" in the page's detailed report which should look something like the picture above

Next steps

This has already proven itself to be somewhat handy for me at work but there's obvious improvement which could be made: this really should be converted into WSGI middleware so any Python-based webapp could benefit.

Inspiration

Thanks to Ilya Grigorik for his blog post about his racks-speedtracer module which inspired a night of tinkering last month. No thanks to my memory for not hitting publish sooner…




09/10/10--09:43: Announcing django-speedtracer (chan 1353137)

I've moved my earlier speed tracer middleware from my django-sugar fork to a separate repository: http://github.com/acdha/django-speedtracer




10/05/10--12:23: Plugging PromoteJS (chan 1353137)

The fine people at Mozilla are working to promote better JavaScript documentation than what you typically find with widely-linked but somewhat stale resources. This definitely deserves a plug, particularly to stay ahead of certain SEO bandits:





11/04/10--06:21: Google BidiChecker bookmarklet (chan 1353137)

Google just announced their rather handy BidiChecker utility for testing that your site correctly handles mixed LTR/RTL text. Running it in a unit test as they demoed is great but sometimes I just want to check a single page quickly, which lead me to combine it with my earlier bookmarklet template to produce this:

Run BidiChecker

Source: https://gist.github.com/662454




12/23/10--12:07: Django quickie: custom admin filters with 1.2.4 (chan 1353137)

NOTE: the call signature changed slightly for Django 1.2.5 - the gist below has been updated but you will need to update anything coded against 1.2.4

Django 1.2.4 was released earlier today with a security improvement which might impact some admin customizations. Django admin uses the querystring to implement list_filter and date_hierarchy but prior to 1.2.4 it also allowed you to simply link to a change list with arbitrary other filters. This wasn't a supported feature but was quite handy when you want to filter on something which doesn't make sense as a normal list filter: for example, a library wouldn't want many thousands of list_filter entries but might want book/change_form.html to link to {% admin:library_book_changelist %}/?author__exact=42 as a convenient way of navigating relationships.

The release notes are somewhat cryptic so it's easier to read the source for the new rules: ModelAdmin now has a lookup_allowed method which is used to either allow a lookup or raise a SuspiciousOperation exception. By default, lookup_allowed whitelists fields in list_filter or date_hierarchy; adding additional fields may be done by a simple subclass:




01/09/11--19:23: See you at PyCon (chan 1353137)

I'm looking forward to heading off to PyCon in a couple months, particularly as this time I'll be staying for the full sprints. Hope to see you there!






05/15/11--15:47: Where to eat in DC around Thomas Circle (chan 1353137)

A relative asked for some tips for a friend who was visiting DC and staying near Thomas Circle. This ran a bit long…

Sounds like fun. Just off the top of my head, you’re right around the corner from one of DC’s best beer bars (Churchkey); I haven’t been to the restaurant downstairs (Birch & Barley) but it’s also supposed to be good. Brasserie Beck is a very popular Belgian place with excellent mussels and a good beer selection. Peregrine might have their second coffeeshop open on 14th St. by then; otherwise your best option for high-quality coffee is probably Chinatown Coffee.

For Ethiopian food, Dukem is quite nice - there’s a big vegetarian platter which is quite good for variety - and Lalibela is good and also quite close to where you’re staying. In general, you can’t go too far wrong if you walk up to U St. (not too far) and stop when you smell something tasty. One word of caution: we haven’t been to a quick Ethiopian place yet, so plan for a comfortable, leisurely dinner.

Heading down to Chinatown, Burma is good for variety and Jaleo has good Spanish food. Rasika is a good high-end Indian restaurant - perhaps if you were thinking “upscale night out” - but to be honest, the best Indian food in the area requires a car trip up to College Park. Zaytinya is great option for Greek/Turkish/Lebanese and has good happy-hour pricing and specials.

Around the Mall, there aren’t many options on the south side - the cafe at the American Indian museum is easily my favorite as they feature a wide selection of Native-American food. One option which has really improved matters during the week is the new wave of food trucks competing with the traditional hot-dog vendors; the locations vary and the easiest way to track the options is to use a Twitter aggregator like Food Truck Fiesta - they frequently tend to cover the areas around the Mall since there are a ton of visitors and federal workers looking for better lunch options.

If you found yourself on Capitol Hill, Seventh Hill Pizza is superb and Belga is another good Belgian option.

Our favorite chocolate covered figs come from Biagio which is heading into the Adams-Morgan (18th St. NW up past U St). If you were up there, perhaps walking on the way to the Zoo, a really fast, tasty and cheap option is the Amsterdam Falafelshop and if you were looking for light meals and good coffee or drinks, Tryst is a coffeeshop with a liquor license and a full kitchen. There’s also a hispanic street market there during the day with most Central & South American cuisines represented by at least one stand. (One note: there are many places which claim to serve Mexican food in DC; none of the ones I’ve tried have been good. The closest option to moving back to San Diego which I’ve found is a taco cart in Northern Virginia).

I’m not sure how you were planning to get around town but one new option is Capital Bikeshare You can spend $5 for unlimited 30-minute rides within a 24-hour period or $15 for 5 days, there’s a great iPhone/Android app to find the closest station, and it’s extremely convenient for dodging traffic while still seeing the city. I use this heavily to head down to work and have been quite happy with it.