So I spent this weekend at the #SNDMakes hackathon.  SND is the Society for News Design, and they hosted a hackathon on improving the experience of news content creators at the offices of Upstatement in Boston, MA, on October 17-19.

This was the nicest hackathon I’ve ever been to.  At other hackathons, I didn’t know many people, didn’t end up getting onto a team, and ended up milling around and leaving  because I had nothing to do.  So, side note on inclusion and diversity: assigning teams and making sure everybody who wants to work gets on a team is a BIG DEAL.  If you’d like to learn more about what happened there, here are my notes, which will recap all the teams and presentations for you.

Most of my technical experience revolves around

1. Finding stuff on Github
2. Downloading it
3. Seeing if I can get it running
4. Adapting it for my own uses.

 

I am a gleaner of the Global Junkyard O’Code, a role which I am delighted with.  I do not write much code from scratch.  My fellow team members, however, do, and so they did the heavy lifting of building our app.  For myself, I stuck to what I did best: seeing if it’s usable, and expressing how to use it to others.

So I created the project site for our app.  It was my first time using Bootstrap, a mobile-ready front-end framework originally developed by Twitter but now used by countless developers to make apps and sites.

What’s a front-end framework?  Let’s break it down:

• “front end” — the portion of an app or website that a user can see or interact with.  Look and feel, images, buttons, fields where a user can type something in.  The front end of many websites and apps are a combination of HTML, CSS, and Javascript, along with elements created by graphic designers so they Look Pretty. 
• “framework” — a framework is a toolkit of prewritten parts.  In this case, there are prewritten parts to help you make a site or app that automatically resizes so that it looks good on a big screen or a phone; nicely formatted input boxes, buttons, and more.

I decided to use Github Pages and Bootstrap to create a one-page site for our project.  I briefly considered using Jekyll, but it seemed to me like Jekyll was more suited to blogging, and all I really wanted was a nice-looking one page/landing page site for our project.

Github pages lets you use the popular code-sharing site Github as a host for simple, static websites.  I love Github pages — they’re wonderful.   I used a Bootstrap template to create the site, and Github Pages to host it.  Here it is.

ABSURDLY ILLUSTRATED TUTORIALS

What basic coding/tech skills meet the following criteria?

• They are relatively easily required with no need for a 4 year CS degree;
• They provide journalistically relevant and useful results
• They are reusable in a journalistic context

In my opinion, there are three skills that meet these criteria:

• Mapping.  Most news happens in a place.  Maps are ancient precisely because they are such an expressive and powerful form of data visualization.
• Grabbing.  Thousands of websites have data gateways called APIs (Application Programming Interfaces) that allow you free access to some or all of that site’s data — as long as you can write a relatively simple program that can grab it and return it to you in a format you can use.
• Scraping.  This is to get the data out of all those sites that don’t require API’s — read “crappy government websites.”

(For a brief and entertaining video on these three, see, “Do I Really Have To Learn How To Program?“)

I’ve done quite a bit of mapping, and some grabbing, but my experiences with scraping have been less successful, primarily because they proved to be less generalizable.  I’d be able to pick my way (often slowly and with much frustration) through a scraping tutorial and get results.  But at the end I did not feel that I could write a script on my own to scrape other things.

After taking a class on scraping at Journalism Interactive with Michelle Minkoff,  I decided to buy Paul Bradshaw’s book “Scraping for Journalists,” and take another run at it.  If you would like to read along, my notes as I pick through the book are after the jump.   I would also like to thank Michelle and Paul for giving me the inspiration to restart this blog.  I have been very busy with my new duties at INN, a network of 90+ investigative and community newsrooms, so I have not been devoting much time to adding to my own store of code-knowledge or developing tutorials to pass on what I’ve learned to others.  But it’s something that I enjoy and believe gives back something of value to my peers in the field, so I welcome the chance to begin anew.

Continue reading →

Not too long ago I was able to attend a demo of the data visualization toolkit Weave.  The person giving the demo was Georges Grinstein, one of the tool’s creators.  Georges hails from the University of Massachusetts at Lowell.

He showed a really amazing demo of foreclosure data from Lowell, MA.  For those of you who aren’t from Massachusetts, Lowell was one of America’s first industrial cities; massive textile mills once dominated the town.  The mill buildings are there — but the kind of jobs they once provided are long gone.  I have a lot of affection for Lowell because my grandmother lived there and my mother was raised there; my dad graduated from the University of Massachusetts at Lowell at the age of 40 with a degree in computer science.

But enough about me!  Let’s get back to dataviz:

So first off, check out this link:

Lowell Foreclosure Demo

Give it awhile to load.  I also recommend loading it in either Firefox or Chrome.

Run your mouse over anything.  Anything at all.  Everything here is highly interactive; you can drill down on almost anything.  That’s pretty exciting all by itself, but now take a look at the menus in the upper right.

It’s not just interactive; it’s generative.  You can remix this, create your own visualizations, change what the dashboard looks like and what it displays, add your own data!

When I think about teaching beginners data visualization, one of the primary questions I ask myself is:  What am I teaching students that they can’t do easier and faster in Excel or Powerpoint?  What new vista am I helping them to see?

To me, one of the primary ways to depart from the “Excel box” is interactivity, but also generativity.  That’s why this is exciting.  

 

Photo Credit: scriptingnews via Compfight cc

Last week I was lucky enough to meet with three folks who work in the newsroom of a daily newspaper.  That’s a big deal to me, because if my work isn’t useful to people who work in a newsroom, a mission-driven nonprofit, or doesn’t work for folks who want to change the world (or even just their little piece of it), I’m wasting my time.

I asked them: “What should my next step-by-step tutorial be?  What would be really useful for a beat reporter who doesn’t think of themselves as a techie to pick up?”  Remember, it wasn’t all that long ago that shooting video was considered a specialty task that print reporters didn’t do — and now everyone just points their iPhone at it and calls it a day.  (Okay, some do a great deal more than that!  But you get my point).

The folks who were kind enough to spend some time with me gave me the following hints:

1. The Absurdly Illustrated Guide To Your First Tableau Public ChartsnGraphs — Tableau Public is a downloadable app that lets users transform datasets into classic data visualizations — bar charts, pie charts, scatterplot, time series, and more.  The end results are embeddable in a web page the way that a YouTube video is, and a few are interactive.   
2. The Absurdly Illustrated Guide To Your First Survey with Crosstabs — There are lots of survey tools out there, but only a few of them do “crosstabs” — that’s the ability to compare one survey answer against another one.  For instance, a survey that asked folks what their favorite flavor ice cream was but also asked their gender and had crosstabs could tell you that 47% of women liked black raspberry ice cream, but only 12% of men.  My job would be to pick the best and most web/mobile friendly tool out there and produce a tutorial on how to use it and serve it up on the web and mobile devices.
3. The Absurdly Illustrated Guide To Your First ArcGIS Online Map.  ArcGIS is a “geographical information system” or GIS.  GIS predates web-based mapping systems like Google Maps by a couple of decades.  They used to be very, very expensive software used by specialists, and to some extent they still are.  But to get with the times, ArcGIS now has an online service too.  I’ve never used it, but hey, before I wrote my TileMill tutorial I never used that either!  Writing tutorials is a spectacularly effective form of learning — if I understand it well enough to explain it to a total beginner, I probably understand it pretty well.

One thing I’ve been thinking about lately that really made an impression on me is how important it will be for me to focus on “zero install” tools.  Many folks have computers and servers that their corporate IT department doesn’t allow them to install anything on for security reasons.  So my TileMill tutorial, which requires you to download an app, won’t work for folks in that situation.  But a tutorial on the data visualization platform ManyEyes would — you don’t have to download or install anything, you just work with the application from your web browser.

That’s super-useful information for me as I go forward and write more data visualization tutorials.

So, two things:

1. If you’re reading this and you guys are the folks who were kind enough to meet with me, you know who you are  I didn’t use your names here because I forgot to ask if I could, and not asking would be really rude! But if you’d like the credit for giving me so many smart ideas please let me know  
2. What about more tutorials?  If you’re reading this, wherever and whoever you are, and you have a burning desire to learn a specific data visualization or mapping tool or technique, please let me know.  I’m a noob like you (if you’re a noob), so I can’t guarantee I’ll do every one, but I do want to know!

Recently I searched for the name of my current project, “Data for Radicals,” and through that magic we know as Serendipity on the Internet, up popped:

Ten Rules for Radicals

by none other than Carl Malamud.  To be honest, before I read “Ten Rules for Radicals,” all I knew about Carl was that my friends who were investigative journalists — particularly those who did the deep data and document dives through FOIA and other means — talked about him in hushed tones of awe.

Reading the title, I could not help but think that the essay, originally an address to the WWW2010 conference, represented one of those strange messages that happen between people of ideas, even if those people are separated by centuries, or thousands of miles, or other barriers, and even if they have never met.  Haven’t you ever had that feeling of picking up a book and feeling that the author is, in an uncanny and spooky way speaking to you — directly to you?  I felt that way about this essay.  You can find it in full here, but interspersed within it are his “Ten Rules for Radicals,” which in the essay he illustrates with stories from his career.  You should do yourself the favor of reading the whole thing, but for my own edification, I am reprinting the ten rules below.

Just as I do when I am learning new code, I did not copy and paste these.  As I sit here, I am typing them word by word with my own ten fingers on my Macbook Air, sitting at my dining room table at 2:21 AM on Saturday, May 18 (What can I say?  A dream woke me up and I couldn’t get back to sleep).

Rule 1:  Call everything an experiment.
Rule 2: When the starting gun goes off, run really fast.  As a small player, the elephant can step on you, but you can outrun the elephant.
Rule 3: Eyeballs rule.  If a million people use your service, and on the Internet you can do that, you’ve got a lot more credibility than if you’re just issuing position papers and flaming The Man.
Rule 4: When the time comes, be nice.
Rule 5: Keep asking until they say yes.  Gordon Bell, the inventor of the VAX, once said that you should keep your vision, but modify your plan.
Rule 6: When you get the microphone, get to the point. Be clear about what you want.
Rule 7: Get standing.  Have some skin in the game, some reason you’re at the table.
Rule 8: Get them to threaten you.
Rule 9:  Look for overreaching, things that are just blatantly, obviously wrong or silly.
Rule 10: Don’t be afraid to fail.  It took Thomas Edison 10,000 times before he got the lightbulb right, and when he was asked about those failures, he said, “I have not failed, I’ve just found 10,000 ways that won’t work.” Fail. Fail often. And don’t forget, you can question authority.

If I could put these on stone tablets, or better yet for our era, put them on plastic tablets extruded by a 3D printer, I’d do it.  They’re a little long for a tattoo, though

 

Sortable, searchable table of law enforcement agencies, Middlesex County, MA

Tabletop.js is a Javascript library that lets you use Google spreadsheets as the data source for web apps.  It’s pretty neat — especially since we know there are so many simple but useful web and mobile apps we can create where setting up a full-on database is overkill.  What if you want to make a sortable, searchable list of craft breweries?  Or a schedule for a music festival?  Do you really have to bust out MySQL for that?

Well, with Tabletop.js, you don’t.  The other great thing about using Tabletop.js is that a lot more people know how to use a Google spreadsheet than know how to enter records into a database.  You can share responsibility for updating your web app with anyone who knows how to use Google Docs, or use Google Forms to let members of the public add new records to a list.

Chris Keller wired up Tabletop.js to a nifty sortable, searchable, and nicely styled web-based data table, which I use here to create a sortable, searchable table of law enforcement agencies in Middlesex County, Massachusetts, where I live. I decided to do that because I live in Watertown, Massachusetts.  Until recently, very few people outside of eastern Massachusetts knew about Watertown — where it was, what it looked like, what happened there.  That was before the Tsarnaev brothers led police on a car chase through my town, culminating in a gunfight and explosions.

So I thought I’d do my latest plunge into civic data by creating a data table showing all the law enforcement agencies in my county, how many full time police officers per 1,000 residents each police department has, and the population of each city or town.  (College police departments are also represented in this table.  I thought about taking them out, until I remembered that Sean Collier, the 26 year old police officer who died, was a campus police officer at the Massachusetts Institute of Technology (MIT)).  You can see the table here: Sortable, searchable table of law enforcement agencies, Middlesex County, MA.

I’m a noob, and I had a tough time getting my mind around Tabletop.js, but it was worth it.  Also, once I finally got it to work…I happened to be wearing my “Watertown Strong” t-shirt.  Coincidence?  Nah.

 

 

Now that I’ve got it working,  I’ll be creating another Absurdly Illustrated Tutorial for those of you who want to try out Tabletop.js.  If you’d like to see my other tutorials, see:

The Absurdly Illustrated Guide To Your First Dynamic, Data-Driven Timeline
The Absurdly Illustrated Guide To Your First Data-Driven TileMill Map