I’m back from sxsw! One of the things I was looking forward to futzing with when I got home was an open source program called Processing. It’s intended to allow programmers to create visualizations of data with basic scripting skills. The ultimate goal is to be able to build crap like this interactive infographic about state of the union speeches, because it’s easier for people to process large amounts of information if they are presented visually. Here’s a hello world I mocked up last night (from a tutorial on Daniel Shiffman’s excellent site, Learning Processing):


<br /> No Java 2 SDK, Standard Edition v 1.4.1 support for APPLET!!<br />

Source code: sketch_mar18a

It requires java, so if you can’t see it, that’s why.

Posted by Sarah Davies, filed under FOSS, information, kim jong-il's righteousness bucket, technology. Date: March 18, 2010, 4:29 pm | View Comments

I’m at a sxsw panel on infographics! Here are the speakers:

51109_thumb

Ben Fry
Processing.org

51110_thumb

Casey Caplowe
Good

51111_thumb

Eric Rodenbeck
Stamen Design

51112_thumb

Shan Carter
The New York Times

Infographics are something that GOOD started to do early on. They created a section called transparency for graphical explorations of data. They are always working with different people and design firms. They’ve done infographics on Coke, Paris Hilton, Walmart, the largest bankruptcies in history, race of supreme court nominees, Iraq war, Darfur. They had a fight in the office over whether the Darfur graphic was too glib about genocide. They like to crack open things that people hear a lot about but don’t necessarily understand.

Infographics are a way of taking a large set of data and seeing what’s in there. If you look at a map of all the roads in the US, the cities and mountains emerge just from the data. The data tells the story, and it’s the job of the designer to get out of the way. You can trick people into thinking about correlations and statistics without making them deal with correlations and statistics. Processing.org gets people up and running quickly with creating infographics. They are hoping to get the designer out of the middle so that programmers can come up with beautiful graphics on their own. They hope to get it running on android.

The New York Times graphic department is about 30 people. They are responsible for all the charts, datamaps, and visualizations online and in the newspaper. About a third of those people do mainly online work. They are trying to do live visualizations, which is harder because you don’t know the story the data is going to tell. You can set up the graphic so the interesting data is in the middle of the page. Just because you’re talking about states doesn’t mean you have to use a map, particularly if only a few states are really unpredictable or interesting.

Graphics can capture moments in time. The OJ trial verdict could be seen in stock markets and phone call lengths. Turning common twitter words into graphics that resize based on their relevance can be very powerful. Consumers are used to zooming in and out, so you can put in a lot of data and let people go as granular as they want to. Infographics don’t have to be for public consumption, they can also tell organizations a lot about their internal information. Visualizations don’t have to just be on one page. They should have APIs and RSS feeds so that people can reclaim the data and use it for their own projects. Visualizations can be used for exploration, not just search.

Question Time!

What new technologies or processes are coming?
We are trying to come up with non-flash solutions to animating, mapping and zooming. We are trying to get off of the screen and into the real world, augmented reality on mobile phones, etc. Literacy is rising over interactive infographics, and we expect to see brands entirely built around this concept.

Does HTML 5 solve some flash problems, or is it just as limited as Flash?
There are limitations to both, but at least you can run HTML 5 on an iPhone. In terms of getting everyone access, being able to get everything in the browser and out of proprietary plug-ins would be great.

What are the resources out there to find good data and APIs?

It depends on your problem. Any time you’re getting started, you shouldn’t start with data, because it will look like data. Start with a problem or a question, and work back to the data. Many cities are getting better about releasing data sets, as is the census. Data.gov is also an excellent resource. Webscraping is always an option as well. Also, we sometimes just call companies and ask for the data we’re looking for. Sometimes they come through. You’d be surprised. They have excel files sitting around, and they’re happy to get the notoriety. It helps if you say you’re from the New York Times.

How do you balance data richness with design and beauty?

Charting is a language, and all the ways of communication are valid. Think about your audience. Are you writing a haiku or a technical manual? It’s a medium more than a toolset. There’s going to be a Beyonce of data visualization. It’s like musical genres or fashion. It’s going to get even more diverse as more people come to visualization, and each person brings their own style.

Are any of you working with augmented reality?

Microsoft is building geolocating maps with timestamps so you can go back in time on any given street corner.

Posted by Sarah Davies, filed under design, information. Date: March 16, 2010, 2:24 pm | View Comments

Hat tip to Chartporn. You’re welcome.

Posted by Sarah Davies, filed under horrible no good very bad slides. Date: March 16, 2010, 5:01 am | View Comments

I’m at a sxsw panel on open science. Here are the presenters:

51312_thumb

Ariel Waldman
Spacehack.org

51313_thumb

Kirsten Sanford
This Week in Science

51314_thumb

Jessy Cowan-Sharp
NASA

51315_thumb

Natalie Villalobos
Google

51316_thumb

Tantek ‡elik
tantek.com

How can you contribute and collaborate in open science?

You can go outside and look at birds! There are several websites that you can contribute to that will help scientists track bird population.

ScienceForCitizens.net is trying to bring together lots of different open science projects.

Galaxy Zoo is great for people at all different levels. You can classify galaxies from home. It gives you a picture of a galaxy, and you identify and classify galaxies. If you do a lot of them, then you unlock a button that tells the scientists when something about the galaxies looks strange, and they will help you do research on them.

Team Frednet is an open source team participating in the Google Lunar X Prize to build a robot that will go to the moon, get data, and send it back. They need help from lawyers, designers, and project managers, not just scientists.

Websites like infochimps will let you upload data about how you use the web.

Hacker Dojo is one of many hacker spaces throughout the Unites States. Hacker spaces have become a movement for open science.

Fold.it is a protein folding game. Playing the game helps scientists understand different ways that protein can fold.

Posted by Sarah Davies, filed under SxSW, science, technology, the intarwebs. Date: March 15, 2010, 8:16 am | View Comments

I’m at a panel at sxsw about using social media for advocacy. Here are the presenters:

50729_thumb

Amy Sample Ward
NetSquared

50730_thumb

Beth Kanter
Beth's Blog

50731_thumb

David J Neff
Lights.Camera.Help.

50732_thumb

Holly Ross
NTEN: Nonprofit Technology Network

50733_thumb

Kari Saratovsky
The Case Foundation

Short stories about crowd sourcing

Beth
Beth started a blog called spider school. She was writing about how nonprofits can use the web. She would get emails from people she didn’t know pointing out grammar errors and typos. She decided to start a feature called spider school police and give a digital badge to anyone who found typos or errors.

Amy
NetSquared facilitates offline events all around the world. Amy organizes a group in her city, and she tries to ask people to speak, but people don’t self-select to speak. One month we didn’t have any speakers, so we asked people for recommendations. They came through with programming for a whole year.

Holly
It’s been imperative for NTEN to utilize the community to keep up with technology trends.

Kari
Case Foundation opened up to the public and encouraged them to get involved in the grant solicitation. People provided feedback on applications, and they finally did a crowdsourced vote of who should get grants.

David
David has recruited volunteers. He has also built a website to allow people to tell their cancer stories including stories, video, and artwork.

About the panel
All the content for this session has been crowdsourced, including the powerpoint presentation. They launched a social media for social good case studies. It had a submission form which they opened in January. They had a ranking system so that people could rate the case studies. We selected case studies based on that ranking and the focus of the panel. They didn’t have funding for the panel, and it actually took very little effort to demonstrate to nonprofits how easy it is.

The Hybrid Model
The hybrid model has been a popular method of crowdsourcing. There is some good and some bad that comes in when you start crowdsourcing. The hybrid model has some responsibility with “experts” and some responsibility with the crowd.

Freerange Studios
Freerange Studios did a project called utopia where they did $30K worth of free work. Anyone could enter, the audience narrowed it down to the top twenty, and then they made the final decision for which project they wanted to work on.

Seattle Free School
The Seattle Free School uses social media as the entire operation mechanism. The idea is that it’s free to teach and learn within the community. It’s how they operate and how they grow. They use social media to distribute the roles of the members, so there’s no mail or fliers. It was even created through social media.

Invisible People
Invisible people is very good at story telling, helping people understand that homeless people aren’t different or scary. They crowdsourced who they should interview. The most amazing thing about the project is that he is unafraid to look away from an issue that almost everyone else looks away from.

Open Green Map
Open Green Map helps communities map themselves. Community members can enter any locations they consider to be green, like bike racks, eco-friendly restaurants, etc. The whole project is open source, so you can take the code and use it for any mapping project. They are actually creating change in their community.

Trends in submitted projects

The organizations who participated were not household names. Most of them mentioned that they had no marketing budget, and they relied on the power of social media and their communities.

Open Street Map
Open Street Map allowed people to add streets to a map. It is the main application being used by relief organizations to share which roads are accessible and blocked.

The Uptake

The Uptake covers Minnesota politics, and they livestream and let people comment in real time. Using the time stamp on the comments, the editors could easily find the video highlights and put them together.

When does crowdsourcing suck?
Anytime the legal department is involved. Any time you are writing by-laws or mission statements – things that need to be carefully worded and come from within the organization.

How can we use crowdsourcing to add value to the target population?

Crowdsourcing is one of the values that we have as social change organizations. We have to live by our values, and not just voting online, but actual online collaboration. The community will tell you what sort of research they want to accomplish together.

How do you prevent crowdsourcing from being a resource suck?
Crowdsourcing within a community is already part of the way a community operates. If you’re crowdsourcing to the crowd, you’re probably doing something simple like an online vote.

Netflix prize
Netflix has offered a prize to individuals who can improve their recommendation algorithms.

How do you convince your senior management that some of the best ideas come from outside your organization?
There are huge benefits to build community. You are bringing great people into the process. If your management doesn’t get it, then quit and bring your resources to an organization that gets it.

How do the panelists define crowdsourcing?

Amy
I’m not the best at everything, and I have to trust people outside my brain and empower them. An expert is someone who has a really good network.

Holly
Some problems should be solved by experts, but sometimes experts lack diversity that only large crowds can provide.

Kari
It’s a recognition that you can tap a wider audience than might exist in your own organization.

Jeff
There are smart people outside your organization. You should tap that potential.

How do you get people to work for free?

It provides value to them. They get to work with a community, which makes everyone more effective and efficient. But you shouldn’t ask for people to provide professional services for free. That’s disrespectful.

Posted by Sarah Davies, filed under NTEN, SxSW, crowdsourcing, nonprofit, nptech, philosophy of technology, technology, the intarwebs. Date: March 14, 2010, 2:32 pm | View Comments

I’m at a sxsw panel on accessible javascript. Here are the presenters:

41825_thumb

Patrick Fox
Razorfish

41826_thumb

Becky Gibson
IBM

Accessibility is all around us in the real world – curb cuts, access ramps, closed captioning. Accessibility is and should be ubiquitous. Accessibility benefits all of us – we watch closed captions when we’re watching a basketball game in a bar, we use curb cuts for strollers and bikes.

Semantic HTML is the foundation of accessibility. Your markup should indicate where headings, menus, links, etc. are. The essential part of making javascript accessible is to start with a normal page with good markup, and add the javascript afterward in such a way that the page looks normal with javascript turned off.

The sxsw website, for example, is completely reliant on javascript for its functionality. It’s unusable with javascript turned off. Also, rather than identifying the links as links, they are divs with a javascript “onclick” command. They should at the very least have anchor links.

You can use style sheets to show content if the user has javascript disabled, and hide it if the user has javascript enabled.

You should provide both mouse and keyboard events so that users can navigate with keyboard only.

Don’t mess with the functionality of the browser – don’t change how the up and down keys work, how the tab key works, etc.

There is a program called WAI-ARIA that is working on making web 2.0 accessible. It transmits data to assistive technologies (ATs) like screen readers informing the AT what sort of element it is looking at. So if you have a menu with collapsible sections, the AT would recognize that and alert the user.

ARIA also makes items “focusable,” meaning the AT can look at specific elements using tabindex.

ARIA can alert the user if ajax has updated any part of the page. Users can turn off the updates, make them “polite” – meaning that it waits until the AT is done reading or completing it’s current task, or make them “assertive” – meaning that it interrupts whatever is going on.

Examples of ajax updates would be autosaving a blog post and new email arrival. A screen reader user set on “polite” would hear that they have new mail after the screen reader finished reading out the email that it is currently reading.

ARIA works on JAWS 10 and Firefox 3.

To interface with ARIA, your page should have different regions which are identified by a “role” attribute, such as “main,” “banner,” or “navigation.” When the user looks at the page with a screen reader, the screen reader will tell the user what regions are on the page and allow the user to choose a region using the keyboard. The page can indicate to ARIA which regions are “live,” meaning that they might be updated by javascript or ajax. ARIA will watch those regions and alert the user if they change.

However, the page should still determine whether the user has javascript turned on, and serve them a static page if they do not.

Coding for ARIA does take more time, but JavaScript Toolkits have ARIA integrated so that it happens automatically.

Dojo is an open source javascript toolkit that fully supports ARIA.

Posted by Sarah Davies, filed under FOSS, SxSW, accessibility, technology, the intarwebs. Date: March 14, 2010, 9:52 am | View Comments

I’m at a sxsw panel on scaling websites. Here are the speakers:

50607_thumb

Serkan Piantino
Facebook Inc

50608_thumb

Alan Schaaf
Imgur LLC

50609_thumb

Kevin Weil
Twitter

Missing_thumb

Christopher Slowe
Reddit

50611_thumb

Jason Kincaid
TechCrunch

Imgur was released a year ago on Reddit. It was on a shared hosting plan. It lasted two days before the site was terminated for generating too much traffic. The site went down. Imgur moved to Mediatemple. That lasted three weeks, so they moved again, and again. Imgur moved four or five times in four months, scaling up to a better server with more bandwidth. Imgur went to foxhole.net, a content delivery network, because they have servers all over the world. That allowed the devs to concentrate on making the site faster rather than keeping the servers up.

Reddit is running on EC2 using about 50 machines. They have 20 app servers. They got a big speed boost by going single-threaded. They use Postgress and memcache.

Twitter started as a rails application tied to a single MySQL database. They have an open source queuing system, so they can do asynchronous processing.

A lot of the core architecture behind Facebook is still LAMP. They run newsfeed, ads, and search all on separate servers. On Facebook, you’re typically friends with 0-5000 people, whereas on Twitter you can follow millions of people, so they can render everything on the fly where Twitter can’t.

What is Reddit using for indexing?
They roll their own indeces using memcachedb. They are getting a .02% failure rate with that.

At what point are LAMP stacks not enough?

Knowing when a machine needs to be replaced is tough. Facebook has a monitoring system set up with alerts and teams dedicated to figuring out where they will have scaling problems before they have them. Monitor. Monitor. Monitor.

How do you scale search?
Search is really hard. The metric you are measured against is Google, which is a ridiculous standard. Reddit does about two queries per second. Getting quality results is really hard to tweak. It’s very qualitative in terms of what is “good” search.

What was the first thing that blew up?
Imgur had apache blow up first. “It was like trying to hammer a nail with a sledgehammer.” Twitter originally put the whole social graph in a MySQL database, but it was getting into the billions of rows. They had to build their own social graph store. They are in the process of open sourcing it.

What modules is Facebook using to convert PHP to C++?
They built a project called hiphop which compiles all their php down to binary C++. There are whitepapers about it, and it’s open source.

How do you deal with deployment?
Facebook and Twitter use BitTorrent to deploy builds to all their servers, cutting deployment from 12 minutes down to 30 seconds. Reddit cobbled something together in perl.

Why haven’t any of you used proprietary databases?
We prefer to work with open source. As you deal with scaling problems, you have to peak under the hood and see what you can tweak. Calling a vendor is a pain. Oracle is expensive. We like to be nimble and play well with the community.

Posted by Sarah Davies, filed under FOSS, Facebook, SxSW, technology, the intarwebs, twitter. Date: March 14, 2010, 8:28 am | View Comments

I’m at a sxsw panel on content strategy put on by Kristina Halvorson of Brain Traffic. These are my notes.

I’m mostly here because ACLU of Washington definitely deals with issues of creating/curating/editing content. We have dealt with this so far by having weekly standing meetings where the techies enter the content into the website as the writers create it. This strategy does take up more of people’s time and “bandwidth” (i.e. they can’t multitask), but it has significantly decreased missed deadlines, and minimized the endless editing problem, as the writers are less likely to send in edits on something if it’s already up on the site, and they’re less likely to change their mind about how they wanted to phrase something if they just came up with the phrasing five minutes before it’s put up on the site. I’m curious what strategies Brain Traffic uses to skin this cat.

Turns out that “ftw” is “for the web,” not “for the win.”

Content online is like the piles of trash in WALL-E. Most of it is crap, but some of it is cool and interesting, and he collects that, and when he finds a friend, the first thing he wants to share with her is all the cool stuff he’s found. That’s what we all do through social media.

The twitter stream for this talk is freakishly worshipful:

@halvorson is a content goddess and an amazing presenter
Kristina Halvorson lights up my life!
I’ve written “I love you” on my eyelids Indianna Jones-style for @halvorson’s talk.

Srsly. I’m a little worried about these people. Maybe they’ve just been in panels so long that they’re suffering from Stockholm syndrome?

Now she’s bitching about how her clients don’t give her enough time to write content. That’s probably accurate for most copy writers.

Copy is unintuitively complex. It’s easy to gloss over 40 hours of work as “that ‘about us’ part of the site.”

Edward Tufte was a breakthrough because he used design to communicate information without text.

Early in the project management projects, information architects will identify what content they will need, and they think about content the same way they think about features. They look at building a great house with a lot of buttons, but we didn’t engage the people who will be using the site. Content is not a feature. We can’t just check off pages on a spreadsheet and call them done. It requires ongoing care and feeding.

Late, poor, and disorganized content is just a reality of the web. We’ve been taught to just accept that, but we can strategize to minimize these issues.

Make sure there is someone in the organization whose fault it is if the content sucks. Make sure that person is at every kickoff meeting.

Text is much more mutable than pictures or video. We need to focus on every word written on the site and make sure that there is a consistent strategy.

Strategy isn’t just “what is on the website?”, it’s “why are we creating it?” “how is it being created?” “how is it being consumed?” and “what happens next?”

Messaging isn’t a tagline, it’s the understanding that we want to impart on our users when they are done interacting with our content.

The Quicken site doesn’t convey that it will help you feel more in control of your money. It’s just shots of different editions of the software. It’s telling the customer that Quicken wants them to buy something, not that Quicken can actually help them.

The Mint.com site on the other hand has an example graph of expenditures, copy that talks about being in control, saving money, and getting out of debt.

Quicken focuses on Quicken. Mint.com focuses on the user. Content strategy helps you identify what your messaging strategy is, and content will flow from that.

Does your content achieve your business objectives and your user’s goals?

You can’t be reactionary about your content. You can’t put something up just because it’s new or just because what was there before is old. You need a larger strategy.

REI did a great job of creating new fresh content by outdoors experts. The topics are closely targeted to the questions that customers ask in the store.

Room and Board’s mission statement says that they work closely with the builders and users to make sure that the furniture is useful. Their website reflects that by creating content from the people who are interacting with builders.

You can’t just give the writer a wireframe and tell them to go to work. The writer needs to know about the audience and the overall intent, otherwise the writer will include everything that anyone might look for by clicking to that page.

You need to take down old content. The swiffer page on youtube is for a campaign that ended in 2008, a broken graphic, and a notice that the last time anyone from swiffer logged in was 9 months ago. It’s like being invited to a party and finding an abandoned house.

Content strategy forces you to take a cold hard look at what you’re doing and why you’re doing it. Websites exist to house content. The reality of content is that it’s messy. It requires infrastructure to keep it clean and current.

Stuff you can do to fix this problem:

1. Audit. You have to know what content you have online. Where are you inviting people to interact with you? Make a spreadsheet that is the complete inventory of your content. What’s the low hanging fruit? What’s redundant, outdated or trivial?

2. Ask. If you are asked to write copy, ask why, and you will be invited into meetings earlier and earlier in the process so that you can understand why someone thought this was important.

3. Analyze. Take a high level view of your content ecosystem. What are your users actually doing? What are your competitors doing? What are trends in your field? Your content doesn’t exist in a vacuum. It is affected by many things out of your control.

4. Alignment. The content strategist sits between several different departments and process phases. It is your responsibility to make sure everyone is in line with the larger goals.

5. Assume responsibility. You should begin to see happiness in all the stakeholders. You should see your metrics improving. You should see better SEO.

Posted by Sarah Davies, filed under SxSW, copy writing, technology. Date: March 13, 2010, 3:15 pm | View Comments

I’m at a sxsw panel on universities. The speakers are Glenn Platt from Miami University Armstrong Institute for Interactive Media Studies and Peg Faimon from Miami University Design Collaborative.

Tenure is a big problem. Imagine trying to run an organization where anyone who has been there for six years gets to stay forever and you can’t fire them. “Change comes funeral by funeral.”

K-12 is accelerating. Many things that used to be taught in college are being taught in high school and even middle school. Florida has a “senior to sophomore” program, where seniors can get credit for their freshman year at college during their senior year of high school.

Students and especially parents are beginning to have a consumer mindset. They have many choices for schools and programs and a lot of data available to them, which changes the amount of marketing that universities have to do.

The open courseware movement is one part of the attempt at that marketing. Open up courses to the general public, and you will get notoriety among parents and students. Stanford has their courses on itunes. As more apps come online for mobile phones, we are seeing more and more educational apps, which are currently being evaluated and rated by everyone who uses them, which crowdsources the task of finding the really good ones.

Google books, Wikipedia, and blogs are starting to disrupt the textbook industry. Professors understand the textbook problem, and they are likely to send their students to the internet to acquire information that used to require the purchase of a textbook.

Students are starting to share notes online and vote for who takes the best notes and reward them with points or even money.

There are now entire universities that are exclusively online. Business and computer science degrees are the most common. Because of the low cost for implementation, tuition is around $100.

University of Phoenix has 150,000 MBAs studying with them right now. The largest meatspace university programs only have a few hundred. These degrees are taken seriously in the job market.

Professors have begun to become experience designers, who have the opportunity to leverage the internet while also meeting in a classroom. Student success is the ultimate goal. Having a degree is not enough to assure success. They need to have experience blogging. They need to have a substantial professional web presence. They need to be able to speak publicly in succinct, ignite-style format, and in long-form keynote-style format.

Universities are starting to work together so they don’t have to reinvent the wheel. Some universities are starting to have exchange programs with other universities in different areas of business, silicon valley, hollywood, manhattan, etc.

Bennington has gotten rid of their departments. They are focusing instead on questions. What do we do about water? The university is structured into teams of people with different skills working together, rather than departments which become echo chambers of people with all the same skills constantly working together. The “question” model is much more similar to the workplace environment.

Posted by Sarah Davies, filed under SxSW, education. Date: March 13, 2010, 10:54 am | View Comments

12  Mar
In Code We Trust

I’m at a sxsw panel on open government. I’m interested in how the ACLU of Washington can help promote dissemination of government data. Here are the presenters:

51088_thumb

Alissa Black
City & County of San Francisco

51089_thumb

Dmitry Kachaev
OCTO Labs/DC Government

51090_thumb

Noel Hidalgo
New York State Senate

I had no idea there were state governments that are so open! New York was the first state senate to adopt Creative Commons. They even send out their web traffic bi-weekly via RSS. They have built idea generator applications and put them under a GPL 3 license.

Alissa Black uptalks! She says that they built out a platform for different departments of the San Francisco city government to submit datasets that would be released freely online, but departments didn’t do it. So they then implemented a scorecard for each department. They have an internal city-wide wiki called citypedia! They pushed open source software adoption through the procurement side. They created a policy that says if a department is evaluating a software purchase over $100K, they have to investigate open source.

OCTOlabs created a contest asking developers to create open source government data applications using one of three government APIs. They have found that if you wait to have complete datasets, you will be waiting forever. Partial datasets should still be put up online. There seems to be a dillemma in that the return on investment of open government is very difficult to measure, so it’s very difficult to draw attention to. They are trying to build a community of people who participate in government through various applications and use that as proof that the program is working.

Audience questions are after the jump! Read the rest of this entry »

Posted by Sarah Davies, filed under SxSW, government, software, technology. Date: March 12, 2010, 2:02 pm | View Comments

« Previous Entries