Sunday, May 31, 2009

Issues with Quantification

In the email that Ian Moss (Createquity) sent me, he raised one of the obvious problems with quantifying the arts health of the community: how to analyze the data sets. How much does it mean if there's $x or n number of festivals in a city?

Using the process of the Thriving Arts report that I keep banging on about, another problem with that arises. The data can reflect the existence of an arts community, but it doesn't seem nearly as easy to reflect the potential for an arts community.

My original reason for investigating was to come up with something that would let the <100k Project analyze which communities to select for its project--after all, there are plenty of <100k communities in the country: some (like Laguna Beach, or Tracy, MN from the report) already have a thriving arts community; others may have nothing at all to speak of. If there was a way to evaluate candidate towns statistically, it would be easier to use them.

But as I looked at the Thriving Arts Report, the benchmarks they put forward of "background factors" for a potential arts community are even harder to measure than, for instance, the benchmarks under "emerging development. The following are their community-related benchmarks for "background factors":
  • Valuing arts for young people
  • Valuing history and sense of place
  • Tradition of arts activity
  • Artistic expression in spiritual life
This is separate from a number of individual and catalytic events that also qualify.

In my attempt to brainstorm some quantifiable benchmarks for those background factors, it quickly became apparent that the presence of institutions that represent these factors come later in the process. A community can value history long before it creates a Historical Society and before it attracts the money to create a surplus of museums (both of which are listed as later on benchmarks).

What this means that, from the perspective of quantification, it is easier to tell the difference between an undeveloped and developed arts community, but difficult to tell the difference between a high potential community and low potential community. To return to my pet example of Irvine, where I am currently locating, long before we could ever hope to develop an artistic community, there would have to be a lot of work just to create those initial benchmarks. On the other hand, some towns in the early days of development might not have a lot, in terms of institutions, to measure with. In such a case, you'd have to measure with more like census data--trying to isolate informal, unestablished data points.


Ian David Moss said...

This is very astute. Another big problem with quantification efforts is that most of the data collection that goes on involves counting: e.g., such-and-such a place has 209 arts organizations, of which 67 are theater, 26 dance, 39 music, etc. And then statistical analysis may be run comparing different places with other factors such as real estate prices or civic engagement indicators, for example. But most observers would argue that the quality of an arts experience or an arts program is at least as, if not more important than its mere existence. Unfortunately, there has been very little attempt thus far to differentiate high-quality arts experiences, programs, organizations from lower-quality ones - and it's hard to see how it would be possible since quality is so subjective (and yet still very real).

Scott Walters said...

I think part of the key is to be creative in where you look. For instance, "Valuing arts for young people" would involve looking at the schools (are there art teachers, arts activities, etc), music teachers, after school programs, etc. None of these things require institutions, but rather a knowledge of what's happening. This is why the research process needs to be focused on a specific community, and must, at least to some extent, be done on-site -- this isn't something where internet research is sufficient.

As far as quality is concerned, the issue Ian raises, I disagree. Quality is relevant if you are thinking in terms of selling individual artistic commodities. I think, in a study for a <100K Project, the question is one of participation: how many people participate in the organization, either as creators or as observers? Evaluating quality is a sticky wicket that gets mixed up with a long string of artistic values that have been propagated by a commodity-oriented tradition that may not be the most healthy or applicable one for this type of activity.

Ian David Moss said...

Just to be clear about something, since I realize I may have been a bit opaque before: I'm not just talking about artistic "quality" in the sense of so-and-so is a great performer. I also mean the quality of execution of programs. So, for example, it's nice that there's a coffee shop in town where musicians can do live performances. But what if the coffee shop's management sucks and ends up alienating all of the musicians? Or what if they're terrible at keeping track of their expenses and end up closing? This kind of stuff is especially hard to deal with when you're trying to measure impact across different kinds of situations in order to prove a point, like for example that coffee shops promote economic growth or rising real estate values. It's tough to distinguish between awesome programs that are really valued by their communities and terrible ones that nobody cares about when all you do is count.

Scott Walters said...

I agree, Ian -- even in the world of business, many excellent ideas have foundered on the personalities and abilities of those who would implement them. As you note, there has been few attempts to differentiate "high-quality arts experiences, programs, organizations" from their opposite. To a large extent, I suspect that is because "quality" is hard to measure -- certainly harder than "quantity," -- as you note, quality is subjective. It is also highly affected by time and place -- one approach in a specific place at a specific time may be highly successful, while the same approach somewhere else at a different time results in total failure. Recognizing that, we can make an effort to gather "best practices" from those organizations that have succeeded, and make those best practices available to those who would like to create something in their own home town.

However, I will make this general rule, at least for the <100K Project. If "such-and-such a place has 209 arts organizations, of which 67 are theater, 26 dance, 39 music, etc.," then you should create your organization somewhere else -- this market is being over-provided.