On November 23rd L2, a digital think tank run by Scott Galloway, an NYU Stern professor, offered a new way to examine digital efficiency called Digital IQ in the Public Sector (full report here). It is a scale composed of several key indicators and was used to examine over 100 companies and agencies in the public sector and put them into 5 categories: Genius, gifted, talented, challenged, and feeble. One of our clients, NASA, was selected as having a Genius status for its digital IQ. In the report it featured most of our work with them and we wanted to see what we were doing right and how we might be of service to other more challenged organizations on the list.

Danielle Bailey, the lead researcher on the study, took time out of her day to give us the scoop.

Nathan: Welcome and thanks for taking time out of your day to talk with us. Please describe your role in this study.

Danielle: I work in the research department and have run a couple of studies prior to the public sector report we just released, one on specialty retail and another on Gen Y and their views towards luxury brands. I’ve been with L2 going on a year now.

N: Please tell us what spurred you to do the Digital IQ survey.

D: We do pioneering studies in multiple industries. The whole concept of digital IQ is the first comprehensive review of several industries; it is exciting to be part of establishing some metrics around organizations and benchmarks about not only people’s adoption of digital technology but also their mastery of it.

N: What makes up the Digital IQ scale?

D: The Digital IQ is comprised of 350 data points including social media metrics and much, much more.

N: We thought the study was very interesting, particularly because we work with many other government agencies. We were wondering if you could tell us a bit more about what kind of response you got from your surveys, what was the follow up from releasing these big results?

D: The most interesting parts are the numbers of people who were contacting us who weren’t in the study. They asked why weren’t we included? There are several reasons from that, but they also then said even if we weren’t included we’re very interested in the content, which has been great. First of all, just to give you a little background, it was done comprehensively by design, we tried to look at a broad swath of the public sector space to help us get a sense of what the players are like and who they are, but also to help us take a comprehensive look of what is going on in there.

Some of the ways we architected and designed who was in and out, a lot was based on site traffic. There were areas we wanted to highlight, so we made a strategic decision to only look at the high level for the executive branch, you know, we’re not going to look at some of the individual organizations like the CDC and NIH within the Health and Human Services agency.

N: I just wanted to make sure we follow up on that, it is an interesting approach to take because you think the worse ones wouldn’t have good site traffic so you’d think there is some sort of potential, you might be missing something if you’re basing it on their digital IQ already on a survey about their digital IQ?

D: Well no. I should say that site traffic was a determining factor in some segments more than others. For instance, we looked at the entire executive branch, all at the high level, we looked at all of the Armed Forces, but when you get into multi-lateral organizations and advocacy groups, we use site traffic as a way to further filter it down. The initial groups was almost 200 and we wanted to pair it down to 100 and site traffic was a way to do that in the advocacy category and with multilateral category. Once you look at the outcome of the study, you see that even with these supposedly high traffic sites you see there’s a tremendous amount of opportunity even at that level. There is still a lot of work to be done even in the most visited sites.

N: That makes sense. So along with that, you said the most interesting parts were the people who weren’t included in the survey, besides that what were some of the other key lessons in terms of the response itself?

D:Overall it has been positive, we’re objective to the space, this is our first foray into the area so that’s been a good validate. We hosted a launch for the participants of the study to preview the results and I don’t think people were really surprised, which is a good thing to show our methodology is actively measuring the temperature of the space. I think it may have been the first time they saw themselves against each other in that way, which may be a good take away.

The other thing that it sparked was ideas for people who were already high performers who now are contacting us about other things we can do. It says something about some of the organizations that are already categorized as geniuses, it shows that a genius always wants to find ways to get better and push the envelope and find ways to differentiate themselves in the space.

N: That’s interesting to see that the people at the top keep growing, sounds like several other industry marketplaces as well.

D: Right. Which is not great, part of this is that you hope that organizations that aren’t doing well with do better. But to look at another segment, luxury, which is where we had our beginnings, we just released a second study on luxury brands and we just saw huge turnover in the brands, there were only 2 brands that were still in the top 10 in the 2nd time we did the study. Hopefully when we do this study again we’ll see this same growth from the sector and a desire to get better.

N: That makes sense. Turning to that segment that is performing a little worse, what sort of response did you hear from them? Did you hear from some organizations how can we improve, or was it just as we were talking about with the high performers that the low performers weren’t that interested so they didn’t follow up with you? What did you find there?

D: Well it was a mixed bag. We heard from some people looking to improve but again the most interesting piece was that people who weren’t in the study at least have some guidelines about what are the levers they should be pulling, how do we dig deeper. I don’t think that there has been much pushback on those folks who found themselves in the lower rollout. There are always a few levers internally that we find with genius organizations. There is leadership, so people at the top bought in as a way to connect with their audience or constituency. There is resources, are they willing to invest time, resources to carry out the strategies. So those are other questions that also underlie some of that.

N: Did you get any challenge from people who were on the lower rung who said we don’t think digital is really that important for what we do and disagree with the premise of your study here?

D: We honestly haven’t gotten a lot of pushback from brands in the study. There were some who said you didn’t consider this particular thing that we did. So for instance, within HHS the CDC is doing something revolutionary, but again, we consider that separate, because most citizens don’t understand how they fit together, they don’t get that the CDC is connected to HHS. So there are instances where an agency says it is doing something with another agency and we say okay that’s great, we definitely have plans to go back and take this to a second level and dig deeper and highlight those sorts of things.

So for instance in luxury, and sorry to keep using it as a comparison, but it is a good one because you have these huge conglomerates like Richmond, most people are like what does that mean, as opposed to one of the individual brands that resonates with them. Does that make sense?

N: Yes.

D: We’re definitely planning to capture those agency levels and refine what we’ve done, but there has been little feedback from those in those lower ranks.

N: Great, thanks so much for sharing with us an overview of the response to your survey. We also were hoping to ask what your goals were and what you might see as the future of where you hope this study will take people?

D: Well you have to give a lot of Credit to Obama for stepping up and putting this on the agenda, but our overall perspective is that as the citizens becomes more savvy it only makes sense that the sector serving these citizens remains relevant.

If you look at the White House and notice that their Youtube gets more views than the mainstream media in those events, it shows a direct ability to engage with the people you are trying to speak with. If that is the interest of all of these organizations then they have to play in this space.

We’re studying what we believe are the levers one needs to pull and the measurements around this… we believe measurements are important in this. There’s been some pushback in the blogosphere about whether this space should be measured. I don’t know how you hope to improve something if you can’t measure it.

N: Very bizarre. That’s a perfect segway to discussing your methodology. Can you tell me more about how you are measuring the tone because I think it is hard to get these qualitative metrics right in a way that everyone respects your methodology?

D: We do much more than count Twitter followers and Facebook fans; we go through each properties and site in detail and not only look at the followers, but the tone, and how people are leveraging these mediums. Something I hope we can get across in this interview is that we do much more than count Facebook fans, our methodology is really robust, we go through all of the sties in detail and not only the followers but also the tone and how they’re leveraging these mediums.

We try to balance qualitative and quantitative, we’re looking at over 350 data points so there’s a lot there, but we’re also going beyond just the quantities and look at the qualitative bits of what is going on. We bring in a brand translation expert, someone who doesn’t work at L2 but is a respected marketer in the space and have them also evaluate the web prosperities.

N: What’s an example of who that might be?

D: For this study it was Craig Markus. He works for McCann. His bio is on the back of the report. He does the army’s campaign within McCann so he is very familiar within these organizations, he’s familiar with both the space and evaluation. One of the things we like to do is see how this outsider aligns with the in depth stuff we’re doing, in most instances it was parallel and in some it wasn’t.

N: What’s the most complicated methodology you use to measure this index? What’s the piece that differentiates you from other social media analytics?

D: I think the major thing is rigor, although there are also some proprietary measurements, but mainly the rigor that we use and mesh of both qualitative and quantitative information is critical.

N: Great. Well that’s really helpful for us to understand. Can you also tell us about the low hanging fruit for the challenged categories, it is such a large list?

D: For some of the organizations you can start at their website as a clear indicator of whether they’re really devoted to digital. I’m a citizen just like you, many of these sites I’ve never been on prior to the study. One of the things that you find within these sites is that there is interesting data that an every day citizen might be interested in knowing about, but it is often buried deep within a site or contained within a PDF somewhere. So how do these organizations bubble up information to a surface level where people access it?

So NASA is a good example of where they put that question in the users hands and ask them to identify what information they want at the front of the site. One of the things that I’m sure all of these organizations struggle with is that they have multiple constituencies, it isn’t just the public that is their customer but it may be other agencies that are interested in their information, so how can they better design their sites to make sure that each organization gets the information they need.

From a social media perspective, most are present on social media, which is great, but it is about how they’re leveraging it. You’ll see people using new media as old media, just broadcasting their press releases. That’s fine, as these social media platforms become a new medium that’s an okay way, but are they really engaging people in the potential of social media?

Beyond social media a big one is search, so what we found is that most people are not purchasing search terms at all or using search as a driver back to their websites. They say we don’t believe that’s a good use of our resources or even a way to use them. That’s a big factor. For instance for advocacy groups we thought we’d see people on pros and cons of an issue buying against each other. So if I was for gun rights, maybe you would see someone who is against that purchasing their search terms.

N: Very interesting, well we’re out of time here but thanks for your time talking to us. We certainly hope to see the next versions of the survey that you mention. Have a great day.

D: You too. Thanks for your time.

Written by Nathan Maton | Project Manager, JESS3

Tags: danielle bailey, interview, JESS3, L2, nathan maton