Response to Heather's Question on Metrics

I want to thank Heather for commenting on the previous post. And while I cannot take her up on a dance off (we haven’t met face-to-face Heather, but large overweight bald white guys are not a pretty sight on the dance floor... I could trip over my two left feet and kill somebody accidentally), I felt her question deserved a response. I decide to do this as a post rather than a comment in reply because long comments are even more tedious (if that’s possible) than long posts.

Also, I apologize for any errors in this post. I had exactly 30 minutes to throw it together before my next meeting.

Investors decide the validity of metrics. They usually decide by evaluating the return along two factors : efficiency and effectiveness.

Efficiency is about reducing the cost and risk to capital. Said another way, whoever can do more with less wins. Effectiveness is about value: with the same amount of capital I can extract more value than my competitors. Both measures are relative to competitors. They are not static.

In Employment Branding, I believe most investors think in the following ways:

Efficiency: By investing in Employment Branding I reduce transaction cost per dollar employed, or, conversely, get more transactions done with less dollars. So if its between putting another dollar into hiring a recruiter or putting a dollar into blogging, I want to know that I can get more done with a blog than I can with a recruiter. Everyone starts to chuckle “Now you are just off in left field.” Actually, no. Look at the total end-to-end recruiting and hiring transaction. In a “bad market” (labor demand outpaces labor supply in positions that are core to competitiveness), more dollars go to front-loading the transaction than to maximizing throughput later in the process. In English: it costs more to source relative to how much you spend on screening, interviewing, closing and on boarding.

If more dollars are front loaded into the system, then the efficiency measure is “Can you source more people with less money, or more people for the same amount of money?” In this respect most corporate blogging (but especially your blog), provides a good return. Microsoft competes against a lot of different companies for marketing talent. At the same time, Microsoft is ever more dependent on marketing talent to expand its markets (there are only so many office suites any one person can buy). In a buyer’s market, differentiation between sellers is key. As the volume and velocity of information increases, rising above the noise is absolutely critical. In a hiring transaction there are only two ways to do this: develop low-load / high value relationships and provide a convenient location for buyers to gather data that is meaningful for them in order to differentiate various competing offerings. Blogs do a really good job of both.

“But wait – how do we measure that?” In this case repeat traffic is not a bad measure. Perfect? Absolutely not. But better than a guess. Repeat traffic from IPs that you want to target is an even better measure. Creating an association between your topic, IP targets and source of hire would be yet another.

Effectiveness: By investing in Employment Branding I (investor) increase the quality of the candidates who will actively consider employment opportunities at Microsoft. Of course this is tricky, but not because the net effect of blogging is difficult. It is difficult because “quality” is a measure of variance to specification, and our specs (job descriptions) are not highly predictive of value delivery probabilities. So it is difficult to determine whether you get “better” people if you don’t know what “better” means.

But let’s get beyond that for a second and assume that we know who the stars are within Microsoft. The “existence proof” you are seeking for the effectiveness measure of your blog is a conversation. Talk to the new hires in marketing that started at least 3 months (an arbitrary number, but intuitively long enough to allow your blog to get traction and inbound links) after you started blogging and ask them whether:

1. They had read your blog prior to starting work.
2. The blog played any part in their decision to join Microsoft.

Now wait for six months and go talk to the hiring manager of all the new hires. How do they stack rank (which is a baloney methodology, but most investors will buy the output), and what is the correlation of those who are “A”s to those who said “Yes” to both questions. Compare that to the people who said “No."

All the statisticians and social scientists who may be reading this can blow holes all day long through this methodology. And they would be right as far as the science goes. But empirical validity is not the problem you are trying to solve. Continued investment is. And most investors would take a positive correlation as “proof” that they are getting a good return for their bucks.

Overall, you have to remember that when a new system is created the measurement is largely along “belief proof” and “existence proof” coordinates. Do you have any data to support the claim that you were able to source more people than before you started blogging? It could be as flippant and anecdotal as one sourcer’s word. It could be more detailed: you could do an alignment survey of the downstream clients of your service “On a scale of 1 to 10 rate how effective blogging is in getting you more and better candidates.” Where you have a high degree of standard deviation in response you have little data. But where you have low standard deviation, either good or bad, you have basic proof.

One of the hardest things that we do in business is to take intangibles and turn them into tangibles. Harder still is to take indicators and turn them from lagging (traffic) to leading (probability that any future candidate will have made a buying decision based on your content). And then there is the hardest job yet… making all meaningful to investors without killing the goose that lays the golden egg. I understand that this, more than anything, is your basic concern. But just because it is really, really difficult does not mean that it is not worth pursuing. We need to get more investors to keep this train rolling, and at the end of the day we are accountable to the value they perceive they get from us.

(And that’s why I pay for my own blog and do it on my own time – I don’t like most investors. ;-) )

 

Comments

[martin snyder]

hey hey- what's with the chubby bald guy references ? That's MY avatar your talking about.....

this topic has been cross posted all over the place, but here is a response I added to another blog:

Sure you can count page views, links, hires, etc. etc. but how do you measure influence? How do you measure the impact of an idea? It's like trying to measure a single snowflake in an avalanche; it may or may not tell you anything about the larger phenomena.

If Heather, or any other blogger, posts a highly novel or provocative idea, and receives one single click, what does that tell you? Well, if that click is the Secretary of Labor or the CEO of GE, and that person takes action based on the idea, your 'metric' is a joke. If that person was looking for information on pimping her ride and landed on the wrong site, that one click means less than nothing.

This is not even to speak of the effect of blogging on the blogger themselves; the possible changes wrought by study or deeper thinking about various topics, or the clarification and validation or rejection of certain positions that can change future efforts.

Measurement in media and throughout business, is critical, and there are a lot of things that can be done better by better measurement. But blogging is one of those things where the 'metrics' can't yet be a reliable mirror or indicator of results.

I think Heather was closer to the real truth than Sumser. Check this article for an interesting summation of this concept:

http://en.wikipedia.org/wiki/Emergent_property

[Jeff Hunter]

Martin -

Let me say that it is an honor to have someone of such intelligence and thoughtful purpose commenting on the blog. I don't agree with you, but that is the beauty of this enterprise. I would ask: when is that magic moment when 'metrics' can be a reliable mirror or indicator of results. If you don't need an existence proof now then when will you? Or will this be the kind of creative acitivty that needs no justification, because to justify it will be to denigrate it?

Seriously, one bald guy to another (we have to stick together you know), the conversations we have had about the video game industry have been inspired. Are you saying that metracritic ratings on games shouldn't count, because the games are billed as "movable art"?

Here is where people with your depth and intelligence need to explore, because it is in fact the crux of our collective business future: how do you make the intangible tangible without destroying it? How do you take something as delicate and beautiful as a flower and not rip it out of the ground in order to find out how much pollen it contains? I don't know the answer to that question, but I don't buy that just because it is difficult that seeking that course will necessarily be destructive to creative enterprises. I am more of an "and" guy.

Again, thanks Martin. It is really great to be out here talking with you.

[Rob McIntosh]

Now we are finally getting somewhere. I have been watching this whole discussion around the blog network for the last little while and poking my nose in from time to time but finally I think we are starting to hit on the essence of the real issue here. Just because it is squishy or hard to put a finger on it does not mean we should not try and measure it.
I am not saying there is a magic solution here to measuring what some feel is immeasurable, but since blogging is now being considered by many as a viable marketing and talent attraction tool, then it is time we at least started trying to measure what works and what does not.
Heck, if we all sat back and cried "oohhh, that is too hard", then we would have had Edison give up on inventing the light bulb after a few attempts vs. the many he took to get it right.
Now is the time that we starting poking and prodding on how to duplicate/replicate the success of what blogging can and should be!

[Heather]

Oh yay, I love hearing Martin's comments!

So Jeff, first, we know you are smart (and you say you are large and bald). I haven't read anything like that since I was in college. Yowza! Your theory breaks down (early) at the point that you 1) assume that investors really care what a single employee does and 2) assume that we don't hire people that are competent to make the determination of HOW to get their jobs done themselves. What you describe is my worst freaking nightmare. Why make it so hard when it doesn't have to be? "Hard to measure" does matter when you have enough anecdotal info to make a go/no go decision and the data gathering sucks up time.

The fact of my situation is that nobody asked me to blog, ever. I was given a job to reach out to people in marketing, make them think about working here, make them *want* to work here. How I decided to do that was up to me (and I was barely able to scrape together the few brain cells it took to figure it out...as if). Investors want to know whether they can get more done with a recruiter or a blogger? Really? Do you think that actually comes up at shareholders meetings? Shareholders want the best senior leaders driving business decisions. They couldn't give a rip about me and my blog, nor should they. Now, I'm not sure you are meaning "investors" literally, but if you mean "management", my simple recommendation to them (in the generic scenario yu describe) is that they hire better people that can make decisions for themselves, if that is not what they are seeing and if they feel the need to have that level of control over employees blogging.

Repeat traffic is a measure? How so? (and how do you measure "repeat" traffic?) I know for a fact that at least one person comes to my blog regularly to look at my picture (hey, life is weird). I'm confident that some people come to my blog because they don't like me. My mom visits my blog regularly to find out what I am up to. There are tons of people that visit my blog that don't have the kinds of marketing backgrounds that we would hire at Microsoft. These folks are traffic. Yet, there's some value there that's equal to the value of traffic from people that match the target profile I'd look for? What if my posts suddenly stink and I start to behave very oddly? People might come by for the spectacle. My traffic will spike (have you seen all the Alaska Airlines comments on my blog...and don't even get me started on blog spam traffic). Traffic is not a generic measurable that equates to value for bloggers or shareholders. I think this is Martin's point. It's how they *feel* about what they see and whether that compels them to do something now or later.

So let me give you a scenario. Let's say that these "investors" of yours say that the number of hires they would want to see (because they are intensely interested in my job?) as a result of my blog is 2 (just an example); the number of hires that justify the time (my salary) investment. Let's say that during the year, five people e-mail me and tell me that they just joined Microsoft BECAUSE of my blog this year. To me, that's anecdotal but the investors should be satisfied, no? Would they prefer that I then spend all my time interviewing all the marketing hires for the year (not possible, by the way, it would be my full-time job)? See the thing is, we do have team metrics. But what you have to consider is that blogging is part of an integrated outreach program. It's one spoke in a wheel. If I am doing my job, my target blog reader is also seeing me via their professional organizations, they are talking on the phone to Maria, they are part of my Jobster network, they respond to job postings that I facilitated through deals with theLadders, they are meeting recruiters at events, they see announcements about Fortune's best companies to work for (and so on). Our metrics look at whether we influence hires through our meta efforts(meaning we get to keep our jobs), how we impact the best hires (top performer analysis), but to try to carve of one relatively minor piece like blogging is silly. It's like measuring "talking" for recruiters. It depends on what you say. You are measuring quantity but I care about quality. As I keep mentioning, blogging is not a sourcing tool, it's a brand building tool. Hires are the bonus.

Jeff, you mentioned that you blog on your own time. I think where we aren't connecting is that you are approaching blogging as a business leader that wants to decide whether he can justify his staff blogging at work; I'm approaching it as a blogger who has the luxury (meaning trust of my management) of deciding whether blogging is an effective part of my job (it's the fact). We don't participate in the same blogging microcosm. In my environment, I'm trusted/empowered to make decisions about what work to do to accomplish my goals. This is why I feel so free to blog about myself (personally). I'm not hearing the same thing in your scenario (not necessarily meaning EA, just the scenario you describe)So, I'll say what I have said before: blogging is not for everyone; a successful blogger MUST have the trust of their management to make decisions about their blogging activity (meaning IF they blog, WHAT they blog about and how often).

My advice: if management is intensely interested in having control of blogging activity, then don't do it (bloggers, run, don't walk). managers/investors, take your money and put it into something measurable so you can have your needs for quantification met. Because the metrics turn a blog into something completely different. If it's about filling individual jobs, invest the money in your career site, buy some LinkedIn job postings. But don't tread on my blog. To put it bluntly: "hands off or the blog gets it!"

By the way, I love that we can share our thoughts about this (I know that we are both very passionate about the subject, though I wish I had Martin's eloquence). You know I have a ton of respect for you and your intellect and opinion. We are just miles apart on this. At the very least, this discussion has really made me appreciate my chain of command and the corporate culture at Microsoft that allows me to do my work. Because the scenario you describe represents something very different.

[Jeff Hunter]

Heather -

Thanks very much for your great comment. You are right - we are worlds apart. I'll let our respective posts and comments speak for themselves.

One thing that I want to make clear: I started blogging before I got to EA. I don't blog on EA's behalf, and I am not held accountable for my blog achieving anything for EA. However, in the things that I am accountable for, I like to have proof that I am actually executing. That has nothing to do with my management - when I have been a CEO I wanted the people who worked for me to have proof that I made their lives better too. We can agree to disagree about that, but that is perhaps where the gulf between us is the widest.

I look forward to seeing you down at ERE! And thanks again for the comment.

[Martin Snyder]

Folks thanks for the kind words about my writing- it's kind of a Parlor Trick because I use Dragon Voice Software much of the time- its quite surprising how different the syntax and flow changes between speaking my ideas and typing them. Also I was born with a bizarre left-brain only configuration, so I can barely do simple arithmetic and the idea learning of a foreign language makes me queasy. No wonder I have in inherent distaste for metrics !

I'm still not that sure that Jeff and Heather are not more in agreement than not. Heather brings up the key qualifier with "meta efforts”: the sum of her and her team's impact. She is saying that her blogging takes up an immaterial percentage of her work time, and that she does not really need a precise measurement about impact because she knows its important from several independent results, any of which exceed the value of that time. Jeff comes from the standpoint that if the blogging is using a material amount of time, than it must be justified, because it’s simply un-businesslike to use resources without the best information regarding the ROI situation.

I think its possible that if Heather would come out and say that her blog takes up two minutes a day and has no marginal cost to host, and that she is finding those two minutes by drinking one less cup of coffee, Jeff might say, yea, its possibly not worth measuring. Now of course it takes more than two minutes, but it still may not take more than 15 or 20 minutes, and you can find the borders there that you wish- as a management exercise.

Which is another key point raised by all the good comments here; how your firm management feels about the blogging is a key factor. This speaks to the actual ownership of the blog. Jeff’s blog clearly ‘belongs’ to him 100% because he does it on his time, under his sole control. It may help EA in any case; I think more highly of that company because I know Jeff through his writing, but its still his blog. I have the luxury of directing some of my own time (please note the necessity of that arrangement as otherwise I might lack employment) and I don’t expect any benefit to explicitly accrue to my company, although some very well may. Heather also has great latitude to determine the mix of her “meta efforts”, but clearly she is blogging for Microsoft, on their time, and they certainly would and could have some input to that blog’s shape and future. It’s a very individual situation.

Jeff you said something provocative about art in agreeing with Gates that “real artists ship”. I could not disagree more. Real designers ship, and real decorators ship, and SOME real artists ship, but many real artists could give a crap about shipping, and rightfully. How much of our cultural heritage has arisen from people who did not see rewards in their time? Plenty. How much of the genesis of individual artistic skills and output was not based on expectation of economic reward? I’d venture to say that quite a high percentage was.

Now blogging may be an art for some, and a product for others- but it’s again an individual thing, and thus the metrics attached are as well. In my case, I would not have encountered Jeff or Heather or so many of the great people who I have the pleasure of sharing ideas with if it had not been for the blogging that I do. There is no ROI question in my mind at all- with or without any metrics to back up my conviction.

May we re-define this conversation in terms of intent, scale, and ownership in relationship to the need for metrics? I would guess that we would probably all be very aligned once the parameters of each situation were a given.

[Heather]

Check out Martin bringing people together. I want to be martin when I grow up or...better yet....Martin for President.

Anyway, Martin, you nailed the common ground. It depends on WHY you are measuring. Is it for a go/no go decision or is it to evaluate how to blog. For the former, sometimes you already have enough anecdotal info to make that decision. I personally think the latter is wrong (or in Jon Stewart's words, "it *hurts* America"...I joke). It's a blogging integrity issue for me.

Jeff-with regard to Ian, can't you just go to his manager and say "Ian is doing something great and really innovative that will have huge impacts to our employment brand on campus over the long term. He's a great employee who I think we can trust to evaluate opportunities to reach his target talent pool; let's let him blog for work". I don't know...now I am kind of feeling bad for Ian (the idea of seeing something that can make a huge impact and then having to justify it with numbers that don't really represent it's value...in my opinion).

Well, this is the kind of interesting debate I was looking for. Kind of wish it was happening on my blog but, oh well (big sigh). ; )

[Brian Toland]

So how do you quantify me? I'm a recent addition to Microsoft but am definitely not a marketer. On the other hand Heather's and Gretchen's blogs were a key reason that I was willing to commit to a very, very long commute. It was only one factor, but I decided that any company that was aspiring to have such transparency in its recruiting efforts - I wanted in.

To Martin's points, Heather's demonstarted influence has now spread beyond marketing and is affecting executive search efforts in the tech space for Microsoft. Before this comment, how would that be measured?

Since her blog is a 'marketing' blog (it's really much more), does her contribution to my decision have less value. As I reach out and recruit high impact tech players to Microsoft, how do we quantify how much credit goes to Heather and should that have any influence on whether she gets to keep blogging?

To Heather's point, you can spend a lot of time gathering a lot of data points and still come to the same decision of doing a managerial gut check (much quicker, less defendable to investors) of saying 'does this add value? Keep doing it.'

I love the time Jeff has taken to explain the thought process that you could take to determine the value of a blog. He's done what a good conversation can do and made me more open to his point of view.

This is what the conversation should have been in the first place.

[Jeff Hunter]

Brian -

I take back the comments about the funny hats. Nicely said.

I think I am starting to understand the hub-bub. It's the term "quantify" that has this discussion wrapped around the axle. Since you both proved and questioned my point in the same comment, I thought I would take some time to explain.

Quantify has two definitions.

Quantify (http://dictionary.reference.com/search?q=quantify)

1.) To determine or express the quantity of.
2.) Logic. To limit the variables of (a proposition) by prefixing an operator such as all or some.

I am dealing with the second definition and have left you with the impression that I am advocating the first at this point in time. I am not.

Brian you said “To Heather's point, you can spend a lot of time gathering a lot of data points and still come to the same decision of doing a managerial gut check (much quicker, less defendable to investors) of saying 'does this add value? Keep doing it.'”

Expressed another way, you said “Does this add value? None, some, a lot? Rudimentary analysis says ‘some.’ “

Therefore you have quantified. You have limited the expression of variables in the equitation. This was my point from the start, but I expressed it as “existence proof.” At the start of any system we seek to understand a binary equation: value or none? Value is determined by the investors (I don’t mean shareholders, I mean investors – as in people who commit their capital or talent in expectation of a return that they determine to be of greater value than that which they provided). Heather’s management (the investors) believe that Heather’s blogging has value. Why? It could be because of comments like yours, which establishes an existence proof and justification of the hypothesis. It may be that an investor wants more (as in the first definition: the investors wants quantities), and then the individual blogger will have to determine whether they want to "produce" given those conditions. But every single one of us who is involved in blogging is in fact measuring value: either none or some. That is a measurement. And because it goes from “any variable at any level” to “some variable (value) and some level (some)” it is a quantification.

I know this may all sound a bit esoteric, but it is important to understand. All innovation (and the tool of blogging represents an innovation is the scope and ease of the personal press) starts this way: somebody with passion and commitment determines the value they derive from the exercise (as in "it feels good" or “a way to express myself” or “building a community”, all of which are expressions of value), somebody else finds value in their output (but perhaps for different reasons) thus creating a market, the market attracts capital (as in dollars and talent), the capital providers demands rationale for continued investment (as in “I get value out of return on my investment, not in the spiritual fulfillment of the product itself"), measurement begins, the innovator believes that the measurement destroys the engine that delivers the value at which point the market either succeeds or fails based on the contribution of other producers who are inspired by the original innovator's contribution.

I have lived through this cycle many times myself (and in fact am living through it now with my blog). In some ways that is what is happening here: people who are blogging because of their passion and commitment to their vision believe that if dunderheads (like me?) demand that there be some measurement of the activities' relative value / success that the very nature of the possibility of that value is destroyed.

Sometimes the innovator is correct. Many times they are wrong. The innovation always changes in response to it's introduction to the market, and what it changes into may not be interesting or compelling to the original adopters of the innovation. Therefore the early adopters claim defeat. But the innovation continues to be important, but in different ways.

The wonderful thing about a market as huge as communication and media is that the tools used for communication (in this case blogging) are unlikely to destroy the desire to connect and create a loosely affiliated community no matter who employs them in what manner. Just because the printing press was used to create the National Enquirer doesn’t mean that nobody cares about the Guttenberg Bible.

Thanks again Brian. I am sure I have just clouded up the issue more than clarified it, but your admission that Heather and Gretchen are part of the reason that you take the long community is an admission that blogging has measurable value. We may not yet be able to count, but we are surely limiting the variables.

[Brian Toland]

Should we tackle World Peace next?

When you talk about metrics, I inherently think about numbers (page views, unique visitors, etc.) Now that we're agreed that we are closer to establishing metrics in Martin's more amorphous realm (circles of influence, impact of an idea - e.g, transparency in recruiting at Microsoft) than I agree with you absolutely.

If you can get to the determination of value by saying:

1. Does it have no value? No, it clearly is have concrete impact on perceptions of Microsoft, so it has 'some' value.

2. Is it efficient? Yes, since one persons blogging has affected the decision process of six others, who likely would have chosen otherwise.

3. Is it worth continuing? Given the relative low cost of continuing this blog rather than the associated time cost of building a community by other means, than yes.

No hard numbers are attached yet we've 'measured' that this particular blog is worthwhile.

Thanks Jeff, I'm a convert. Halleluiah and pass the fried chicken.

PS - Show this thread to Ian's Boss and if the decision is no go on his blogging, I'll hire him here at Microsoft.