Does pioneer scoring overoptimize for building?
Shared by christopherchen · 201d ago · 14 comments

One of the pitfalls of any gamified system is when it's gamified on the wrong criteria. The current reward function seems to be very much oriented towards: picking one idea, and sticking with building it all the way through. It's much easier to say, player A made more progress than player B when they clearly demonstrated much more output - but what if that progress was towards an idea that did not have PMF?

In the past 2 weekly votings that I've done, I feel that many very early startups are trying to build, build, build, without really truly understanding the problem they're solving, and that their solution is the right solution for it.

My background is as an engineer, so I might be biased, but I almost always find the building part the easiest part of solving any given problem, as long as the problem is very well-defined, and solution is well-proven.

One proposal I have is that there should be more specific voting criteria, and categorizations of startups based on pre-PMF and post-PMF (though these definitions are handwavey, we could let people self-identify). We shouldn't try to compare the actions of a company without PMF with one with PMF!

For companies pre-PMF, we should attempt to vote on "which company made more progress towards specifying or proving their value prop," and for post-PMF, we can vote on "which company made more progress towards growth."

*growth - where the goal of a startup is specifically hyper growth, as opposed to just sustaining itself.

I'd love to know what the rest of the pioneer community thinks. Does pioneer scoring overoptimize for building, and reward the wrong behavior for pre-PMF startups?

braddwyer · 199d ago

I think I'm pretty qualified to answer this definitively (we've been in the top 5 for 96 consecutive weeks): no.

Different voters value different things. Our best weeks are the ones that are most balanced between product, marketing, sales, and KPI progress. A healthy mix gives voters of all persuasions something to love.

The community does tend to underweight (or even adversely select for) hiring and fundraising. Our updates that mention recruiting or fundraising activities tend to be disproportionately downvoted. This is probably healthy; these things need to be done eventually to grow, but they're not valuable objectives in and of themselves.

sam@novamoney · 187d ago

I 100% agree with Brad.

manojranaweera · 197d ago

Hi Chris

People have many differing views about PMF. My own is that it's demonstrated when you are reaching £100k MRR and not before. I personally do not think it's a good indicator for here, as most companies are very young and unproven.

Personally, I don't like voting up and down. Prefer just voting up and not removing points for voting down.

christopherchen · 199d ago

These comments are really helpful! I'm definitely new to the pioneer game and so these are definitely good thoughts into re-evaluating how I interpret what Pioneer is asking of me. Is the general consensus that when we evaluate, who had more impressive progress, we evaluate progress relative against the KPIs that the Pioneer has set?

I might be misinterpreting something, but does this then reward setting KPIs that are more reachable?

My initial thinking regarding Pioneer rewarding building more is from the weekly voting process being voting with low context. From the outside looking in when I don't know a lot about the stage and maturity, it's much easier to interpret significant progress has been made if a goal was to "build X thing," and it is built, than say, more deeply understanding users problems (though this might be a poorly framed goal on its own). Building just "feels" more significant without context.

All this said, I'd love to hear other folks' recommendations for how they approach voting on progress.

benjamin_a · 199d ago

There are not a lot of guidelines for voting - so the impact of KPIs probably depends on the voter. For example, I personally do not give very much attention to their KPIs and focus mostly on their goals/progress instead. But I will definitely give points to those who set very ambitious KPIs and reach them/almost reach them ("talked to 45/50 users this month). However I do not generally give points for KPIs that were reached but do not seem very impressive ("talked to 5/3 users this month").

Ultimately, KPI goals don't really matter much as it's progress itself that is more important - if someone went from 20%-40% progress on their long-term KPI this week whereas someone else only went from 80%-90% progress on theirs, I will usually vote for the person who made 20% progress above the other person, assuming their weekly progress on other things is roughly equivalent.

fernandocordeiro · 198d ago

I sense that a lot.

I've been with tendinitis on both my wrists lately, so my product output too a great hit (since I can't well, type - or type fast at least). I took the opportunity to talk to users and learn B2B Account Based Marketing... and my position absolutely plunged. I even got 100% downvotes once, which means this was not by random chance.

Of course, it's very possible this is all a projection on the way I measure other candidates progress.

AndyDent-Touchgram · 196d ago

Fernando, try using the dictation features on your phone or computer to rest your typing. I'm a big believer in Tai Chi having saved my wrists. I had problems years ago and thankfully had read good advice on how inflammation works and how rest works - you need to totally stop using an arm for days to rest it. Inflammation kicks back in with very little movement. I strapped my right arm to my side for 3 weeks and had to learn to mouse left-handed.

fernandocordeiro · 196d ago

I tried dictation... but, for coding, it sucks: it fails to understand my English accent to a hilarious degree. For mobile, I often use swipe-typing, so it's not really a problem.

Having it on both wrists really screwed things up. At first, I started to use my left hand too but, because of my setup, my left wrist started to act up as well. Now I have an ergonomic keyboard and use braces on both hands. They allow me to work (at my day job) but I can no longer afford to do code work on Pluckd during the nights and weekends.

AndyDent-Touchgram · 195d ago

Ouch, yeah I take your point about setup. My episode was a combination of using a track-ball and an IDE where triple-click was used to select a line. I was in the most intense project of my life, writing a FORTRAN to Object Pascal translator and wrote about 30K lines of code in 3 months, most of which was copy-paste-modify.

It might still be worth investing in dictation alternatives, if you can find something which tolerates the accent. For a British-born Aussie, son of a Yorkshireman and Liverpool lass, I have a weirdly neutral mild English accent so Siri mostly gets me. Every little bit of rest you can give your wrists helps.

fernandocordeiro · 195d ago

Thanks for the kind words and advice, Andy! Appreciated it!

teeldear · 198d ago

Pioneer has called itself a "generator" as opposed to the common accelerators. I would think a "generator" would optimize for building.

benjamin_a · 200d ago

As either people have pointed out, building is definitely not the easiest part for most startups, and many startups actually don't focus on it enough in my opinion. Someone at YC once said something like "the only two things you should be doing now are coding and talking to users" which I think is true. Everything else - marketing, finding PMF, etc. becomes easier when you have a working MVP and real potential users to interview.

oras · 200d ago

I don't recall seeing anything about building. You set your own KPIs, you write about your own progress, whether it's marketing, selling, new features, pivot based on feedback .. you name it.

There is another point you got wrong due to your technical background. "almost always find the building part the easiest part of solving any given problem"

Tell that to non-technical founders learning to code or using no-code to make a solution they can validate.

I am an engineer too, and I've meeting founders from all over the world in the last 3 months. Seeing how many non-technical founders struggle with the building would change your perspective.

By the way, where did you get the impression that Pioneer rewards building?

AndyDent-Touchgram · 200d ago

I'm pretty sure you're wrong and trying to overcomplicate a perceived problem. I'm an engineering type too. I'm disagreeing with you on the back of over a year of membership, tracking my results every single week.

Pioneer does not set any rules as to how progress is measured - that's your perception.

It is purely up to the individual startup as to what they report as goals to be achieved and how well they met them. I have had weeks where my upvotes were due purely to meeting marketing activities and community outreach, as I attempted to refine my understanding of what people wanted.

I haven't managed to get a new version on the App Store for 10 months, although I've come close, because of an insanely ambitious core engine rewrite. (Thankfully over, I'm just dealing with some cleanup & finally getting into the cycle of UX improvement on top.)