Some thoughts about games and pioneer.
Shared by fcomplexity.com · 8d ago · 5 comments

As far as I understand, the game at the core of pioneer is that in order to perform well, a player has to show progress. The rate of progress is determined by peers (i.e. other players).

However, peers themselves have very little time to deeply review a project, therefore a players' progress ends up being determined from the players' explanation of the progress and a cursory evaluation of the project's landing page. This means that the entire game is optimizing for better landing pages and ideas that resonate with the peers.

This leads to the same problem that most research deals with in peer-reviews. Original ideas will rarely come out of conferences that employ a peer-review mechanism. Most interesting ideas are presented as journal-first papers (where ideas are evaluated by editors). That's the general trend in PL and software engineering research.

Therefore, a few recommendations.

1. It might be better to think about reducing the number of progress evaluations from 10->5->3. This will give peers more time to actually evaluate the project.

2. Like in a journal paper submission, add a high-level player (, ideally someone from the pioneer team,) to serve as a super-node. The high-level player conducts the peer-review on a weekly basis and has the final say about the projects'-progress. Moderation of peer-review, I guess, is the goal here.

3. The hot-or-not style game is a good starting point, but I would try to experiment with a rank-based evaluation. In a rank-based evaluation, peers see 5 projects and answer questions about them; in the end, they rank the projects from 1 to 5 with 1 being the highest rank. This will give you more fidelity in terms of actual performance.

cheers,
eff

AndyDent-Touchgram · 2d ago

the entire game is optimizing for better landing pages and

I often include links to longer form explanations especially in my ask and that has led to higher-quality feedback. I encourage many people I review to do similarly, especially if they haven't had time to create a landing page or it is very simple.

ideas that resonate with the peers

That's a proxy for ideas that resonate with the public assuming there's some significant overlap between your peers on Pioneer and the market you're targeting. Unless you're in a very specialist domain, the ability to get that instant resonance is something I rate very highly. It's very close to the fabled elevator pitch or the what do you do? test. If I'd honed my answer to the degree I have through Pioneer, I might have caught the interest of Bill Tai when I had my 10 second encounter with him a couple of years ago.

The high-level player conducts the peer-review on a weekly basis

That high-level review does occur periodically for people who have climbed the leaderboard. You're trying to improve the very broadest part of their funnel.

The hot-or-not style game is a good starting point

It's not the starting point - Pioneer has been evolving for a few years. Do you have any references comparing a rank-based evaluation to the pair-wise comparison? Note that Pioneer has significant logic behind the generation of the pairs - they are not completely random. Thinking about a rank comparison process, that sounds like a lot more work for us reviewers. It also would make it even harder to compare startups at radically different points and domains - are you proposing that instead of generating roughly-matched pairs, that Pioneer should generate sets of five?

andrew-miit.co · 8d ago

My take is that the game is designed to replicate the reality of the startup/fundraising environment. Every Monday we get a little taste of what VC's see every day - dozens of pitches with very little time to perform in-depth analysis and needing to make a decision to pass or promote. They have to look at the limited information that we do: landing pages, clarity around the idea/description, basic metrics, trajectory, ability to do what you say you're going to do.

So by optimising for players to succeed in the game they're also optimising for players to succeed out in the wild.

Just my two cents.

manojranaweera · 7d ago

Very good analysis Andrew.

I'm here for two reasons. One is to get feedback and improve https://skilledup.life. I got a lot of feedback - thank you. Next is to hang out with few other startups. I already have 27 tech startups I look after from the UK (https://techcelerate.ventures/members). But unfortunately, I don't have this type of engagement facility.

gandalf-016894 · 4d ago

Yes as a player I would tend to believe your affirmations.
But I would guess Pioneer people are not in the theoricresearch. More like business : they are entrepreneurs. And as an entrepreneur, I suppose you prioritize the 'cost reduction'. Hence you ask pioneers players to give 20 reviews per week, not 5...
And do not forget : Pionner is clearly free-to-play, so we can say it's absolubly legitimuss from them.

manojranaweera · 7d ago

Seriously, you should build your product for your users and customers and not to please anyone here. I am even against the idea of building MVPs. I prefer V1. MVPs are introduced by investors to simply tick off their checklists.

Then every Tom and Dick think they have an investible proposition. Most tech startups aren't investible.

Please build what you are building for real users and paying customers. Thank you.