Making Your Weights Great: Lessons Learned from 30,000 Downloads

In Making Your Docs Rock, we discussed how documentation quality relates to a model's popularity after analyzing 1,000 public ModelCards. This week, we're revising that thesis after one of our models captured the community's attention with more than 30,000 downloads over the weekend.

In the graph above, SpaceQwen spiked in download counts just as we announced the new SpaceThinker reasoning model beats gpt-4o on quantitative spatial reasoning benchmarks, and again when we shared an unrelated meme.

Surprisingly, the SpaceQwen model was NOT the new SOTA result with the top-notch docs. So what happened?

As our SpaceThinker model announcement garnered visibility from the r/localllama community, they could choose to download it or one associated with an earlier result, which had already established 1,000 monthly downloads.

Even though the older, more popular SpaceQwen points to the new_version, SpaceThinker with documented SOTA task performance, the incumbency bias favoring the tried-and-true is challenging to overcome.

With the benefit of hindsight, documentation quality is a less influential factor to users of your weights deciding which to download than existing popularity signals like download counts.

Of course, this points to a popularity bias affecting model selection. Due to this halo effect, we also observe a feedback loop that exaggerates the incumbent's lead. The more users download a model, the more it gets downloaded. Viewing the Hub as a popularity-based recommender for model discovery, new models suffer from the cold-start problem, where a lack of prior user engagement can make it harder for new models to gain initial traction.

However, the AI community seeks to move beyond reliance on simple benchmarks and popularity metrics. We see that sentiment in r/localllama is a leading indicator of model traction as the community becomes hyped about the narrative around a new model. Conversely, the monthly download count on Huggingface represents a lagging indicator of adoption, pointing to evidence of past user enthusiasm that has already converted to use.

How can we close the gap between users and model makers, scale community insights, and facilitate feedback that leads to improvements? It begins by recognizing that usage-specific context is much more relevant to making the best choice.

In the future, model artifacts will link directly to the social context of community feedback, like taster's notes from the model sommeliers, facilitating shared discovery. Likewise, HF spaces to demo models will give way to data flywheels, bringing makers and users closer together through a deeper understanding of its application, grounded on the data.

AI users want to participate in a moment of shared discovery. But how do model makers build for enduring relevance and cultivate a loyal user base?

As millions of models saturate intelligence benchmarks, users are left to ask: What is your movement? What does your AI exist to advocate for?

Going beyond the model of the week to become a cultural touchstone with a highly engaged community requires more than climbing the benchmarks. It's about building trust through greater transparency.

That's why I'm excited about building AI in public. Viewing AI merely as a product launch misses the opportunity for continuous improvement through community-driven feedback. Your AI can transcend the purpose of an intelligent tool to reach broader cultural relevance by engaging users in the story of how you construct delightful AI and sharing your discovery and methodology.

I expect more model makers to share their findings contemporaneously rather than packaging them as post-hoc analyses and technical reports censored of key insights. Transparency will also extend to data sourcing and curation.

As model and data artifacts have become commoditized, the way you combine and present these ingredients will become crucial to your AI's enduring impact.

So what is the legacy of your weights?