How do we measure productivity with devs in this new age? What is the purpose of that measurement? Do we have a goal or habit that we want to promote with this metric?

I had this thought because Shopify (from what my intern friends there told me) had full on leaderboards for AI usage and it was seen as desirable to use the most AI possible. My first thought was that it makes no sense. A terrible measure of productivity because AI use does not translate into real gains. After reflection, I asked myself what the intent behind that metric was. Maybe they wanted to promote AI use because they believe that familiarity with these tools will benefit everyone, so they're incentivizing the use of AI tools. Thus, the metric is less for productivity but more an incentive to gain skills through repeated practice. Talking to a Product Manager at Shopify, I learned that this is a large part of the story. Another part is that AI truly does give productivity gains compared to no AI. So if someone's work is falling behind, their performance review could look at their AI use and ask why they aren't using AI. On the other hand, someone spending too much can be talked to because they clearly aren't getting enough out of their usage. Rather than a metric for performance like it was initially explained to me as, it was more of a metric that supports performance reviews.

In my opinion, productivity cannot be measured well. That's why we have such misaligned metrics like lines of code. No one knows how to measure productivity. I believe that the "most optimal" metrics would involve slightly more friction on the human side but lots of gain.

One idea relates to having human-generated (or AI-generated with human verification) tags/ratings/measures on tasks/PRs. By having some numeric (or even textual) code like on a scale of 1-5 (or "minimal" to "org-wide impact"), we can better estimate how much impact someone has. The trade-off here is that people can be doing important work but it simply isn't at the top level all the time. After all, even "minimal" impact work needs to be done, so you shouldn't think badly of someone if they're always stuck with that work. If you look at it in another light, perhaps this is a good way to keep track of what people are being assigned and realize that you need to give everyone their fair share of impactful work, or at least distribute it better.

Another idea involves regular check-ins with a manager or leader. AI can be used to pull relevant context related to an employee's work the past x days. A leader can then decide based on what they've seen from that employee and the summary as to how well that employee has done the past week. It can be as simple as "needs improvement", "doing well", "exceptional", aka a "you should do more", "this is expected performance", "you're going above and beyond". Here, AI tools are used to reduce the burden on the manager and hopefully make them take minimal time to decide how an employee has done. The more context the better, meaning it should be able to pull from all relevant sources about this person's work.

Honestly, these ideas are just that. At the end of the day, I don't know if we'll ever get close to a perfect solution, because productivity is often found in the intangible. At the very least, when it comes to business outcomes, I think there's enough traceability in emails, meetings, code, and PRs to at least attribute some success to the people involved.