How Will LLMs Change the Shape of our Product Teams?

Amidst the fervor around AI, I wanted to look at the changes we may start to see to the shape of engineering and product teams building and delivering software due to the use of LLMs (large language models) in the design and development process.

These are my opinions on related questions: what will software delivery start to look like as we utilize LLMs in practice? Will the human roles we've taken as standard be eliminated, augmented, or elevated?

My conclusion is that the practice of design as we know it today, be it in software systems, product experiences, or otherwise, will be the essential role on product development teams – those skills and that philosophy will be of tremendous value in the coming years as value in other roles erodes and software is increasingly designed for LLM collaboration.

Synthesis and Customer Facing Roles

Starting with our outward facing roles such as product management and research, all of this software we are tasked to build is to solve human challenges, not the challenges of LLMs, and requires discussion with those same human stakeholders to direct the solution and gather requirements. This role of synthesizing the problem being solved and directing the architecture/design is not going anywhere. For example, an application to help hairdressers manage appointments will still need humans in the loop to design and shape the core product experience, given it will interface with both customers getting their hair cut through integrations or experiences and the businesses cutting hair. I think it is safe to assume that our human behaviors and limitations will still be the lowest common denominator in the economic engine.

These roles that require synthesis of the problems our product teams set out to solve will be increasingly relevant; they will collaborate closely with architects and system designers, speaking to users and shaping the design of the solution. "Product owners", product designers, researchers, sales people, and industry experts will utilize LLMs in parts of their roles, but their main goals and workflows will remain the same. I do not feel LLMs are well-suited for this type of work, given it is highly empathetic, experiential, and emotional.

Collaborating with customers or prospective customers is also an opportunity to build trust and sell your solution – again, re-enforcing human connection and empathy will continue to be prized in the product development process.

These team members that operate from design principles and can collaborate with customers in a sales-minded capacity to feed into the design of the software are not going anywhere. The product designer, product manager, or researcher are extremely well positioned. Their influence and formality around their role will continue to increase.

Software Architecture and Systems Design

I expect the role of "architects" or software engineers and designers who design the fundamental shape of the system to be elevated in their role and be the least likely to be collaborating with LLMs on a day-to-day basis. Their role will increasingly shift into a systems design focused one; collaborating with the customer focused roles mentioned above and formulating what a system may look like that could solve the stated problem.

Specifically, understanding inputs, interfaces, and system requirements, and producing the outlines of a system that other engineers and designers can collaborate with LLMs to augment and fill in. This will be critical for maintainability in an LLM-collaborating team downstream – teams will want to ring-fence areas where LLMs are heavily utilized.

These systems and guidelines won't be unfamiliar to us. The role of “integration engineers” or engineers that strictly utilize specific APIs and SDKs will fit closely with the patterns that will evolve. We will increasingly design systems like this with stronger guardrails around contributions to “protect” the maintainability and coherence of the system in the long-term, and let downstream collaborators reap the productivity benefits of LLM collaboration. This strategy is clear in the release of ChatGPT plugins, a system designed for LLMs to be used by engineers to do the majority of the labor. For many software engineers working today, they will need to lean into LLM collaboration to increase their productivity or “move up” the layers of abstraction in systems towards a design role.

Integration and Development

As laid out above, within a well-designed system, we will soon see the value of “filling in the blanks” with LLM collaboration and see incredible improvements to productivity and team performance. Endpoints, forms, duplicative pages, third party integrations, and other work with strong guard-rails in a well-designed system will be prime candidates for LLM collaboration. We may see fewer engineers contributing directly to this work, and increasingly see them synthesizing and validating LLM-generated implementation in smaller teams as designers, not programmers.

These engineers will be extremely productive only if the system is appropriately designed and the outputs can be effectively validated; highlighting the other roles of synthesis, architecture, and validation.

Validation and Quality Assurance

The traditional roles in quality assurance have already been automated heavily by human programmers; that pattern will only continue. LLMs will be exceptional at fuzzing (in more sophisticated ways than today) for security and quality, mimicking even our best QA testers.

However, with much of the original and creative work being done in the stage of synthesis and design, and a large amount of surface area being driven by engineers collaborating closely with LLMs, the role of validation and QA becomes increasingly vital in a different capacity than is commonly seen today.

These teams will look less and less like they do now, and shift more to being roles that are filled by product designers and managers and require a large degree of empathy and understanding of the user. There are quality assurance teams already trending in this direction — they are ahead of the curve. I expect they will collaborate with LLMs more than the design/synthesis teams, using the comprehension of LLMs to help with validation, but they will also need to stay grounded in the original goals of the project to ensure the solution actually matches the human problem being solved, and doesn't just pass the resolution derived from the language model.

QA engineers will want to move towards a deeper understanding of the broader goals of the system, towards architects and designers, and play a key role in validating the product does what it was set-out to do. This may or may not come with the same title: these roles could very well just be ownership or design roles in a simplified team.

Education and Training

End-user and internal team education can be a defining requirement for the success of a project. Engineers and designers need to fully understand what they are being tasked with building, what the expectations from the end-user are, and how to work within the architecture of the solution being designed. Customer facing roles will need to deeply understand the product, and learn about it as it is deployed and sold to customers. And importantly, end-user customers will continue to need training and education on how products work and can be utilized.

For engineers, this may look like an understanding of integration surface area, like an SDK or API that they are integrating with in collaboration with an LLM. For an end-user customer, documentation or guidelines for how to use the system.

LLMs can improve on the status quo in this area as a collaborator for all parties: as one of the best onboarding buddies you could ask for. Though they will not derive the answer for the perfect design of the system, with context on the surface area they can guess and guide implementation, usage, and spur human creativity by making non-obvious connections or writing integrations.

One great hope I have is that LLM collaborating teams can level the playing field for collaborators of all backgrounds and disciplines and make human collaboration on the design and validation of the system more inclusive and effective within complex domains. As a new customer or team-member onboarding onto the product, having an interactive and expert knowledge-base for the product available will be extremely powerful in driving productivity and job satisfaction.

Organizational Charts

How might your software team's organizational chart look like with an emphasis on the roles above? This first diagram is an example of what a team might look like today.

And the second, what roles may start to emerge and what roles may change.

The above diagrams imply some role elimination, as my guess is our ability to protect this class of labor will be increasingly hard, and our teams will look far leaner. I'm unsure of the impact that will have on our livelihoods. I'm not sure anyone has that answer currently.

My Conclusion

Our teams will not be supplanted by LLMs; but we will certainly begin to augment and replace aspects and some roles in our jobs. Software teams, always the early adopters of software tools, are already integrating use of LLMs with the mainstream usage of ChatGPT, GitHub Copilot, and countless open source solutions, and we should start proactively thinking about the impact this has on our organizations. I hope we can view LLM collaboration as a tool to be utilized and not one to fear; but balance that with reminders of whom we are building all this software for in the first place.

If I'm at all right about this I'm left wondering what parts of my personal skill-set do and don't fit in this version of a team. My sense is that design as a practice: be it in software systems as an architect, visually or experientially as a product designer…that version of creativity, expressiveness, and thoughtful outlook will not be supplanted or replaced. I wish the design minded product people all the power in the world to build us better software – and it's going to happen a heckuva lot faster.

Written by Jack Pearkes on March 24th, 2023. Thanks to my friends for reading early drafts of this.