I work on the Product Security team at a large US corporation that provides (amongst other things) IoT solutions to the worldwide water utility industry. This is a outline of how I came to be in this role, and what the first year in critical infrastructure-adjacent security looked like.
How did I end up here?
In 2021, I was a software developer. I’ve been involved in software development for almost my entire professional life, starting in help desk and documentation management in 2006 before moving into development in 2008. Add that up, and that’s a lot of years writing a lot of software in SMEs.
I keep trying to recall how I ended up interested in security and when that happened. If I look back through my own writing history, in 2015 I wrote about how email addresses had become de facto usernames and the risk of automatic deanonymisation in services that didn’t require it (see also my whoami page). So I was already thinking about privacy in 2015. That was also the year I wrote a screed on biometrics and passwords and how the two were not equals and the dangers of them being thought of as such. I’m fun at parties.
I might not have a Hollywood-style tale of a life changing moment, but it probably started in 2014, through a mix of reading Troy Hunt and SwiftOnSecurity. Note, that’s a Mastodon link. Tay has a large historical body of work on the other site, but we don’t talk about that place for now.
In early 2020, my employer asked me to step into an information security role alongside my existing software development responsibilities. My subtle and not so subtle hints to that point about my preferred career progression had apparently worked, and I received the nebulous title “Security Architect” along with some equally nebulous responsibilities that to a certain extent were mine to define.
Truth be told, what they really needed someone in that role for was ISO 27001. They had been certified for several years up to that point, but recent staff changes had meant the organisation’s information security management system (ISMS), the cornerstone of ISO 27001 compliance, had been left without the care of someone with the required domain expertise to maintain it properly.
ISO 27001 is the gold standard when it comes to internationally recognised information security standards, and I’d just inherited an in place system as an additional responsibility, no experience, and a few hours a year of advice from a consultant. Somehow, we made it through our external surveillance audit and recertification.
But I wanted to do more than just keep the organisation compliant. I saw opportunities for improving the existing program, making the ISMS more relevant and specific to the company’s ways of working, and a myriad of ways in which implementation of security in software development could be improved. I was able to build a convincing enough case that I ended up with two days a week dedicated to security work, with my employer back-filling the gap that had been left in the software development team.
In 2021 we were gobbled up by a large US corporation via acquisition which, despite having a sizeable IoT platform offering for water utilities, had no meaningful product security program. While I was originally offered a place in the cyber security team, a recent external audit by one of the big names had highlighted the lack of any meaningful product security program as a significant risk.
The existing security team was given the task of focusing on information security, and a new team was established at the beginning of 2022 to concentrate specifically on product security. It was a team of two, and I had just become 50% of that team. Within five months, we would double to a team of four, our responsibilities covering the security of all smart infrastructure products for customers.
Priorities
I had been plucked from a small and highly productive software development team in a dynamic investor backed startup, and dropped into a brand new security team in a billion dollar US corporation. It was just me and my manager, the Head of Product Security. We were both new to the business, and responsible for implementing a security program that would protect the myriad of solutions that we offered.
But how do you even begin to tackle a problem like that? We didn’t know how many engineering teams there were, never mind how many individual engineers. And as for the products that the company sold, we were also in the dark on that.
Any good security program has to start at the beginning, and the beginning is typically comprised of building out risk and asset registers. You can’t go about protecting assets that you don’t know about, and you can’t prioritise the protection of those assets if you don’t know what risks they bring to the business.
Those first few weeks and months were comprised of discovery. We had to start with the basics. What products and solutions do we provide? What are the business’s priorities? How many engineering teams are there? What do those teams do and who is on them?
The answers to all of these questions were made more complicated by a long history of acquisitions, changing business direction and strategy, and outsourced contractors that were often almost impossible to tell apart from permanent employees.
We started this work in January and it would carry us slowly and painfully right through into April, as we picked our way through identifying key stakeholders, hardware, software and cloud solutions, and began to develop solid contacts in different teams. A lot of this work involved talking directly to people well embedded in the business, tracking down architecture diagrams, and building out our own more security focused documentation such as trust boundary diagrams and data flow diagrams.
Alongside this practical work, we were simultaneously beginning to build out and establish governance practices that teams would have to adhere to when developing solutions, from design and architecture choices, right through to solution maintenance and eventual decommissioning.
Our external auditor had seen what was in place for product security in mid-2021 and determined, quite politely, that we should start at the beginning, with governance.
Governance
I found those months of discovery to be exhausting and often wondered if I was really making any progress at all. Every time I answered a question, I would find two more, and as the weeks went by my to do list only grew larger. There were tasks that sat on that list for weeks at a time, pushed down the priority list by other things, and I was starting to realise that a lot of the items I might simply never get done.
The experience was in stark contrast to what I had grown accustomed to in the previous six years on the software development team, where a particular work item could be split into a set of well defined and discrete tasks that could be worked through to completion, with concrete progress to point to at the other end.
So being able to break up the seemingly endless discovery tasks with governance work actually came as a relief. Governance is about defining how an organisation operates, defining its focus, and formalising what good practice looks like and how to measure it. In the realm of Product Security, this materialises as a set of policies and standards.
Writing policy and standards documents gave me an opportunity to finally unleash the security best practices and knowledge that I’d built up over years of curiosity, but never had a real outlet for. I was lucky in having a line manager with decades of industry experience to help fill in the gaps in my knowledge and point me in the right direction in areas that I’d never covered in depth before. All this while having significant input in what would become organisation wide engineering security practices.
As we built out the governance that would become the foundation of everything we did going forward, we selected OWASP SAMM as the model we would use to measure the maturity of the organisation’s security behaviours. I was able to align SAMM with the model used by our external auditor, allowing us to establish internal targets that would map to the recommendations of our auditors. This was an important step, because it would help us with senior management buy-in and in turn lend us the authority that we might need when tough decisions had to be made.
One of the core challenges in information security is that the more effective your program is, the less visible it becomes. If a company has zero data breaches in a year, then what does that mean about its security team? A favourable view is that a lack of breaches is a sign of an effective security program. A less favourable view is that no breaches means the program is over resourced, or maybe not needed at all. The industry tries to solve this problem with metrics, building out a selection of indicators that try to quantify the performance and value of a security program.
Ultimately, the business exists to make money, not to protect users’ data. That’s simply the brutal reality of doing business. The only reason security teams exist is to protect the overall balance sheet of an organisation. Put simply, any security program must cost less to operate than the predicted losses that would be incurred by not having it. Some of those losses might be easily quantifiable, such as the cost of lawsuits from mishandled data, whereas others might be difficult to quantify, such as reputational harm.
This is the level we’re operating at when we talk about governance. Risk versus reward. Not all risks are created equal and so any compensating controls we choose to put in place will not be equal either.
Afterward
I started writing this in December 2022. Well I never did quite finish it.
It’s now November 2024, and I wanted to revisit this for the same reason I wrote it two years ago. The Product Security team that I joined in 2022 has both grown and shrunk in that time. The organisation has gone through significant restructures, and going into 2025, I will find myself likely flying solo for a period of time as the new Head of Product Security.
Looking at my notes, I had originally planned to write about all the new concepts and practices I’d internalised with the shift in career path. Looking back now, I can see how far I’ve come since 2022, which gives me an optimism and sense of excitement for what comes next.
I never will finish writing “A Year in Product Security”, but I intend to give myself opportunity to being documenting this new chapter, responsible for the department, and what I hope will become a team.
And who knows where I will be when I read back over the draft in another two years.