Work Stories | Matthew Devney

What's This?

A resume is intentionally compressed. Interviews are also short on time. This page is the long-form version: what the challenge was, why it mattered, how I approached it, and what happened next. Think of this page as the interview itself, but with enough space to tell the full story.

Click any role to expand the context.

Fast Facts

1,700+

Security findings remediated through an AI-assisted SAST program.

30%

Annual cloud cost reduction from audit and remediation work.

~40 hires

L1/L2 operations team members hired and trained as founding lead.

25+ years

Production systems, DevOps, site reliability engineering, and incident response leadership.

Tools I Have Worked With

Tell Us About Each Role

Customer Success Engineer - L2 Team Lead

Nike had a problem. Nike has literally over 1,000 technical teams, and they generally need similar things. Every team needs a kubernetes cluster (for example), so every team spins up a kubernetes cluster, and now we have 1,000 kubernetes clusters all configured differently, and who knows whether they're patched or secure or use proper RBAC or anything, really.

They called the solution "Platform Strategy" and the general idea is Nike is going to create a management interface so when a team needs a kubernetes cluster, they clicky-click through a form and a cluster gets spun up for them. This cluster starts in compliance with all the corporate policies, and it's maintained centrally so it remains in compliance. We had 14 of these platforms.

Support is part of any mature service offering, and that's where I came in. They created a "Customer Success Engineering" team and I was the team lead. They told me they were going to hire 40 engineers and I don't want to train that many people, so I created a handbook of how to be a Customer Success Engineer: all the access we need and how to request it; CSE-focused documentation on how to support our 14 platforms; I documented polices about how to interact with customers and dev teams, but first I had to create those policies.

I interviewed over 100 candidates, and we did hire about 40 engineers in 6 time zones. The other major focus of this role was training and mentoring these engineers. In my role as Lead with decades of experience, I tried to pass on as much knowledge as I could, but more importantly I tried to teach these engineers modes of thinking that help with troubleshooting and problem solving.

At some point a corporate decision was made to outsource this work, so I smoothly passed the whole enterprise off to an external contract partner and moved on to my second role at Nike.

Lead Software Engineer - BAAT team

After a short stint on the GitHub team where I implemented some Actions to automate housekeeping, I landed on the Build And Agile Tools (BAAT) team which manages Jira and Confluence and a third Atlassian app Crowd which helps to manage those. My main focus was the Jenkins pipelines that roll out these apps.

Over the next year I added features to these pipelines: blue/green deployments, proper tagging, unit tests and code coverage, and of course optimizing for reduced cost. I was able to bring these pipelines into 100% compliance with all applicable corporate requirements and standards, and reduce cost by 30%.

Once I implemented code scanning our code scanner identified about 1,700 vulnerabilities across all the pipelines. I found AI to be very useful in resolving these, and quickly got the number down to 0.

Corporate decisions above my pay grade put our team in charge of Miro, Lucid, and Figma, three near-identical whiteboarding SAAS tools. Once again there were no processes, so the first thing to do was work out who does what and how things like escalations are done. The next thing was just to get a handle on the tool itself, so I used Cursor AI to create several dashboards to display availability and license info.

Bloomfire just had not concentrated on their software infrastructure for a while and it was all creaking with old age.

The version of ssh that they were using was so old that all the ciphers it supported had been deprecated, and chef-solo couldn't ssh to modern servers. Each time they spun up an instance they would manually ssh to that instance and enable old, deprecated ciphers.
The version of chef-solo they used depended on that old ssh, so they couldn't upgrade ssh. Also it wouldn't run on modern ubuntu because of openssl and ohai version incompatibility, and some dependency headers mismatch modern libraries.
The product apps were set up to start from Upstart, which was unavailable in modern versions of Ubuntu.

We called this the chicken-egg-egg problem, because each of these 3 parts (ssh, ubuntu, chef-solo) could easily be upgraded on its own, but that would break the system unless the other 2 were also upgraded at the same time.

First I switched the product apps from upstart to systemd. This theoretically allowed our apps to run on Ubuntu 18, if everything was set up properly. It was set up by chef, which wouldn't run on modern Ubuntu.

So I re-implemented it in Ansible. The head engineer wanted to move to ansible anyway, so I spent months with 2 windows up on my screen: reading chef on the left, writing ansible on the right. This got rid of both the ssh problem and the chef version problem both at once.

Once I had a minimum viable subset of chef re-implemented in ansible, then I did the ubuntu upgrade in production to address security concerns in the old ssh, old chef, old ubuntu. Then I completed the chef-to-ansible re-implementation project.

I completed one other major task here. The running installation of the SaaS product was running in a bespoke VPC hand-crafted years ago by the head engineer and added to by hand ever since. I created a series of rake tasks and terraform modules to roll out a new instance of the product in a different region.