How to convince my lead to take more care of the pipeline?
I am working in a project where we have a pipeline which runs automated tests, lint, and other type checks.
But we are merging PRs even if the pipeline fails 😅
On my case, every time I got a pipeline error, I fix it in my PRs, and some of my coworkers are starting doing the same, but still we are merging some PRs with the pipeline failing.
Our manager is a software engineer too, and has the role of merging the PRs.
I tried to convince him to avoid merging PRs if the pipeline is failing, but while he is open to discuss this topic, he thinks that since other teams also need to merge things. He doesn't want to block them because of the pipeline.
More context:
- It is a startup and we want to get the job done faster.
- I have 2 months working there.
- The pipeline was always failing because of another step that was removed recently. I think they got used to ignore the pipeline because of that.
I believe we are paying 10x of the future time, for short term quick time (10 minutes of the future for 1 minute today).
I thing If we continue with this, all will blow up in our faces.
I am tiring of fix the pipeline almost every day, and checking my team PRs as well.
Not sure if I should just keep pushing, or stop worrying if the pipeline passes or not and just see how the things blow up, and then try to convince the team of keep the pipeline passing as a strict requisite to merge a PR.
What would you do in my case?
![Mid-Level Software Engineer [SDE 2] at Amazon company logo](/_next/image/?url=https%3A%2F%2Ffirebasestorage.googleapis.com%2Fv0%2Fb%2Ftech-career-growth.appspot.com%2Fo%2Fcompany_logos%252Famazon_logo.jpg%3Falt%3Dmedia%26token%3Dff5f2302-08be-43bc-b7b1-d381ba508d99&w=48&q=75){kind=logo}
Discussion
(3 comments)- 011 days agoTech Lead/Manager at Meta, Pinterest, Kosei
Merging code when the pipeline is broken seems... not great 😳 This part of your argument makes the most sense to me:
I believe we are paying 10x of the future time, for short term quick time (10 minutes of the future for 1 minute today).
A few thoughts:
- Can you break down the build steps into several distinct steps? Perhaps there are some minor issues in the pipeline that should not be considered merge-blocking (e.g. lint errors)? Can you get agreement on what is a severe issue that should truly block developers and other teams?
- Can you talk to others on the team to get an idea about how much time the team is wasting on these issues, and how much benefit there would be? This will be a more convincing argument for the manager.
I absolutely think this is worth having a conversation about, even if you're new. This feels like pretty easy impact to achieve!
- 0
11 days agoTech Lead @ Robinhood, Meta, Course HeroFirst, I recommend reading through this thread around how to build a quality pipeline that people actually respect (it's about mobile but still applies to everyone): "What do mobile testing strategies look like at top tech companies?"
At the end of the day, it's all about impact, especially for startups. For many startups, having a quality pipeline is poor ROI as it takes time away from building features, which is generally the highest ROI thing for startups.
In short, you need to prove that at your stage, the pipeline is worth paying attention to and maintaining. In particular, you need to come up with cases where people ignored the pipeline, landed production code anyways, and something blew up in their face. In terms of output, it can be a simple list document, but if you want to go above and beyond, you can make a spreadsheet where you organize the regressions by time, severity of bug, and what feature was affected.
After you create your case for why the pipeline is valuable, take it to your manager.
Another thing I recommend is to not view the pipeline as all or nothing. Linting and automated tests are very different things. If you can't save the entire pipeline, treat it as a modular resource and target specific pieces that aren't as noisy and are more respected by the team. Even for automated tests, it can be a mix of many tests like unit tests, integration tests, and end-to-end. If end-to-end tests are the flakiest, then you can try to only save the unit tests for example.
- 011 days agoEng @ Taro
The pipeline was always failing because of another step that was removed recently. I think they got used to ignore the pipeline because of that.
I've seen this happen in the past: there is a testing infrastructure problem that causes the testing pipeline to produce flaky failures, and everyone is told to just ignore the problem to get their PRs in. The issue is that habits get formed during time, and it can be really hard to break the habits. As an engineer, you are mainly focused on merging your PR in, so you probably don't care about the testing infra issue.
The issue is, what's the point of having a testing pipeline if half of the people are ignoring it? I'm actually surprised that there hasn't been an issue yet from ignoring the testing pipeline. (If there have been bugs deployed to production, you can use that as evidence that there needs to be a mindset shift to treat the testing pipeline errors seriously).
The more people merge in their lint and test errors, the harder it will be to get the testing pipeline into a passing state again. So, you'll end up compounding more and more errors.
Ideally, you need to get everyone treating the testing pipeline seriously again. If your manager isn't going to help out, you can lead the initiative to fix the testing pipeline.
- Make sure the initial issue where the removed step caused the failure has been fixed.
- Make sure the all of the checks pass again.
- Send out an announcement to all of the teams that the testing pipeline is working again and that everyone's PRs need to pass the testing pipeline to be merged in.
- Don't allow people to merge in their PRs if they aren't passing the test (or have them pass through a special flag to do this, and make sure you get notified when this happens)
