I’m annoyed at Vercel.
They’ve silently changed their system to break the way we were doing deploys.
We are on the Hobby plan and they now require the person who does the GitHub push to be the member of the Vercel account.
I’m not annoyed at the fact they have nerfed the hobby plan.
I’m amazed we’ve been able to run the frontend to the Drivible dealership website (with our 100+ active users) without paying and without getting anywhere near the usage limits.
What I’m annoyed at is that our builds have been broken for days and we got no indication, messaging or warning.
Issue
The issue is that we are on a Hobby plan, but committing code as ourselves not as the company Github account which is the Owner.
This has worked for years, and suddenly stopped working. Without warning.
From the Vercel Documentation
If you are pushing commits to a private repository connected to a Vercel project that belongs to a Team, you must be a member of that Pro team or the Owner of the Hobby team on Vercel in order for the deployment to trigger successfully. You can read more about this in our documentation.
I will note that if I used the Github Pull request feature, then the Vercel bot does give you a warning. But until this issue occurred, I’ve just been using Source Tree or my IDE and pushing to the Development branch and then merging to Staging when I want it deployed to our Staging environment.
Going into Github breaks the workflow and seems to be a thing only big companies would deal with. We are a 3 person startup using the Hobby plan.
Stopgap Fix
If you are having this issue, then one option that currently seems to work is to commit to a different branch than what you are deploying on and use the Github website to create a pull request that merges into the branch, whilst logged in as the owner account.
The merge commit is then seen by Vercel as a commit by the main account.
Alternatively you can just change your Git push to be done as the owner account. Although you’ll miss the magic green boxes in Github pointing out how awesome you are as a programmer.
The Investigation story
When our most active front end engineer Tzaddi pointed out the issue a couple of days ago there was also a Vercel outage indicating the deploy logs weren’t displaying. I figured that meant deploys were being done just not shown.
I made a non-visible code change and couldn’t see any indication it had deployed.
Of course, I’d incorrectly thought that Build Logs meant anything about a deployment. But in hindsight I realise they just meant the actual logs from a deployment.
I subscribed to the outage notifications and saw an email it was resolved overnight.
I spent nearly 20mins on a video call trying to investigate the issue and couldn’t work it out.
Thursday as usual is a meetings day at Drivible.
We are a remote first SaaS Document Storage and payments solution for Car Dealerships. Currently a 3.5 person startup.
So when my co-founder reminded me of the fact he couldn’t check some changes I verified the outage was considered resolved.
I looked around on the Vercel platform and there wasn’t anything indicating they’d made changes, just that no deploys had happened in the last 3 days.
I checked our usage. We are barely using anything. We were happy to upgrade if needed. I mean we are paying thousands to Google, MongoDB and AWS. But for how trivial our usage is, with only 2.3GB of data transfer in the last 12 months and the fact we aren’t yet profitable. We knew we’d be effectively on the free tier for AWS if we were hosting there. Of course Vercel is who developed NextJS and are meant to be the best in class, so we are using them. But we aren’t going to proactively pay for something and intentionally shorten our runway.
I disconnected and reconnected to the GitHub account and made another mostly whitespace non-visible code change and saw nothing.
So I tried pushing another change with a trivial but noticeable change. We went back to our discussion about business related things.
My next meeting was with Tzaddi, and we investigated on the call which is when I finally found some activity log where it indicated that the git pushes weren’t being deployed because it rejected the user who made the push.
It was only by going to the activity tab that I saw
The deployment with SHA e75f40d48345ac6a64f76075a22791bbd2ca053a is blocked due to insufficient permissions. The github user kublermdk is not part of the team.
Our Vercel and GitHub account are setup using a generic company email address. Something so the company owns them.
We then use our personal GitHub profiles to push changes.
This has worked for at least over a year.
It’s how we’ve used it since we started using Vercel.
So it was a shock to see the activity notification.
There was no emails that I can find announcing a change.
The only notice of any kind was a generic banner saying to go Pro for 2x CPUs and faster builds.
But we don’t need faster deploys, we need deploys that actually trigger a build.
As per the Stopgap fix I did manage to get a deploy to work. By using a GitHub pull request as the Drivible user.
That’s when suddenly there was a bunch of info about how it wouldn’t deploy. I definitely didn’t expect to see info in GitHub when trying to do an pull request. That’s also not our workflow. We push into the branch.
Conclusion
I’m grateful that we didn’t have to do any emergency deployments to Production to do during this time.
Trying to figure this out whilst there was an outage notification with Vercel, the pressure of an issue on our system… That would have been hard. I would have been tempted to rig up something where we deploy our static NextJS build files to an S3 bucket and Cloudfront CDN or our own web servers. Something only possible because we aren’t using any of the server side aspects of NextJS and are running our own backend.
I don’t know why Vercel didn’t announce the change ahead of time. I’d have been fine if they suggested we add more users to our account, which you can only do by upgrading to a Pro account and paying $20/month per developer (specifically, per GitHub account).
My only thought is that Vercel states this is the way it should be and maybe they simply fixed a bug and enabled the reinforcement. So to them it wasn’t a change in business model. They just fixed up the enforcement of existing rules instead of saw this as adding a new one.
https://vercel.com/guides/why-aren-t-commits-triggering-deployments-on-vercel
Maybe they figured that the Github message was enough. Not realising that not everyone uses Pull requests.
Or maybe they just don’t care about people on the Hobby plan. Figuring anyone using it isn’t using it for anything important. That’s poor form on their behalf then as we always figured we’d go Pro when we hit any
Or maybe it was a change coming from the growth/sales team push to spruce up the numbers. So was rushed out the door. Although I’d have expected to get an email from them in this case.
I’m not sure. I had a search online to see if anyone has had this particular issue but couldn’t find anything.
Let me know if you’ve also been affected by this.
Update
It looks like we have now received an email from Vercel about the rejected push.
The email was considered spam by Google Apps and I only just received a request to approve the email to the forwarding email list.
Given this email, I’ll back down a bit about their lack of communication.
I think they’ve only just added this recently. But at least they have both the Github bot and are proactively sending an email which can cover the workflow more like how we are using it.
NB: This has also been posted to Medium.
