A dollar sign broke my GitLab CI login

Then we had locked accounts, just to make debugging a little more convenient.

I had copied every variable from the pipeline schedule into my local run. Same values, side by side. I wanted to rule out something being misconfigured before I started pulling apart the login flow.

Local login still worked. CI still sat there.

I thought I'd reproduced the environment

Once the variables looked identical, I started looking elsewhere. The login code. The URL handling. Whether we'd set up the test pipelines wrong. Something about running this in CI had to explain why the redirect never happened.

A five-minute wait gives you plenty of time to come up with theories. The retries give you time to become emotionally attached to them.

I kept coming back to the fact that the same login worked on my machine. I'd checked the configuration. I'd copied the values. What else was I supposed to reproduce?

Apparently, the part where GitLab changed one of them.

The password had a dollar sign in it

A couple of weeks earlier, I'd changed my password and put a $ in it.

Variable expansion was enabled for the password variable in the scheduled pipeline. The $ followed by letters was being interpreted as a reference to another variable.

Locally, my setup passed the password literally. In CI, it went through expansion before the test received it.

GitLab documents this behavior: its runner can expand values from pipeline schedules, recognizing references written as $variable or ${variable}. That means a saved value can be different from the value the job actually gets. GitLab's explanation of variable expansion covers the stages involved.

So checking the saved passwords side by side had told me exactly what was saved. It hadn't told me what Playwright was typing.

The fix was embarrassingly small

I removed the $ from my password and updated the pipeline schedules to match. The login worked again.

After all that digging into the login flow and CI setup, the difference was how one character in the password was being handled.

Removing it was my workaround. The configuration fix to aim for is passing credentials literally, with expansion disabled where the variable is defined. GitLab documents an Expand variable reference setting for project and group variables. The controls depend on where you define the variable, so check the configuration feeding your scheduled job. GitLab's variable expansion settings explain that option.

I wouldn't take this as a reason to avoid dollar signs in passwords. I would take it as a reason to check what happens to a secret between saving it and using it.

The bit I missed

I had reproduced the saved configuration locally. I hadn't reproduced what GitLab did to it before the test received it.

That's the part I'm going to remember the next time something works locally and fails in CI. Before I get too deep into the application code, check which values get expanded, substituted, or otherwise changed on the way in. Use a dummy value to investigate that behavior; real passwords don't need another home in the job logs.

And maybe don't let a login test spend five minutes waiting for success when the useful information is that authentication already failed.

Felt like a genius and an idiot at the same time.