Dealing with CROSSSLOT Keys error in Redis

Preface

We were in the process of moving to the new serverless Redis from AWS ElastiCache. We had started using it for cache but had not yet moved our session store or horizon queue to it.

The Problem

When we moved everything else to the new serverless Redis, we started seeing a lot of errors in our logs. The error was CROSSSLOT Keys in request don't hash to the same slot. After a lot of debugging, we found this error seemed to be caused by Laravel Horizon attempting to track the metrics of the queues. This didn't make a ton of sense since from our logs it looked like jobs were still being processed okay, but we were still being flooded with these errors.

The Solution

We found a GitHub Issue talking about wrapping the REDIS_PREFIX in {} to fix the issue. We tried this but it did not work. After a couple more hours of debugging, we went back to the article and tried not only wrapping the REDIS_PREFIX in {} but also the HORIZON_PREFIX. This fixed the issue and we were no longer seeing the CROSSSLOT errors in our logs.

So in our .env file we ended up with the following:

...

REDIS_PREFIX={my-prefix}

...

HORIZON_PREFIX={my-horizon-prefix}

Hopefully this helps someone else who is running into this issue. It was a massive pain to debug and there doesn't seem to be a lot of information out there about it specifically for Laravel Horizon.