Are you using a self-hosted runner or are you using the hosted Github runner?

the step before these should provide the caching. Have a look in the runner logs to see if there are any hints to why it is having to rebuild instead of pulling from these caches.

      - uses: docker/build-push-action@v5
        with:
          context: .
          cache-from: type=gha
          cache-to: type=gha,mode=max