CanCanCan, Pagy, and MeiliSearch, How to combine the 3 gems?

I'm working on a project that has Cue model that could be accessed and searched by multiple roles like guest and admin. The authorization happens using CanCanCan gem based on some logic written in the Ability class. While the search happens using Pagy/MeiliSearch combination to do the full text search and the pagination. The issue I'm facing now is when I search as a guest, I can see the correct items a guest should see, and when I search as an admin, I can see the correct item an admin should see, BUT Pagy gem counts are incorrect. This is my code:

class SearchesController < ApplicationController
  INCLUDES = %i[medium speakers].freeze

  include Pagy::Backend

  authorize_resource class: false

  def search
    filter = ''
    filter = "speaker_ids IN [#{params[:speaker]}]" if params[:speaker].present?

    search_results = Cue.accessible_by(current_ability).includes(INCLUDES).pagy_search(params[:query], filter:)
    @pagy, @search_results = pagy_meilisearch(search_results)
  end
end

If there are some cues not accessible by a guest, the search will not return them in `@search_results`, but the `@pagy` object will count them in. This shows incorrect counts to the user and also shows less than the required `Pagy::DEFAULT[:items]`. I tried a lot of options, but with no success. The only thing I can think about (And I don't want to go this path TBH) is to add the filtering logic itself into the `filter` string and add the required parameters to the index. But this will be complicated and it will increase the index size. Do you have any thoughts?