Failover Requests

Episode #534 by Teacher's Avatar David Kimura

Summary

In this episode, we look at creating a failover mechanism for API requests. This can be a handy trick in situations where you want to add fault tolerance to an API request. We'll use the example of the Ollama Cloud as a failover to a locally hosted instance of Ollama.
rails ai api 11:06

Chapters

  • Introduction (0:00)
  • Adding the gem (1:26)
  • Creating the simple request (1:51)
  • Handling when one endpoint fails (5:17)
  • Final Thoughts (10:22)

Resources

Download Source Code

Summary

# Terminal
bundle add ruby-openai

# app/models/ollama_client.rb
# messages = [
#   { role: "system", content: "You are a helpful agent" },
#   { role: "user", content: "Why is the sky blue?" }
# ]
# OllamaClient.chat(messages: messages)

class OllamaClient
  class AllEndpointsFailed < StandardError; end
  ENDPOINTS = [
    "https://your.ollama.endpoint",
    "https://ollama.com"
  ]

  def self.chat(messages:, model: "gpt-oss:20b", temperature: 0.7, stream: false)
    new.chat(
      messages: messages,
      model: model,
      temperature: temperature,
      stream: stream
    )
  end

  def chat(messages:, model: "gpt-oss:20b", temperature: 0.7, stream: false)
    parameters = {
      messages: messages,
      model: model,
      temperature: temperature,
      stream: stream
    }

    attempt_with_failover do |client|
      response = client.chat(parameters: parameters)
      response.dig("choices", 0, "message", "content")
    end
  end

  private

  def attempt_with_failover
    ENDPOINTS.each_with_index do |endpoint, index|
      client = build_client(endpoint)
      result = yield(client)
      return result
    rescue Faraday::ServerError, Faraday::ConnectionFailed => e
      next if index < ENDPOINTS.length - 1

      raise AllEndpointsFailed, "All ollama endpoints failed. Last error: #{e.message}"
    rescue StandardError => e
      if index < ENDPOINTS.length - 1
        next
      else
        raise
      end
    end
  end

  def build_client(endpoint)
    OpenAI::Client.new(
      uri_base: endpoint,
      access_token: "YOUR_API_TOKEN"
    )
  end
end