LLM Prompt V2 Node

The LLM Prompt V2 node is used for talking to configured LLMs in flow, allowing for the use of a single LLM or a primary / fallback pair.

In the CLI

LLMs for use in this node can be configured using the ServisBOT CLI, using the commands listed here.

LLM Prompt V2 Node

The node, llmPromptV2, supports using either one or a pair of configured LLMs. When using two LLMs, one model is treated as the primary model and the other is used as a fallback.

LLM prompt V2 node

LLM prompt V2 node editor

Configuring the LLM Prompt V2 Node

  • Name (required) - A unique name for the node. Successful responses are set in the msg object using this name.

Primary model:

  • Primary model (required) - The alias of the configuration to use.
  • Prompt (optional) - The prompt to use when executing the LLM. If not set, the prompt from the configuration will be used.
  • Context (required) - The context to use when executing the LLM. This is used to provide additional information to the LLM.
  • Redact (optional) - The keys to redact from the context.
  • Max Retries (optional) - The maximum number of times to retry the LLM if it fails. 0 or blank means no retries.
  • Backoff (ms) (optional) - The initial backoff time in milliseconds, determining how long to wait before retrying the LLM on failure. Each subsequent retry will double the backoff time of the previous delay. 0 or blank means no backoff.
  • Timeout (ms) (optional) - Instead of per-request, this timeout is for the entire model, shared amongst its retries. When set, the model has n milliseconds to return a successful prompt response before giving up and moving to fallback / going out the bottom port if fallback is disabled. 0 or blank means no timeout.

Fallback model:

  • Enable (optional) - Whether to use a fallback model. If not set, the node outputs through the bottom port if the primary model fails.
  • Fallback model (optional) - The alias of the configuration to use. If not set, the primary model will be used.
  • Prompt (optional) - The prompt to use when executing the fallback LLM. If not set, the prompt from the configuration will be used.
  • Context (optional) - The context to use when executing the fallback LLM. This is used to provide additional information to the fallback LLM.
  • Redact (optional) - The keys to redact from the context.
  • Max Retries (optional) - The maximum number of times to retry the fallback LLM if it fails. 0 or blank means no retries.
  • Backoff (ms) (optional) - The initial backoff time in milliseconds, determining how long to wait before retrying the fallback LLM on failure. Each subsequent retry will double the backoff time of the previous delay. 0 or blank means no backoff.
  • Timeout (ms) (optional) - Instead of per-request, this timeout is for the entire model, shared amongst its retries. When set, the model has n milliseconds to return a successful prompt response before giving up and outputting through the bottom port. 0 or blank means no timeout.

Node Output

On success, the node outputs through the top port. The LLM response is set in the msg object under llm using the name of the node, e.g. if the node is named faqLLM, the response will be set in msg.llm.faqLLM. The result is a JSON object with the following fields:

  • response - The text response from the LLM.
  • model - ‘primary’ or ‘fallback’, depending on which model returned the successful response.

On failure, the node outputs through the bottom port and does not update the msg object.