Tru Rails for NeMo Guardrails¶
trulens_eval.tru_rails.TruRails
¶
Bases: App
Recorder for apps defined using NeMo Guardrails.
PARAMETER | DESCRIPTION |
---|---|
app |
A NeMo Guardrails application.
TYPE:
|
Functions¶
main_output
¶
main_output(
func: Callable,
sig: Signature,
bindings: BoundArguments,
ret: Any,
) -> JSON
Determine the main out string for the given function func
with
signature sig
after it is called with the given bindings
and has
returned ret
.
main_input
¶
main_input(
func: Callable, sig: Signature, bindings: BoundArguments
) -> JSON
Determine the main input string for the given function func
with
signature sig
after it is called with the given bindings
and has
returned ret
.
trulens_eval.tru_rails.RailsActionSelect
¶
Bases: Select
Selector shorthands for NeMo Guardrails apps when used for evaluating feedback in actions.
These should not be used for feedback functions given to TruRails
but
instead for selectors in the FeedbackActions
action invoked from with a
rails app.
Attributes¶
Events
class-attribute
instance-attribute
¶
Events = events
Selector for events in action call parameters.
Context
class-attribute
instance-attribute
¶
Context = context
Selector for context in action call parameters.
Warning
This is not the same "context" as in RAG triad. This is a parameter to rails actions that stores context of the rails app execution.
LLM
class-attribute
instance-attribute
¶
LLM = llm
Selector for the language model in action call parameters.
Config
class-attribute
instance-attribute
¶
Config = config
Selector for the configuration in action call parameters.
RetrievalContexts
class-attribute
instance-attribute
¶
RetrievalContexts = relevant_chunks_sep
Selector for the retrieved contexts chunks returned from a KB search.
Equivalent to $relevant_chunks_sep
in colang.
UserMessage
class-attribute
instance-attribute
¶
UserMessage = user_message
Selector for the user message.
Equivalent to $user_message
in colang.
BotMessage
class-attribute
instance-attribute
¶
BotMessage = bot_message
Selector for the bot message.
Equivalent to $bot_message
in colang.
LastUserMessage
class-attribute
instance-attribute
¶
LastUserMessage = last_user_message
Selector for the last user message.
Equivalent to $last_user_message
in colang.
LastBotMessage
class-attribute
instance-attribute
¶
LastBotMessage = last_bot_message
Selector for the last bot message.
Equivalent to $last_bot_message
in colang.
trulens_eval.tru_rails.FeedbackActions
¶
Feedback action action for NeMo Guardrails apps.
See docstring of method feedback
.
Functions¶
register_feedback_functions
staticmethod
¶
Register one or more feedback functions to use in rails feedback
action.
All keyword arguments indicate the key as the keyword. All positional arguments use the feedback name as the key.
action_of_feedback
staticmethod
¶
Create a custom rails action for the given feedback function.
PARAMETER | DESCRIPTION |
---|---|
feedback_instance |
A feedback function to register as an action.
TYPE:
|
verbose |
Print out info on invocation upon invocation.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
Callable
|
A custom action that will run the feedback function. The name is the same as the feedback function's name. |
feedback_action
async
staticmethod
¶
feedback_action(
events: Optional[List[Dict]] = None,
context: Optional[Dict] = None,
llm: Optional[BaseLanguageModel] = None,
config: Optional[RailsConfig] = None,
function: Optional[str] = None,
selectors: Optional[Dict[str, Union[str, Lens]]] = None,
verbose: bool = False,
) -> ActionResult
Run the specified feedback function from trulens_eval.
To use this action, it needs to be registered with your rails app and
feedback functions themselves need to be registered with this function.
The name under which this action is registered for rails is feedback
.
Usage
rails: LLMRails = ... # your app
language_match: Feedback = Feedback(...) # your feedback function
# First we register some feedback functions with the custom action:
FeedbackAction.register_feedback_functions(language_match)
# Can also use kwargs expansion from dict like produced by rag_triad:
# FeedbackAction.register_feedback_functions(**rag_triad(...))
# Then the feedback method needs to be registered with the rails app:
rails.register_action(FeedbackAction.feedback)
PARAMETER | DESCRIPTION |
---|---|
events |
See Action parameters. |
context |
See Action parameters. |
llm |
See Action parameters.
TYPE:
|
config |
See Action parameters.
TYPE:
|
function |
Name of the feedback function to run. |
selectors |
Selectors for the function. Can be provided either as strings to be parsed into lenses or lenses themselves. |
verbose |
Print the values of the selectors before running feedback and print the result after running feedback.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
ActionResult
|
An action result containing the result of the feedback.
TYPE:
|
Example
define subflow check language match
$result = execute feedback(\
function="language_match",\
selectors={\
"text1":"action.context.last_user_message",\
"text2":"action.context.bot_message"\
}\
)
if $result < 0.8
bot inform language mismatch
stop
trulens_eval.tru_rails.RailsInstrument
¶
Bases: Instrument
Instrumentation specification for NeMo Guardrails apps.
Classes¶
Default
¶
Default instrumentation specification.
Attributes¶
MODULES
class-attribute
instance-attribute
¶
MODULES = union(MODULES)
Modules to instrument by name prefix.
Note that NeMo Guardrails uses LangChain internally for some things.
CLASSES
class-attribute
instance-attribute
¶
CLASSES = lambda: union(CLASSES())
Instrument only these classes.
METHODS
class-attribute
instance-attribute
¶
METHODS: Dict[str, ClassFilter] = dict_set_with_multikey(
dict(METHODS),
{
"execute_action": ActionDispatcher,
(
"generate",
"generate_async",
"stream_async",
"generate_events",
"generate_events_async",
"_get_events_for_messages",
): LLMRails,
"search_relevant_chunks": KnowledgeBase,
(
"generate_user_intent",
"generate_next_step",
"generate_bot_message",
"generate_value",
"generate_intent_steps_message",
): LLMGenerationActions,
"feedback": FeedbackActions,
},
)
Instrument only methods with these names and of these classes.