Misc. Utilities¶
trulens_eval.utils.generated
¶
Utilities for dealing with LLM-generated text.
Attributes¶
PATTERN_0_10
module-attribute
¶
Regex that matches the last integer.
PATTERN_NUMBER
module-attribute
¶
Regex that matches floating point and integer numbers.
PATTERN_INTEGER
module-attribute
¶
Regex that matches integers.
Classes¶
Functions¶
re_0_10_rating
¶
Extract a 0-10 rating from a string.
If the string does not match an integer or matches an integer outside the 0-10 range, raises an error instead. If multiple numbers are found within the expected 0-10 range, the smallest is returned.
PARAMETER | DESCRIPTION |
---|---|
s |
String to extract rating from.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
int
|
Extracted rating.
TYPE:
|
RAISES | DESCRIPTION |
---|---|
ParseError
|
If no integers between 0 and 10 are found in the string. |
trulens_eval.utils.pace
¶
Classes¶
Pace
¶
Bases: BaseModel
Keep a given pace.
Calls to Pace.mark
may block until the pace of its returns is kept to a
constraint: the number of returns in the given period of time cannot exceed
marks_per_second * seconds_per_period
. This means the average number of
returns in that period is bounded above exactly by marks_per_second
.
Attributes¶
marks_per_second
class-attribute
instance-attribute
¶
marks_per_second: float = 1.0
The pace in number of mark returns per second.
seconds_per_period
class-attribute
instance-attribute
¶
seconds_per_period: float = 60.0
Evaluate pace as overage over this period.
Assumes that prior to construction of this Pace instance, the period did not have any marks called. The longer this period is, the bigger burst of marks will be allowed initially and after long periods of no marks.
seconds_per_period_timedelta
class-attribute
instance-attribute
¶
The above period as a timedelta.
mark_expirations
class-attribute
instance-attribute
¶
Keep track of returns that happened in the last period
seconds.
Store the datetime at which they expire (they become longer than period
seconds old).
max_marks
instance-attribute
¶
max_marks: int
The maximum number of marks to keep track in the above deque.
It is set to (seconds_per_period * returns_per_second) so that the average returns per second over period is no more than exactly returns_per_second.
last_mark
class-attribute
instance-attribute
¶
Time of the last mark return.
lock
class-attribute
instance-attribute
¶
Thread Lock to ensure mark method details run only one at a time.