LangKit is WhyLabs' open source LLM text indicator toolkit (Python library). It provides out-of-the-box indicators such as text quality, text relevance, security and privacy, sentiment and toxicity analysis, etc. It is deeply integrated with the whylogs data log library and is suitable for observability and security monitoring of production-bound LLM input/output. AI programming

LangKit

Core parameters and statistics

LangKit is WhyLabs' open source Python text metrics toolkit, focused on providing structured observability signals for large language model (LLM) interactions in production contexts. It is not a standalone application, but a library embedded in Python data pipelines, deeply integrated with the open source data logging library whylogs.

Projects	Public Information
Official positioning	Open-source text metrics toolkit for monitoring language models
Delivery form	Python library (PyPI package)
Open Source License	Apache-2.0
GitHub Stars	~991
GitHub Forks	72
Latest version	v0.0.35 (2024-11-07)
Total releases	35 versions
Supported platforms	Python API (works with whylogs)
Core modules	Text Quality, Text Relevance, Security & Privacy, Sentiment & Toxicity

Positioning Boundaries: LangKit is not an end-to-end monitoring platform, nor is it a secure firewall. It is a set of programmable UDFs (user-defined functions) that output structured indicators for recording by whylogs, and then analyzed by the WhyLabs platform or custom pipelines. Its value lies in converting unstructured LLM text into quantifiable numerical indicators.

Performance reference: The official benchmark test shows that on the c5.xlarge instance, the "Light metrics" mode can reach 2335 chats/sec, but after enabling all LLM metrics, it drops to 8.2 chats/sec; on g4dn.xlarge (including GPU), the full metric mode can reach 1.79 chats/sec. This data shows that there is significant computing overhead for all indicators, and they need to be selectively enabled according to scenarios during production deployment.

User and market recognition

Community base: GitHub has about 991 stars, 72 forks, 10 contributors, and 15 watchers. The community size is medium but accurate, mainly focusing on the fields of AI observability and LLMOps. 35 releases indicate ongoing maintenance investment in the project.

Industry Background: LangKit is developed and open sourced by WhyLabs. WhyLabs was once a leading startup in the AI ​​observability track, founded by teams from Amazon, Google, New Relic and other companies. The company has ceased operations between 2025 and 2026, but the complete platform has been open source, and LangKit and whylogs as core components will continue to exist in open source form.

Adoption premise: The value of LangKit is highly dependent on the whylogs ecosystem. If the team is already using whylogs for data logging, there is almost no cost to integrate LangKit; otherwise, an additional data logging pipeline needs to be introduced, and this part of the integration work needs to be included in the evaluation.

Cost advantage

LangKit itself is an Apache-2.0 licensed open source project with zero cost to use.

• C-side/Personal Developer: Free. It can be used through pip install langkit[all], with no quota limit and no API Key requirement. Computing resources are borne by local or self-managed servers.
• API/Developer: Free. LangKit itself does not incur API call fees, but if you use functions that require external models (such as hallucination detection based on OpenAI), you must bear the corresponding API fees yourself.
• Enterprise/Private: Free self-hosting. Since WhyLabs has ceased operations, hosting services for the original WhyLabs platform are no longer available. Enterprises can only take the self-hosted path, integrate LangKit into their own data pipelines, use whylogs to record indicators and build their own visualization/alerting links. Hidden costs include: integrated development man-hours, GPU/CPU computing resources, storage and operation and maintenance.

Hidden Cost: The overhead of the full LLM indicator is not negligible. Official benchmarks show that the throughput of full-metric mode on CPU instances is only 0.28 chats/sec (c5.xlarge), which means that high-concurrency scenarios require GPU instances or selectively enable some metrics. It is recommended to use the official notebook to do stress testing in the target environment before adoption.

Main functions

• Text Quality: Automatically calculates readability scores, complexity and grade scores. Suitable for monitoring whether the output is too obscure or too simple, helping to detect model degradation or format drift.
• Text Relevance: Calculate the similarity score between prompt and response, and support similarity comparison of user-defined themes. Used to detect whether the model "answers the wrong question" or deviates from the business theme.
• Security and Privacy: Contains five sub-modules - patterns (custom regular match count), jailbreaks (jailbreak attack similarity), prompt injection (prompt injection similarity), hallucinations (response consistency check), refusals (rejection response similarity). This set of features can upgrade security signals from manual spot checks to continuous automated scanning.
• Sentiment and Toxicity: Provides sentiment analysis and toxicity analysis. It is suitable for scenarios such as social content review and customer service dialogue monitoring that require real-time content security.
• PII Detection: A new PII (Personally Identifiable Information) detection module has been added since v0.0.29, which can identify sensitive information in text.

Expert View: The core value of LangKit lies not in a single indicator, but in the "synergistic effect of indicator combinations". For example, enabling the Text Quality + Relevance + Security modules simultaneously allows you to monitor output quality degradation, topic deviations, injection attacks, and hallucination risks simultaneously in a single data pipeline without having to switch between multiple tools. Deep integration with whylogs enables all indicators to be automatically aligned into the same set of data profiles, making it easier to conduct cross-dimensional anomaly correlation analysis.

Model and version evolution

LangKit follows semantic versioning and continues to iterate with v0.0.x as the main line. The following are key release nodes:

Mainline release

• v0.0.35 (2024-11-07): Fix rolling logger schema, update notebook opt-in upload logic, currently the latest stable version.
• v0.0.34 (2024-10-30): Upgrade dependencies to whylogs 1.5 to maintain compatibility with upstream whylogs.
• v0.0.33 (2024-08-12): Replace textstat with the self-maintained whylabs-textstat to reduce the risk of external dependencies.
• v0.0.32 (2024-05-29): Added local models support, allowing models to be run locally instead of relying on external APIs.
• v0.0.31 (2024-03-06): Added configurable detoxify model option for toxicity analysis, improving error exposure in hallucination detection.
• v0.0.30 (2024-02-07): Remove faiss dependency, upgrade embeddings to v2, and reduce installation volume and complexity.
• v0.0.29 (2024-01-23): Added PII detection module and VADER emotional support.
• v0.0.28 (2023-12-09): Supports OpenAI client v0/v1 dual compatibility, and adds Proactive Injection Detection.
• v0.0.27 (2023-11-28): Added feature extraction wrapper CUDA benchmark.
• v0.0.26 (2023-11-22): Supports Azure OpenAI hosting model GPU usage control.

Iteration characteristics: The project took about one year from v0.0.26 to v0.0.35, maintaining an average monthly rhythm of 1-2 versions. The early version focused on functional expansion (security module PII, local model), and later on stability and dependency maintenance. After WhyLabs is discontinued, the frequency of project updates may further slow down.

Technical advantages

Mechanism — Effect — Scenario Analysis:

• UDF architecture based on whylogs : Each indicator module of LangKit is implemented as one or more whylogs UDF, which is automatically mounted into the whylogs log pipeline. This means that users do not need to manually write feature extraction logic. They only need to import the module and initialize the schema to synchronize the output indicators during the data logging process. For teams that already have whylogs integration, this mechanism enables "zero extra code" observability enhancements.
• Similarity-based security detection: Security modules such as jailbreak, prompt injection, and refusal use vector similarity to match known attack/denial patterns instead of a rule engine. The advantage of this method is that it can be generalized to unseen variant attacks, but the disadvantage is that it is sensitive to the quality of the embedding model and has false positives. For production deployments, it is recommended to first calibrate the thresholds with a benchmark data set.
• Modular on-demand loading: LangKit's indicators are split into independent modules, and users only install and import the required functions to avoid unnecessary computing overhead. The officially provided three-level configurations of "light metrics", "LLM metrics" and "all metrics" are the embodiment of this design - on c5.xlarge, light mode (2335 chats/sec) is about 8300 times faster than full mode (0.28 chats/sec).
• Local model optional: Starting from v0.0.32, it is supported to run the detection model locally without forcing dependence on external APIs. This is critical for data sovereignty-sensitive or offline bordered deployments - security metrics can be calculated completely without an external network.

How to use

The relevant information has not been made public, please refer to the official real-time page.

Product Pricing

LangKit itself is completely free and open source. Pricing tiers are as follows:

Tier	Fee	Description
Python library	Free	Apache-2.0 license, no usage restrictions
Computing resources	Own responsibility	Indicator calculation consumes CPU/GPU resources, and cloud service fees are borne according to actual usage
Hosting Platform	Unavailable	WhyLabs platform is no longer operational, no commercial hosting options
Enterprise Support	Unavailable	WhyLabs Inc. is discontinued and has no commercial support contract

Following options: Due to the shutdown of the WhyLabs platform, LangKit users are currently only able to use it in a self-hosted manner. The community may fork the project or build new visualization solutions based on whylogs in the future. When evaluating adoption, you need to confirm that your own team has the ability to maintain this open source link.

Application scenarios

• LLM Production Monitoring: Continuously collect text quality, relevance, and security indicators of prompt/response in the production environment to detect model degradation (reduction in output quality), topic drift (reduction in relevance score), or abnormal input (injection/jailbreak attack). Applicable positions: MLOps engineer AI application developer.
• Security Compliance Audit: Enable PII detection, injection detection and jailbreak detection modules, and perform automated security scans on LLM interaction logs. Suitable for LLM deployment scenarios in regulated industries such as financial institutions and healthcare. Verification focus: the balance between false positive rate and recall rate.
• Content Quality Control: Analyze model output for readability, sentiment, and toxicity to ensure that user-oriented content meets brand tone and safety standards. It is suitable for scenarios such as customer service dialogue, social media content generation, and educational content production.
• Prompt word engineering evaluation: Use text relevance indicators to compare the semantic distance between different prompt variants and ideal output during the development stage, and quantify the prompt optimization effect. Key points of verification: The similarity threshold needs to be defined according to business semantics, rather than being one-size-fits-all.

Applicable people

• MLOps/AI Infrastructure Engineer: The technical team required to build observability pipelines for LLM applications. LangKit provides standardized indicator extraction to reduce duplication of work in self-built monitoring systems. Prerequisite: The team already has or is willing to introduce whylogs.
• AI Security Researcher: Researchers who need to analyze prompt security in batches. LangKit's injection/jailbreaking/hallucination modules provide reusable similarity detection baselines. Unsuitable scenarios: Scenarios that require high-precision customized rules or real-time interception. LangKit only generates indicators and does not perform blocking.
• LLM Application Developer: Developer who is building an LLM-based product and needs to know if the model is behaving abnormally. Production monitoring can be connected with a few Python codes. Unsuitable scenarios: Teams with non-Python technology stacks (LangKit is a pure Python library).
• Unsuitable people: Teams that need a zero-code visual monitoring panel (LangKit does not provide a UI, and indicators need to be visualized by themselves); teams that need a real-time security interception gateway (LangKit is a log analysis tool rather than a firewall); existing users who are dependent on the WhyLabs platform (the platform has been discontinued and needs to be migrated to a self-hosted solution).

Summary and Outlook

The value of LangKit is to provide a standardized, programmable indicator extraction layer for LLM production monitoring. It encapsulates multi-dimensional signals such as text quality, relevance, safety, and emotion into whylogs UDF. Users can transform unstructured LLM interactions into structured observable data with just a few lines of code. The project's engineering maturity is demonstrated by the Apache-2.0 license and an iteration record of 35 releases.

Current Limitations and Uncertainties:

• WhyLabs company has ceased operations and the platform hosting link is invalid. There is uncertainty about the future maintenance rhythm and community activity of LangKit. It is recommended to pay attention to the commit records of the GitHub warehouse in the past three months before adopting it.
• The calculation overhead of full indicators is very high (<1 chat/sec on CPU instances). Production deployment must selectively enable modules according to scenarios and plan GPU resources.
• LangKit only provides indicator extraction and does not include monitoring functions such as visualization, alarms, and dashboards. It needs to be built on the whylogs output by yourself.
• The security module is based on similarity matching and is sensitive to the quality of the embedded model. The threshold needs to be calibrated with actual data in the production environment.

Procurement and Adoption Risk Assessment: LangKit is suitable for Python technical teams that already use or plan to use whylogs as the metric source layer of the LLM observability pipeline. Three points need to be confirmed before evaluation and adoption: (1) The team has the ability to self-host whylogs and process the output Profile data; (2) The concurrency of the target scenario is within the performance budget of LangKit's selected module (it is recommended to use the official benchmark for stress testing first); (3) There is reasonable expectation management for the future maintenance rhythm of the project. For teams that need an out-of-the-box monitoring platform or a non-Python technology stack, LangKit is not currently a suitable choice.

Related tools: , Cursor

How to use LangKit

LangKit is distributed as a Python library, installed through PyPI, and the entry point is SDK integration with whylogs.

Installation:

pip install langkit[all]

For a minimal installation (excluding optional dependencies such as torch and transformers), use:

pip install langkit

Basic usage:

import whylogs as why
from langkit import llm_metrics

#Initialize LLM indicator schema
schema = llm_metrics.init()

# Record a prompt/response interaction and automatically calculate indicators
results = why.log(
    {"prompt": "What is machine learning,", "response": "Machine learning is..."},
    schema=schema
)

# The results include statistics on all registered indicators such as text quality, relevance, and security.
profile = results.profile()

Enable specific modules on demand:

from langkit import text_quality, sentiment

schema = llm_metrics.init()
# Only enable text quality and sentiment analysis modules
schema = text_quality.init() # Register readability and complexity indicators
schema = sentiment.init() # Register sentiment and toxicity indicators

Integrated into the production pipeline: The typical way to use LangKit is to insert a log point in the LLM call chain, pass prompt and response into why.log(), and then regularly upload the profile to the WhyLabs platform (original plan, the platform has been discontinued) or write it to local storage for subsequent analysis.

Entrance: GitHub repository (https://github.com/whylabs/langkit) contains complete documentation FAQ, sample notebooks and module documentation.

Version Info

• LangKit 0.0.35 ：Fix the rolling logger schema and update the notebook's opt-in upload logic. （2024-11-07）
• LangKit 0.0.34 ：Update dependencies to whylogs 1.5 to fix compatibility issues. （2024-10-30）
• LangKit 0.0.33 ：Replaced textstat with whylabs-textstat, updated README image. （2024-08-12）
• LangKit 0.0.32 ：Added local models support and updated textstat code snippet. （2024-05-29）
• LangKit 0.0.31 ：Added configurable detoxify model support and improved error handling in response to hallucination detection. （2024-03-06）
• LangKit 0.0.30 ：Remove faiss dependency and upgrade embeddings v2. （2024-02-07）