Nvidia is introducing an open-source initiative to help prevent chatbots from making factual inaccuracies, such as those seen in ChatGPT. NeMo Guardrails is designed to mitigate this problem by providing features that help organizations ensure their chatbot applications are accurate and secure.
NeMo Guardrails includes three types of guardrails: topical, safety, and security. The topical guardrails can prevent applications from discussing unwanted topics, such as the weather while assisting customer service. The safety guardrails can detect errors and ensure the LLM uses only trusted sources. The security guardrails prevent the LLM from making contact with dangerous third-party applications by whitelisting known-to-be-safe applications.
Nvidia is making NeMo Guardrails open-source to contribute to the security of AI. It is available as part of Nvidia AI Foundations, a cloud service for developing and running proprietary AI models. The initiative comes amid a fierce debate about the security of AI, with prominent tech figures such as Elon Musk arguing for a six-month break in the development of complex large language models.
