archive/netdata_netdata
0
0
Fork 0
mirror of https://github.com/netdata/netdata.git synced 2025-05-05 01:30:32 +00:00
Code Issues Projects Releases Wiki Activity

on prem files moved to their own repo ()

Browse source
This commit is contained in:
Fotis Voutsas 2025-04-01 15:19:38 +03:00 • committed by GitHub
parent 5a9b3ecadf
commit 9a4ac79ed2
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
6 changed files with 2 additions and 374 deletions

55
docs/netdata-cloud/netdata-cloud-on-prem/README.md
View file

@ -1,55 +0,0 @@
# Netdata Cloud On-Prem
Netdata Cloud is built as microservices and is orchestrated by a Kubernetes cluster, providing a highly available and auto-scaled observability platform.
The overall architecture looks like this:
```mermaid
flowchart TD
Agents("🌍 <b>Netdata Agents</b><br/>Users' infrastructure<br/>Netdata Children & Parents")
users[["🔥 <b>Unified Dashboards</b><br/>Integrated Infrastructure<br/>Dashboards"]]
ingress("🛡️ <b>Ingress Gateway</b><br/>TLS termination")
traefik((("🔒 <b>Traefik</b><br/>Authentication &<br/>Authorization")))
emqx(("📤 <b>EMQX</b><br/>Agents Communication<br/>Message Bus<br/>MQTT"))
pulsar(("⚡ <b>Pulsar</b><br/>Internal Microservices<br/>Message Bus"))
frontend("🌐 <b>Front-End</b><br/>Static Web Files")
auth("👨‍💼 <b>Users &amp; Agents</b><br/>Authorization<br/>Microservices")
spaceroom("🏡 <b>Spaces, Rooms,<br/>Nodes, Settings</b><br/>Microservices for<br/>managing Spaces,<br/>Rooms, Nodes and<br/>related settings")
charts("📈 <b>Metrics & Queries</b><br/>Microservices for<br/>dispatching queries<br/>to Netdata Agents")
alerts("🔔 <b>Alerts & Notifications</b><br/>Microservices for<br/>tracking alert<br/>transitions and<br/>deduplicating alerts")
sql[("✨ <b>PostgreSQL</b><br/>Users, Spaces, Rooms,<br/>Agents, Nodes, Metric<br/>Names, Metrics Retention,<br/>Custom Dashboards,<br/>Settings")]
redis[("🗒️ <b>Redis</b><br/>Caches needed<br/>by Microservices")]
elk[("🗞️ <b>Elasticsearch</b><br/>Feed Events Database")]
bridges("🤝 <b>Input & Output</b><br/>Microservices bridging<br/>agents to internal<br/>components")
notifications("📢 <b>Notifications Integrations</b><br/>Dispatch alert<br/>notifications to<br/>3rd party services")
feed("📝 <b>Feed & Events</b><br/>Microservices for<br/>managing the events feed")
users --> ingress
agents --> ingress
ingress --> traefik
ingress ==>|agents<br/>websockets| emqx
traefik -.- auth
traefik ==>|http| spaceroom
traefik ==>|http| frontend
traefik ==>|http| charts
traefik ==>|http| alerts
spaceroom o-...-o pulsar
spaceroom -.- redis
spaceroom x-..-x sql
spaceroom -.-> feed
charts o-.-o pulsar
charts -.- redis
charts x-.-x sql
charts -..-> feed
alerts o-.-o pulsar
alerts -.- redis
alerts x-.-x sql
alerts -..-> feed
auth o-.-o pulsar
auth -.- redis
auth x-.-x sql
auth -.-> feed
feed <--> elk
alerts ----> notifications
%% auth ~~~ spaceroom
emqx <.-> bridges o-..-o pulsar
```

BIN
docs/netdata-cloud/netdata-cloud-on-prem/infrastructure.jpeg
View file

Binary file not shown.
Side by side

Before

(image error) Size: 505 KiB

137
docs/netdata-cloud/netdata-cloud-on-prem/installation.md
View file

@ -1,137 +0,0 @@
# Netdata Cloud On-Prem Installation
## System Requirements
| Component | Details |
|:---------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------|
| **Kubernetes** | - Version 1.23 or newer<br/>- Metrics server installed (for autoscaling)<br/>- Default storage class configured (SSD-based preferred) |
| **TLS certificate** | - Single certificate for all endpoints, or separate certificates for frontend, API, and MQTT<br/>- Must be trusted by all connecting entities. |
| **Netdata Cloud Services** | - 4 CPU cores<br/>- 15GiB memory<br/>- Note: Cloud services are ephemeral |
| **Third-Party Services** | - 8 CPU cores<br/>- 14GiB memory<br/>- 160GiB SSD storage for PVCs |
> **Note**:
>
> These requirements were tested with 1,000 directly connected nodes.
> Resource needs may vary based on your workload.
> The initial sync of directly connected Agents is the most compute-intensive operation.
> For example, a Postgres instance with 2 vCPU, 8GiB memory, and 1k IOPS can handle 1,000 nodes in a steady environment when adding nodes in batches of 10–30.
## Required Components
### Third-Party Services
All components below are included in the `netdata-cloud-dependency` package:
| Component | Version | Purpose |
|------------------------|---------|-------------------------------------|
| **PostgreSQL** | 13.7 | Main metadata database |
| **EMQX** | 5.11 | MQTT Broker for Agent communication |
| **Apache Pulsar** | 2.10+ | Inter-container message broker |
| **Traefik** | 2.11.x | Internal API Gateway |
| **Elasticsearch** | 8.8.x | Events feed storage |
| **Redis** | 6.2 | Caching layer |
| **Ingress Controller** | - | HTTPS support |
| **imagePullSecret** | - | Secured ECR repository access |
> **Important**:
>
> The provided dependency versions require additional configuration for production use.
> Customers should configure these applications according to their production requirements and policies.
### Installation Tools
- [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html)
- Helm (version 3.12+ with OCI Configuration)
- Kubectl
## Installation
1. **AWS CLI Configuration**
Configure AWS credentials using either environment variables:
```bash
export AWS_ACCESS_KEY_ID=<your_secret_id>
export AWS_SECRET_ACCESS_KEY=<your_secret_key>
```
Or through interactive setup:
```bash
aws configure
```
2. **Configure Helm for ECR Access**
Generate token for ECR access:
```bash
aws ecr get-login-password --region us-east-1 | helm registry login --username AWS --password-stdin 362923047827.dkr.ecr.us-east-1.amazonaws.com
```
3. **Pull Required Helm Charts**
```bash
helm pull oci://362923047827.dkr.ecr.us-east-1.amazonaws.com/netdata-cloud-dependency --untar # Optional
helm pull oci://362923047827.dkr.ecr.us-east-1.amazonaws.com/netdata-cloud-onprem --untar
```
The charts will be extracted to your current working directory.
4. **Install Dependencies**
The `netdata-cloud-dependency` chart installs all required third-party applications. While we provide this for easy setup, **production environments should use their own configured versions of these components**:
- Configure the installation by editing `values.yaml` in your `netdata-cloud-dependency` chart directory.
- Install the dependencies:
```bash
cd [your helm chart location]
helm upgrade --wait --install netdata-cloud-dependency -n netdata-cloud --create-namespace -f values.yaml .
```
5. **Install Netdata Cloud On-Prem**
- Configure the installation by editing `values.yaml` in your `netdata-cloud-onprem` chart directory.
- Install the application:
```bash
cd [your helm chart location]
helm upgrade --wait --install netdata-cloud-onprem -n netdata-cloud --create-namespace -f values.yaml .
```
> **Important**:
>
> Installation includes resource provisioning with migration services.
>
> During the first installation, a `netdata-cloud-common` secret is created containing critical encryption data. This secret persists through reinstalls and should never be deleted, as this will result in data loss.
## Architecture Components
<details><summary>View detailed microservices description</summary>
| Microservice | Description |
|:---------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| cloud-accounts-service | Handles user registration & authentication |
| cloud-agent-data-ctrl-service | Forwards request from the OCP to the relevant Agents. The requests include fetching Chart metadata, Chart data and Function data from the Agents. |
| cloud-agent-mqtt-input-service | Forwards MQTT messages emitted by the Agent to the internal Pulsar broker. They are related to the Agent entities and include Agent connection state updates. |
| cloud-agent-mqtt-output-service | Forwards Pulsar messages emitted on the OCP to the MQTT broker. They are related to the Agent entities. From there, the messages reach the relevant Agent. |
| cloud-alarm-config-mqtt-input-service | Forwards MQTT messages emitted by the Agent to the internal Pulsar broker. They related to the alarm-config entities like data for the alarm configuration as seen by the Agent. |
| cloud-alarm-log-mqtt-input-service | Forwards MQTT messages emitted by the Agent to the internal Pulsar broker. They are related to the alarm-log entities containing data about the alarm transitions that occurred in an Agent. |
| cloud-alarm-mqtt-output-service | Forwards Pulsar messages emitted in the Cloud to the MQTT broker. They are related to the alarm entities and from there, the messages reach the relevant Agent. |
| cloud-alarm-processor-service | Persists latest Alert status received from the Agent in the OCP<br/>Aggregates Alert statuses from relevant node instances<br/>Exposes API endpoints to fetch Alert data for visualization on the Cloud<br/>Determines if notifications need to be sent when Alert statuses change and emits relevant messages to Pulsar<br/>Exposes API endpoints to store and return notification-silencing data |
| cloud-alarm-streaming-service | Responsible for starting the Alert stream between the Agent and the OCP<br/>Ensures that messages are processed in the correct order, and starts a reconciliation process between the Cloud and the Agent if out-of-order processing occurs |
| cloud-charts-mqtt-input-service | Forwards MQTT messages emitted by the Agent related to the chart entities to the internal Pulsar broker. These include the chart metadata that is used to display relevant charts on the Cloud. |
| cloud-charts-mqtt-output-service | Forwards Pulsar messages emitted in the Cloud related to the charts entities to the MQTT broker. From there, the messages reach the relevant Agent. |
| cloud-charts-service | Exposes API endpoints to fetch the chart metadata<br/>Forwards data requests via the `cloud-agent-data-ctrl-service` to the relevant Agents to fetch chart data points<br/>Exposes API endpoints to call various other endpoints on the Agent, for instance, functions |
| cloud-custom-dashboard-service | Exposes API endpoints to fetch and store custom dashboard data |
| cloud-environment-service | Serves as the first contact point between the Agent and the OCP<br/>Returns authentication and MQTT endpoints to connecting Agents |
| cloud-feed-service | Processes incoming feed events and stores them in Elasticsearch<br/>Exposes API endpoints to fetch feed events from Elasticsearch |
| cloud-frontend | Contains the OCP website. Serves static content. |
| cloud-iam-user-service | Acts as a middleware for authentication on most of the API endpoints<br/>Validates incoming token headers, injects the relevant ones, and forwards the requests |
| cloud-metrics-exporter | Exports various metrics from an OCP installation<br/>Uses the Prometheus metric exposition format |
| cloud-netdata-assistant | Exposes API endpoints to fetch a human-friendly explanation of various Netdata configuration options, namely the Alerts. |
| cloud-node-mqtt-input-service | Forwards MQTT messages emitted by the Agent related to the node entities to the internal Pulsar broker<br/>These include the node metadata as well as their connectivity state, either direct or via Parents |
| cloud-node-mqtt-output-service | Forwards Pulsar messages emitted in the OCP related to the charts entities to the MQTT broker<br/>From there, the messages reach the relevant Agent |
| cloud-notifications-dispatcher-service | Exposes API endpoints to handle integrations<br/>Handles incoming notification messages and uses the relevant channels(email, slack...) to notify relevant users |
| cloud-spaceroom-service | Exposes API endpoints to fetch and store relations between Agents, nodes, spaces, users, and rooms<br/>Acts as a provider of authorization for other Cloud endpoints<br/>Exposes API endpoints to authenticate Agents connecting to the Cloud |
</details>

75
docs/netdata-cloud/netdata-cloud-on-prem/poc-without-k8s.md
View file

@ -1,75 +0,0 @@
# Netdata Cloud On-Prem PoC without k8s
These instructions are about installing a light version of Netdata Cloud for clients who do not have a Kubernetes cluster installed. This setup is **for demonstration purposes only**, as it has no built-in resiliency on failures of any kind.
## Prerequisites
- Ubuntu 22.04 (clean installation will work best).
- 10 CPU Cores and 24 GiB of memory.
- Access to shell as a sudo.
- TLS certificate for Netdata Cloud On-Prem PoC. A single endpoint is required. The certificate must be trusted by all entities connecting to this installation.
- AWS ID and License Key - we should have provided this to you, if not contact us: <info@netdata.cloud>.
To install the whole environment, log in to the designated host and run:
```bash
curl https://netdata-cloud-netdata-static-content.s3.amazonaws.com/provision.sh -o provision.sh
chmod +x provision.sh
sudo ./provision.sh install \
-key-id "" \
-access-key "" \
-onprem-license-key "" \
-onprem-license-subject "" \
-onprem-url "" \
-certificate-path "" \
-private-key-path ""
```
The script above is responsible for:
1. Prompting the user to provide:
- `-key-id` - AWS ECR access key ID.
- `-access-key` - AWS ECR Access Key.
- `-onprem-license-key` - Netdata Cloud On-Prem license key.
- `-onprem-license-subject` - Netdata Cloud On-Prem license subject.
- `-onprem-url` - URL for the On-prem (without http(s) protocol).
- `-certificate-path` - path to your PEM encoded certificate.
- `-private-key-path` - path to your PEM encoded key.
2. Installation will begin. The script will install:
- Helm
- Kubectl
- AWS CLI
- K3s cluster (single node)
3. The script starts to provision the K3s cluster with gathered data.
After cluster provisioning, the PoC Cloud is ready to be used.
> **Warning**
>
> This script will automatically expose Netdata but also a mailcatcher under `<URL from point 1.>/mailcatcher`.
## Logging-in
Only login by mail can work without further configuration. Every mail this PoC Cloud sends will appear on the mailcatcher, which acts as the SMTP server with a simple GUI to read the mails.
1. Open PoC Cloud in the web browser on the URL you specified
2. Provide an email
3. Mailcatcher will catch all the emails so go to `<URL from point 1.>/mailcatcher`. Find yours and click the link.
4. You are now logged into the PoC Cloud. Add your first nodes!
## Uninstalling
To uninstall the whole PoC, use the same script that installed it, with the `uninstall` switch.
```shell
cd <script dir>
sudo ./provision.sh uninstall
```
> **Note**
>
> In most cases of a failed installation, an automatic prompt will trigger for the user to start the uninstallation process automatically.

105
docs/netdata-cloud/netdata-cloud-on-prem/troubleshooting.md
View file

@ -1,105 +0,0 @@
# Netdata Cloud On-Prem Troubleshooting
Netdata Cloud On-Prem is an enterprise-grade monitoring solution that relies on several infrastructure components:
- Databases: PostgreSQL, Redis, Elasticsearch
- Message Brokers: Pulsar, EMQX
- Traffic Controllers: Ingress, Traefik
- Kubernetes Cluster
These components should be monitored and managed according to your organization's established practices and requirements.
## Common Issues
### Timeout During Installation
If your installation fails with this error:
```
Installing netdata-cloud-onprem (or netdata-cloud-dependency) helm chart...
[...]
Error: client rate limiter Wait returned an error: Context deadline exceeded.
```
This error typically indicates **insufficient cluster resources**. Here's how to diagnose and resolve the issue.
#### Diagnosis Steps
> **Important**
>
> - For full installation: Ensure you're in the correct cluster context.
> - For Light PoC: SSH into the Ubuntu VM with `kubectl` pre-configured.
> - For Light PoC, always perform a complete uninstallation before attempting a new installation.
1. Check for pods stuck in Pending state:
```shell
kubectl get pods -n netdata-cloud | grep -v Running
```
2. If you find Pending pods, examine the resource constraints:
```shell
kubectl describe pod <POD_NAME> -n netdata-cloud
```
Review the Events section at the bottom of the output. Look for messages about:
- Insufficient CPU
- Insufficient Memory
- Node capacity issues
3. View overall cluster resources:
```shell
# Check resource allocation across nodes
kubectl top nodes
# View detailed node capacity
kubectl describe nodes | grep -A 5 "Allocated resources"
```
#### Solution
1. Compare your available resources against the [minimum requirements](https://github.com/netdata/netdata/blob/master/docs/netdata-cloud/netdata-cloud-on-prem/installation.md#system-requirements).
2. Take one of these actions:
- Add more resources to your cluster.
- Free up existing resources.
### Login Issues After Installation
Installation may complete successfully, but login issues can occur due to configuration mismatches. This table provides a quick reference for troubleshooting common login issues after installation.
| Issue | Symptoms | Cause | Solution |
|-------------------------------|---------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------|
| SSO Login Failure | Unable to authenticate via SSO providers | - Invalid callback URLs<br/>- Expired/invalid SSO tokens<br/>- Untrusted certificates<br/>- Incorrect FQDN in `global.public` | - Update SSO configuration in `values.yaml`<br/>- Verify certificates are valid and trusted<br/>- Ensure FQDN matches certificate |
| MailCatcher Login (Light PoC) | - Magic links not arriving<br/>- "Invalid token" errors | - Incorrect hostname during installation<br/>- Modified default MailCatcher values | - Reinstall with correct FQDN<br/>- Restore default MailCatcher settings<br/>- Ensure hostname matches certificate |
| Custom Mail Server Login | Magic links not arriving | - Incorrect SMTP configuration<br/>- Network connectivity issues | - Update SMTP settings in `values.yaml`<br/>- Verify network allows SMTP traffic<br/>- Check mail server logs |
| Invalid Token Error | "Something went wrong - invalid token" message | - Mismatched `netdata-cloud-common` secret<br/>- Database hash mismatch<br/>- Namespace change without secret migration | - Migrate secret before namespace change<br/>- Perform fresh installation<br/>- Contact support for data recovery |
> **Warning**
>
> If you're modifying the installation namespace, the `netdata-cloud-common` secret will be recreated.
>
> **Before proceeding**: Back up the existing `netdata-cloud-common` secret. Alternatively, wipe the PostgreSQL database to prevent data conflicts.
### Slow Chart Loading or Chart Errors
When charts take a long time to load or fail with errors, the issue typically stems from data collection challenges. The `charts` service must gather data from multiple Agents within a Room, requiring successful responses from all queried Agents.
| Issue | Symptoms | Cause | Solution |
|----------------------|-----------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Agent Connectivity | - Queries stall or timeout<br/>- Inconsistent chart loading | Slow Agents or unreliable network connections prevent timely data collection | Deploy additional [Parent](/docs/observability-centralization-points/README.md) nodes to provide reliable backends. The system will automatically prefer these for queries when available |
| Kubernetes Resources | - Service throttling<br/>- Slow data processing<br/>- Delayed dashboard updates | Resource saturation at the node level or restrictive container limits | Review and adjust container resource limits and node capacity as needed |
| Database Performance | - Slow query responses<br/>- Increased latency across services | PostgreSQL performance bottlenecks | Monitor and optimize database resource utilization:<br/>- CPU usage<br/>- Memory allocation<br/>- Disk I/O performance |
| Message Broker | - Delayed node status updates (online/offline/stale)<br/>- Slow alert transitions<br/>- Dashboard update delays | Message accumulation in Pulsar due to processing bottlenecks | - Review Pulsar configuration<br/>- Adjust microservice resource allocation<br/>- Monitor message processing rates |
## Need Help?
If issues persist:
1. Gather the following information:
- Installation logs
- Your cluster specifications
2. Contact support at `support@netdata.cloud`.

4
docs/netdata-cloud/versions.md
View file

@ -18,5 +18,5 @@ Visit our [Pricing](https://www.netdata.cloud/pricing/) page to compare features
To learn more about installing Netdata Cloud in your own infrastructure:
1. Review the [Architecture Diagram](/docs/netdata-cloud/netdata-cloud-on-prem/README.md) to understand the system components.
2. Follow the [Deployment Guide](/docs/netdata-cloud/netdata-cloud-on-prem/installation.md) for detailed installation instructions
1. Review the [Architecture Diagram](https://github.com/netdata/netdata-cloud-onprem/blob/master/docs/learn.netdata.cloud/README.md) to understand the system components.
2. Follow the [Deployment Guide](https://github.com/netdata/netdata-cloud-onprem/blob/master/docs/learn.netdata.cloud/installation.md) for detailed installation instructions