DocsModel Evaluation

Model Evaluation Tools

Comprehensive tools for assessing model performance, diagnosing issues, and comparing models.

Overview

Classification Metrics

Accuracy, precision, recall, F1, ROC-AUC

Regression Metrics

MSE, MAE, R-squared, MAPE

Cross-Validation

K-fold, stratified, time series

Model Comparison

Side-by-side performance analysis

Error Analysis

Confusion matrices, residual plots

Classification Metrics

Metric	Formula	Range	Best
Accuracy	(TP+TN)/(TP+TN+FP+FN)	0-1	Higher
Precision	TP/(TP+FP)	0-1	Higher
Recall	TP/(TP+FN)	0-1	Higher
F1-Score	2(PR)/(P+R)	0-1	Higher
ROC-AUC	Area under ROC curve	0-1	Higher
Log Loss	Cross-entropy loss	0-infinity	Lower

Regression Metrics

Metric	Description	Range	Best
MSE	Mean squared error	0-infinity	Lower
RMSE	Root mean squared error	0-infinity	Lower
MAE	Mean absolute error	0-infinity	Lower
MAPE	Mean absolute percentage error	0-100%	Lower
R-squared	Variance explained	0-1	Higher

Cross-Validation

CV Strategies

K-Fold: Split data into k equal folds
Stratified K-Fold: Maintains class distribution
Time Series: Respects temporal order
Leave-One-Out: Each sample as test set
Group K-Fold: Groups stay together

from cyxwiz.evaluation import cross_val_score, KFold, StratifiedKFold

scores = cross_val_score(model, X, y, cv=5, scoring='accuracy')
print(f"CV Accuracy: {scores.mean():.4f} (+/- {scores.std()*2:.4f}")

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
for train_idx, val_idx in cv.split(X, y):
    X_train, X_val = X[train_idx], X[val_idx]
    y_train, y_val = y[train_idx], y[val_idx]

Usage Examples

Classification

from cyxwiz.evaluation import (
    accuracy_score,
    classification_report,
    confusion_matrix
)

acc = accuracy_score(y_true, y_pred)
report = classification_report(
    y_true, y_pred)
cm = confusion_matrix(y_true, y_pred)

Regression

from cyxwiz.evaluation import (
    mean_squared_error,
    mean_absolute_error,
    r2_score
)

mse = mean_squared_error(y_true, y_pred)
mae = mean_absolute_error(y_true, y_pred)
r2 = r2_score(y_true, y_pred)

Node Editor Integration

Node	Inputs	Outputs
Accuracy	Predictions, Labels	Accuracy score
Classification Report	Predictions, Labels	Report dict
Confusion Matrix	Predictions, Labels	Matrix tensor
ROC Curve	Probabilities, Labels	FPR, TPR, AUC
Cross Validate	Model, Data	Scores array