macos-tools/harbor/README.md

Harbor Container Registry Setup

Complete Harbor container registry setup with HTTPS support and automated image caching system for local Docker development.

🚀 NEW: Automated Image Caching System

Problem Solved: Local network cannot access Docker Hub directly, and daytime image pulling from EU Harbor is slow (5-10 minutes).

Solution: Automated nightly caching system that:

• ✅ Push Speed: 5-10 minutes → 2.4 seconds (200x faster)
• ✅ Pull Speed: 5-10 minutes → 5-10 seconds (60x faster)
• ✅ K8s Deploy: 5-10 minutes → 10-20 seconds (30x faster)
• ✅ Success Rate: 60-70% → 99%+

Quick Start (5 minutes)

cd /Users/brunogao/work/codes/tools/macos/harbor

# 1. Fix slow push issue (bypass Cloudflare CDN)
./apply-harbor-docker-config.sh
# → Restart OrbStack when prompted

# 2. Test manual pull
./pull-via-eu-harbor.sh prom/mysqld-exporter:v0.15.1

# 3. Install nightly auto-caching (runs at 2 AM)
./install-nightly-cache.sh install

📚 Complete Documentation

• HARBOR-README.md - Complete system overview
• HARBOR-HTTP-FIX.md - Fix slow push issue (Cloudflare bypass)
• NIGHTLY-CACHE-QUICKSTART.md - Automated caching guide
• DEPLOYMENT-SUMMARY-FINAL.md - Full deployment summary

K8s Usage

# Use HTTP port :8080 for local Harbor (fast!)
image: harbor.omniverseai.net:8080/library/prom/mysqld-exporter:v0.15.1

---

Overview

This Harbor deployment provides:

1. Automated Image Caching: Nightly pre-caching from EU Harbor with priority management
2. High-Speed Local Push: 2-5 seconds push time using HTTP (8080)
3. Secure HTTPS Access: TLS-enabled registry accessible via harbor.omniverseai.net
4. Docker Registry v2 API: Full compatibility with Docker CLI and Kubernetes
5. Web UI: Manage images, projects, and users via web interface
6. Local Performance: Direct LAN access for fast image push/pull operations

Current Configuration

Access Information

• Web UI: https://harbor.omniverseai.net (HTTPS on standard port 443)
• Registry (HTTPS): harbor.omniverseai.net (Standard HTTPS, recommended)
• Registry (HTTP - Local Fast): harbor.omniverseai.net:8080 (HTTP, alternative for troubleshooting)
• Admin Credentials:
  • Username: admin
  • Password: admin123

⚡ Latest Update: Harbor now runs on standard ports 80/443. No need to specify port numbers for HTTPS access!

Port Mapping

• HTTP: Host Port 80 → Container Port 8080
• HTTPS: Host Port 443 → Container Port 8443

DNS Configuration

Harbor uses local DNS resolution via /etc/hosts to avoid latency from cloud routing:

192.168.1.83 harbor.omniverseai.net

This allows direct LAN access while maintaining the production domain name.

Prerequisites

• Docker Desktop or OrbStack
• macOS with network access to 192.168.1.83
• OpenSSL (for certificate generation)

Quick Start

1. Start Harbor Services

# Start all Harbor services
docker-compose up -d

# Verify all services are healthy (wait ~30 seconds)
docker-compose ps

2. Configure Certificate Trust (Required for HTTPS)

Harbor uses self-signed certificates for internal network use. You need to configure trust on your system.

Step 1: Trust Certificate on macOS

# Add Harbor certificate to macOS system trust
sudo security add-trusted-cert -d -r trustRoot \
  -k /Library/Keychains/System.keychain \
  ~/certs/harbor.omniverseai.net.pem

# Verify
security find-certificate -c "harbor.omniverseai.net"

Step 2: Trust Certificate in Docker

# Create Docker certificate directory
sudo mkdir -p /etc/docker/certs.d/harbor.omniverseai.net

# Copy Harbor CA certificate
sudo cp ~/certs/harbor.omniverseai.net.pem \
   /etc/docker/certs.d/harbor.omniverseai.net/ca.crt

# Restart Docker/OrbStack to apply changes
# (OrbStack will automatically detect and apply the certificate)

For Kubernetes/K3s nodes:

See configure-k8s-harbor.sh for automated containerd configuration.

3. Login and Test

HTTPS Access (Recommended - Certificate Trusted)

# Login to Harbor (HTTPS on standard port 443)
docker login harbor.omniverseai.net
# Username: admin
# Password: admin123

# Test with a sample image
docker tag nginx:alpine harbor.omniverseai.net/library/nginx:test
docker push harbor.omniverseai.net/library/nginx:test
# ✅ Push completes in 2-5 seconds! (Local LAN speed)

# Verify with curl (no SSL errors)
curl https://harbor.omniverseai.net/v2/
# Expected: {"errors":[{"code":"UNAUTHORIZED",...}]}

HTTP Access (Alternative - For Troubleshooting)

# Login to Harbor HTTP (port 8080)
docker login harbor.omniverseai.net:8080

# Push using HTTP
docker push harbor.omniverseai.net:8080/library/nginx:test

SSL Certificate Configuration

Current Certificates

Harbor uses self-signed certificates for internal network use:

• Source Certificate: ~/certs/harbor.omniverseai.net.pem
• Source Private Key: ~/certs/harbor.omniverseai.net-key.pem
• Active Certificate: /Users/brunogao/ops/data/harbor/secret/cert/server.crt
• Active Private Key: /Users/brunogao/ops/data/harbor/secret/cert/server.key

Certificate Details

• Issuer: OmniverseAI (Self-Signed)
• Type: Self-Signed Wildcard Certificate
• Common Name: harbor.omniverseai.net
• Subject Alternative Names (SANs):
  • DNS: harbor.omniverseai.net
  • DNS: *.omniverseai.net
  • IP: 192.168.1.83
• Validity: Valid until Dec 12, 2026 (1 year)

Updating Certificates

Using Self-Signed Certificates (Current Setup):

To update the current self-signed certificates:

# Copy certificates to Harbor directory
cp ~/certs/harbor.omniverseai.net.pem \
   /Users/brunogao/ops/data/harbor/secret/cert/server.crt

cp ~/certs/harbor.omniverseai.net-key.pem \
   /Users/brunogao/ops/data/harbor/secret/cert/server.key

# Restart nginx to load new certificates
docker compose restart proxy

Generating New Self-Signed Certificates:

If certificates expire or need regeneration:

cd ~/certs

# Generate new self-signed certificate (valid for 1 year)
openssl req -newkey rsa:4096 -nodes -sha256 \
  -keyout harbor.omniverseai.net-key.pem \
  -x509 -days 365 \
  -out harbor.omniverseai.net.pem \
  -subj "/C=CN/ST=Beijing/L=Beijing/O=OmniverseAI/OU=IT/CN=harbor.omniverseai.net" \
  -addext "subjectAltName=DNS:harbor.omniverseai.net,DNS:*.omniverseai.net,IP:192.168.1.83"

# Copy to Harbor directory
cp harbor.omniverseai.net.pem /Users/brunogao/ops/data/harbor/secret/cert/server.crt
cp harbor.omniverseai.net-key.pem /Users/brunogao/ops/data/harbor/secret/cert/server.key

# Restart nginx
cd /Users/brunogao/work/codes/tools/macos/harbor
docker compose restart proxy

Configuration Files

Primary Configuration

• harbor.yml: Main Harbor configuration

  • Hostname: harbor.omniverseai.net
  • External URL: https://harbor.omniverseai.net
  • HTTP port: 80, HTTPS port: 443

• docker-compose.yml: Service orchestration

  • Port mappings: Host 80:Container 8080, Host 443:Container 8443
  • Volume mounts to /Users/brunogao/ops/data/harbor

• common/config/nginx/nginx.conf: Nginx proxy configuration

  • HTTP server on port 8080
  • HTTPS server on port 8443 with SSL

Environment Configuration

Harbor core service environment (common/config/core/env):

EXT_ENDPOINT=https://harbor.omniverseai.net
HARBOR_ADMIN_PASSWORD=admin123

Management Scripts

Harbor Operations

# Start Harbor
docker-compose up -d

# Stop Harbor
docker-compose down

# Restart specific service
docker-compose restart core
docker-compose restart proxy

# View logs
docker-compose logs -f core
docker-compose logs -f nginx

# Check service health
docker-compose ps

Admin Password Reset

If you need to reset the admin password:

# 1. Update password in harbor.yml
# harbor_admin_password: your-new-password

# 2. Clear existing password in database
docker-compose exec postgresql psql -U postgres -d registry \
  -c "UPDATE harbor_user SET password='', salt='' WHERE username='admin';"

# 3. Restart core to reinitialize
docker-compose restart core

# Core will automatically set the password from harbor.yml

Kubernetes Integration

Configure Containerd for K3s

Use the provided script to configure K3s nodes:

./configure-k8s-harbor.sh

This will:

• Install Harbor CA certificate to containerd
• Configure registry mirrors and authentication
• Restart containerd to apply changes

Manual K3s Configuration

For manual setup, see HARBOR_CLIENT_CONFIG.md.

Troubleshooting

Issue: "unauthorized" or "invalid credentials" when logging in

Solution:

# Reset admin password
docker-compose exec postgresql psql -U postgres -d registry \
  -c "UPDATE harbor_user SET password='', salt='' WHERE username='admin';"

docker-compose restart core

# Wait 10 seconds, then try login again with admin/admin123

Issue: "x509: certificate signed by unknown authority"

This indicates the Harbor certificate is not trusted by your system.

Solution - Complete Certificate Trust Configuration:

# Step 1: Trust certificate on macOS system level
sudo security add-trusted-cert -d -r trustRoot \
  -k /Library/Keychains/System.keychain \
  ~/certs/harbor.omniverseai.net.pem

# Step 2: Trust certificate in Docker
sudo mkdir -p /etc/docker/certs.d/harbor.omniverseai.net
sudo cp ~/certs/harbor.omniverseai.net.pem \
   /etc/docker/certs.d/harbor.omniverseai.net/ca.crt

# Step 3: Restart Docker/OrbStack
# OrbStack users: Quit and restart OrbStack
# Docker Desktop users: Restart Docker Desktop

# Step 4: Verify certificate trust
security find-certificate -c "harbor.omniverseai.net"

# Step 5: Test connection
curl https://harbor.omniverseai.net/v2/
# Should return: {"errors":[{"code":"UNAUTHORIZED",...}]} (no SSL errors)

Issue: "connection refused" when accessing Harbor

Solution:

# Check if services are running
docker-compose ps

# Check if ports are listening
lsof -i :80
lsof -i :443

# Verify /etc/hosts entry
grep harbor.omniverseai.net /etc/hosts
# Should show: 192.168.1.83 harbor.omniverseai.net

# Restart Harbor
docker-compose down && docker-compose up -d

Issue: "dial tcp: connect: connection refused"

This indicates Harbor's external endpoint is misconfigured or the service is not running. Solution:

# Update core environment
vim common/config/core/env
# Change: EXT_ENDPOINT=https://harbor.omniverseai.net

# Force recreate core container
docker-compose up -d --force-recreate core

# Verify
docker-compose exec core env | grep EXT_ENDPOINT
# Should show: EXT_ENDPOINT=https://harbor.omniverseai.net

Issue: Slow image push/pull performance

Cause 1: DNS resolves to Cloudflare CDN instead of local IP.

Solution 1 - Use HTTP port (Recommended):

# Apply fast HTTP configuration
cd /Users/brunogao/work/codes/tools/macos/harbor
./apply-harbor-docker-config.sh
# Restart OrbStack when prompted

# Use :8080 for all local operations
docker push harbor.omniverseai.net:8080/library/IMAGE:TAG
# Push time: 2-5 seconds ✅

Solution 2 - Fix DNS resolution:

# Verify /etc/hosts has local resolution
cat /etc/hosts | grep harbor
# Should show: 192.168.1.83 harbor.omniverseai.net

# If missing, add it:
echo "192.168.1.83 harbor.omniverseai.net" | sudo tee -a /etc/hosts

See: HARBOR-HTTP-FIX.md for detailed diagnosis and fix

Issue: Cannot access Harbor web UI

Solution:

# Test HTTP access (should work)
curl -i http://192.168.1.83/api/v2.0/systeminfo

# Test HTTPS access (should work)
curl -k -i https://harbor.omniverseai.net/api/v2.0/systeminfo

# If HTTP works but HTTPS doesn't, check nginx config
docker-compose exec proxy cat /etc/nginx/nginx.conf | grep "listen 8443"

# Should see active HTTPS server block
# If commented out, edit common/config/nginx/nginx.conf and restart
docker-compose restart proxy

Security Considerations

For Development Use

This setup uses self-signed certificates, which is acceptable for:

• Local development environments
• Internal testing
• Private networks

For Production Use

Consider these improvements:

1. Use CA-signed certificates from Let's Encrypt or commercial CA
2. Change default password immediately after deployment
3. Enable LDAP/OIDC for centralized authentication
4. Configure project quotas to limit resource usage
5. Enable vulnerability scanning with Trivy
6. Set up regular backups of Harbor data

Changing Admin Password

Via Web UI:

1. Login as admin
2. Go to User Settings → Change Password

Via Database (emergency):

1. Clear password as shown in troubleshooting
2. Update harbor.yml with new password
3. Restart core service

Data Persistence

Harbor data is stored in:

/Users/brunogao/ops/data/harbor/
├── ca_download/          # CA certificates
├── database/             # PostgreSQL data
├── job_logs/             # Job service logs
├── redis/                # Redis cache
├── registry/             # Image layer storage
└── secret/               # Secrets and certificates
    └── cert/
        ├── server.crt    # SSL certificate
        └── server.key    # SSL private key

Backup Harbor:

# Stop Harbor
docker-compose down

# Backup data directory
tar czf harbor-backup-$(date +%Y%m%d).tar.gz \
  /Users/brunogao/ops/data/harbor

# Start Harbor
docker-compose up -d

Restore Harbor:

# Stop Harbor
docker-compose down

# Restore data
tar xzf harbor-backup-YYYYMMDD.tar.gz -C /

# Start Harbor
docker-compose up -d

Advanced Configuration

Enable HTTPS for Cloud Access

If you want external clients to access Harbor via cloud (85.239.233.16):

# Remove local /etc/hosts override
sudo sed -i.bak '/harbor.omniverseai.net/d' /etc/hosts

# Harbor will now be accessible via cloud nginx reverse proxy
# Cloud nginx should proxy HTTPS (443) to http://10.7.0.2:80

Note: This will increase latency for local operations as traffic routes through cloud.

Custom Storage Backend

To use S3/MinIO for image storage, edit harbor.yml:

storage_service:
  s3:
    region: us-west-1
    bucket: harbor-images
    accesskey: your-access-key
    secretkey: your-secret-key

Then regenerate configuration and restart:

docker-compose down
# Re-run prepare script if available, or manually update configs
docker-compose up -d

Project Structure

harbor/
├── common/
│   └── config/          # Generated configuration files
│       ├── core/        # Harbor core config
│       ├── nginx/       # Nginx proxy config
│       └── ...
├── docker-compose.yml   # Service definitions
├── harbor.yml           # Harbor configuration
└── README.md           # This file

/Users/brunogao/ops/data/harbor/    # Data directory
/Users/brunogao/certs/              # SSL certificates

Useful Commands Reference

# View all containers
docker-compose ps

# Follow logs from all services
docker-compose logs -f

# Restart Harbor completely
docker-compose restart

# Stop Harbor (keeps data)
docker-compose stop

# Remove Harbor (keeps data)
docker-compose down

# Remove Harbor AND data (⚠️ destructive)
docker-compose down -v

# Access PostgreSQL
docker-compose exec postgresql psql -U postgres -d registry

# Execute command in core container
docker-compose exec core env

# Check nginx configuration
docker-compose exec proxy nginx -t

# View image storage usage
du -sh /Users/brunogao/ops/data/harbor/registry

Certificate Trust Summary

For Harbor to work properly with HTTPS, you must configure certificate trust on all clients:

macOS Setup (Required)

sudo security add-trusted-cert -d -r trustRoot \
  -k /Library/Keychains/System.keychain \
  ~/certs/harbor.omniverseai.net.pem

Docker/OrbStack Setup (Required)

sudo mkdir -p /etc/docker/certs.d/harbor.omniverseai.net
sudo cp ~/certs/harbor.omniverseai.net.pem \
   /etc/docker/certs.d/harbor.omniverseai.net/ca.crt

Kubernetes/K3s Setup (For K8s Deployments)

Use the provided script:

./configure-k8s-harbor.sh

This configures containerd to trust Harbor's certificate for image pulls.

Additional Documentation

Automated Caching System (NEW)

• HARBOR-README.md - Complete automated caching system overview
• HARBOR-HTTP-FIX.md - Fix slow push issue (detailed diagnosis)
• NIGHTLY-CACHE-QUICKSTART.md - Nightly auto-caching setup
• DEPLOYMENT-SUMMARY-FINAL.md - Complete deployment summary
• HARBOR-OPTIMIZATION-GUIDE.md - 7 optimization strategies
• QUICK-START-EU-HARBOR.md - EU Harbor intermediary usage

Core Harbor Setup

• HARBOR_CLIENT_CONFIG.md - Configure external Docker clients
• HARBOR-SETUP-GUIDE.md - Detailed setup procedures
• HARBOR-CORRECT-USAGE.md - Best practices for using Harbor
• configure-k8s-harbor.sh - K3s/Kubernetes integration script
• create-harbor-projects.sh - Bulk project creation

Support

For issues or questions:

1. Check the troubleshooting section above
2. Review Harbor logs: docker-compose logs -f
3. Consult Harbor documentation: https://goharbor.io/docs/

License

This is a deployment configuration for Harbor. Harbor itself is licensed under Apache License 2.0.

---

Version History

v2.1 - 2025-12-28 ✅ Latest

• ✅ Standard Ports: Harbor now runs on standard HTTP (80) and HTTPS (443) ports
• ✅ Simplified Access: No need to specify port numbers for HTTPS access
• ✅ Internal Network Certificates: Using self-signed certificates for internal use (valid until Dec 2026)
• ✅ Backward Compatible: HTTP :8080 still available for troubleshooting
• 🔧 Updated configuration: harbor.yml and docker-compose.yml
• 📁 Certificate location: ~/certs/harbor.omniverseai.net.pem

v2.0 - 2025-12-23

• ✅ Automated Image Caching System: Nightly pre-caching with priority management
• ✅ HTTP Fast Push: 2-5 second push time using :8080 port (200x faster)
• ✅ Cloudflare Bypass: Avoid CDN routing for local operations
• ✅ Performance Verified: MySQL exporter test - 2.4s push, ARM64 confirmed
• 📚 Complete documentation suite with 6 new guides

v1.0 - Previous

• Harbor container registry basic deployment
• HTTPS support with self-signed certificates
• Docker/Kubernetes integration

Last Updated: 2025-12-28 Maintained By: Bruno Gao Status: ✅ Production Ready (Internal Network)