Loading

Introduction

Calvin T. Nesbitt

This is a live streamed presentation. You will automatically follow the presenter and see the slide they're currently on.

No presentation currently in progress.

Make a free appointment today!

\( \copyright \) 2022-2024 DESIGNER SYSTEMS LLC
All Rights Reserved

D

E

S

I

g

n

E

R

s
y
s
t
e
m
s

Waiting for humans

\( \copyright \) 2022-2024 DESIGNER SYSTEMS LLC
All Rights Reserved

Presented by Calvin T. Nesbitt

Founder, Principal Engineer

Founder, Principal Engineer

calvin@designer.systems

\( \copyright \) 2022-2024 DESIGNER SYSTEMS LLC
All Rights Reserved

Managed
Services

This you must always bear in mind,
what is the nature of the whole,
and what is my nature,
and how is this related to that,
and what kind of a part it is
of what kind of a whole.

—Marcus Aurelius

consumer expectations

Managed Services

extent of modern technology

Vendor

Client

service

V

one service to many organizations

C

C

C

Vendor

many services to one organization

V

V

V

C

Client

V

V

V

C

\( \approx \) fees

V

V

V

network & switching costs

configuration costs

C

Costs

Cutover Algorithm

\( \bold a \)

V

\( \bold b \)

network & switching costs

configuration costs

C

\begin{aligned} & \textit{if} && \textcolor{#f1c232}{\text{cfg}} (\bold{a}) > \textcolor{#f1c232}{\text{cfg}} (\bold{b}) + \textcolor{#f1c232}{\text{nas}} (\bold{a, b}) \\ & \textit{then} && \text{cutover} (\bold{a, b}) \end{aligned}

matrix complexity grows

External Service Matrix

V

V

V

C

C

C

config costs rise

Bundling/PAAS

C

C

C

V

V

V

reduced config costs

increased N&S costs

many-to-many service-organization mapping

M

V

V

V

C

C

C

Managed Services Providers

multisourcing

\( - \space \text{network costs} \)

standard
cutover
procedure

\( - \space \text{switching costs} \)

staff specialization
& automation

\( - \space \text{config costs} \)

economies
of scale

\( - \space \text{config costs} \)

M

Platform Decay

V

V

V

C

C

C

The Vendor must grow

Platform Decay

M

V

V

V

C

C

C

Generalization

increased MSP fees

larger client pools

increased config costs

Platform Decay

M

C

C

C

V

\( \bold v \)

V

\( \text{nas}(\bold v, x) \propto \text{cfg}(\bold v) \)

Never attribute to malice or stupidity that which can be explained by moderately rational individuals following incentives in a complex system.

—Douglas W. Hubbard

next-gen msp

Competitive MSPs mitigate platform decay via

Service Customization

V

V

V

C

C

C

next-gen msp

Service Customization

V

V

V

C

C

C

custom scale

custom form

general service

destructuring

V

V

V

C

C

C

Designer Systems LLC

msp Policy

msp Infra

  1. Robotic Process Automation
  2. Operations Platform Integration
  3. Service-Oriented Architecture

Implementation Strata

msp Infra

msp Policy

input

output

NMS

ITSM

IAM

CRM

data vendor

client

digest

client

client

control vendors

data vendor

data vendor

Data Flow Category

Input

Campus

Dispatch

hook (reactive)

timer (proactive)

ISP/MSP

data vendors

Input

vendor-agnostic
data structure

stateless functions

Digest

Digest

Direct vendor↔vendor integration

NMS

ITSM

IAM

CRM

Interfaces between platform APIs (hopefully)

Output

\( O_a \)

\( O_b \)

\( O_c \)

Enterprise

STartup

SMB

Composable output services

NMS

ITSM

IAM

CRM

data vendor

control vendors

data vendor

data vendor

ACME, Inc.

\( \text{complex vendor} \)

input

digest

API

NMS

ITSM

IAM

CRM

data vendor

client

client

client

data vendor

data vendor

Nadir LLC

\( \text{complex client} \)

input

output

Operations

The significance of the chrysalis is change.

—Thomas Harris

Platforms

  • Virtual
    Workplace
  • HMC
  • Projects
  • Incidents
  • Interfaces
  • Daemons
  • Database
  • Rules
  • Hierarchy
  • Security
    (AAA)
  • RACI (?)
  • Secrets
  • Reporting
  • Notification
  • Feedback
  • Narrative

IT Service
Management

Identity
& Access
Management

Customer

Relationship

Management

Network
Monitoring
Systems

NMS

ITSM

CRM

IAM

presentation
procedure*

Value Streams

Stages of information development

narrative
(subjective)

business process as data structure
(objective)

human
reification

HMC pattern

permissions?
pattern

NMS

ITSM

CRM

IAM

Basic Infrastructure

(hardware, networking, virtualization, cloud, multi-cloud, etc.)

Automation!

Corporate Policy

Integration Management

NMS

ITSM

CRM

IAM

Basic Infrastructure

Corporate Policy

Promises:

  1. Replace ad-hoc automation
  2. Visualize system design
  3. Retain business logic ownership

Integration Management

NMS

ITSM

CRM

IAM

Basic Infrastructure

Corporate Policy

Realities:

  1. Partition automation
  2. Limit systems design to 2D/3D
  3. Rent third-party business platform

NMS

ITSM

CRM

IAM

Basic Infrastructure

implementation

Integration mitigates vendor change

Architecture tolerates
administration change

Corporate Policy

Automation

INTEGRATION

Architecture

Automation reflects management change

Designer Systems LLC

INTEGRATION

Automation

Infrastructure

Architecture

MSP

Full Customer
Ownership

Business Model

Technology

PRINCIPLES

Complexity

Business Model

Technology

PRINCIPLES

Flexibility

Business Model

Technology

PRINCIPLES

Factoring

Business Model

Technology

PRINCIPLES

Growth

 vs. Entropy

Business Model

Technology

PRINCIPLES

automation

integration

architecture

input

digest

output

\( \text{``An input automation"} \)

\( \text{``A digest integration"} \)

\( \text{``An output structure"} \)

Business Model

Technology

PRINCIPLES

automation

integration

architecture

input

digest

output

bloat

Business Model

Technology

PRINCIPLES

automation

integration

architecture

input

digest

output

\( \text{Refactoring} \)

Business Model

PRINCIPLES

Technology

automation
reified via

architecture
based on

integration
supervised by

Business Model

  • Full customer ownership

    • No proprietary code

    • No license fees

    • Microcontracts

      • Layered Statements of Work

      • Guaranteed Maximum Price

      • Unconditional prorated cancellation

Technology

PRINCIPLES

MSP

Service Architecture

SoW-1

Start Date

End Date

Guaranteed Maximum Price

Microcontracts

MSP

Service Architecture

SoW-1

Platform Integration

SoW-2

Start Date

End Date

Guaranteed Maximum Price

Microcontracts

MSP

Service Architecture

SoW-1

Policy Automation

SoW-3

Platform Integration

SoW-2

Start Date

End Date

Guaranteed Maximum Price

Microcontracts

MSP

Service Architecture

SoW-1

Policy Automation

SoW-3

Platform Integration

SoW-2

complete:
\( CTC = GMP \)

half-elapsed:
\( CTC \approx \frac{GMP}{2} \)

pending:
\( CTC = \$0 \)

Unconditional
Prorated Cancellation

\( CTC \approx GMP \cdot \frac{\text{elapsed days}}{\text{total days}} \)

Microcontracts

MSP

Documentation

SoW-0

Report & Proposal

SoW-1 Agreement

MSP

Documentation

SoW-0

Report & Proposal

SoW-1 Agreement

  • Customer domains
  • Basic infra, apps, integration
  • Operations hierarchy
  • Handover routine
  • Staff interviews

In Summary

  • MSPs have innate advantages
  • Platform decay: next-gen liability
  • Designer Systems: next-gen solutions
    • Internal service matrix
    • Custom implementation
    • Full client ownership
  • Increasingly complex domain
  • Flexible technical foundation
  • Let's build bridges!

\( \bf Birds \)

A true story of
correlated failure.

Afternoon of Thursday, July 4, 2019

Gainesville, VA (Artist's rendition)

Municipal Junction Boxes
High Availability Pair

\(\approx\) 600 endpoints
(Gainesville and surrounding counties)

Internet

No service
interruption

Internet

log message:
backup down,
no impact

NOC

Undisclosed
Location, VA

Site

Gainesville, VA

Dispatch

Centerville, VA

1. log

3. query

4. technician

NOC

Undisclosed

2. ticket

Fully-Automated™
Dispatch Procedure

Site

Gainesville, VA

NOC

Undisclosed

Dispatch

Centerville, VA

Technician

  • Dispatched procedurally
  • Travel paid by mileage, not time

Massive traffic jam

Highway utility corridor

Analyst

  • Skeleton crew
  • Ticket log of 400 Malaysian POS's
  • Sees no site impact

Two
Hours
Later

Site

Gainesville, VA

NOC

Undisclosed

Dispatch

Centerville, VA

ticket update:
Cause: Birds
(no resolution)

RNA

  • Receives cryptic update
  • Calls dispatch; RNA
  • Escalates ticket for managerial review

Analyst

escalation

What did the Analyst forget?

Thursday, July 4, 2019
US Independence Day

escalation

Management

NOC

\(\approx\) 600 endpoints

Internet

Municipal
customer's
"cost-saving
devices"

OS update

Router (L3)

Switch (L2)

Router (L3)

Switch (L2)

\(\approx\) 600 endpoints

  • update
    
  • reboot
  • OSPF DR election
    (Layer 3)
    

STP Storm

STP Trunk

Router (L3)

Switch (L2)

Internet

\(\approx\) 600 endpoints ISOLATED

Device flooded;
can't route WAN;
enters boot loop

Router (L3)

Switch (L2)

Internet

600 endpoints ISOLATED
PRIORITY One Major Incident

Site

NOC

Analyst

CTO

It was birds.

What's going on?

Correlated Failure™

You're fired.

Ah.

It's not the birds, it's

  • Municipal governments should have quality routers
  • Degraded parent devices should have high priority
  • Site access procedures should be reviewed carefully
  • Vendor dispatch should be measured individually
  • Staff availability should be reified regularly
  • All escalations should be reviewed promptly

Correlated Failure

  • Municipal governments should have quality routers
  • Degraded parent devices should have high priority
  • Site access procedures should be reviewed carefully
  • Vendor dispatch should be measured individually
  • Staff availability should be reified regularly
  • All escalations should be reviewed promptly
  • Northern Virginia drivers should be more careful

Correlated Failure

  • Municipal governments should have quality routers
  • Degraded parent devices should have high priority
  • Site access procedures should be reviewed carefully
  • Vendor dispatch should be measured individually
  • Staff availability should be reified regularly
  • All escalations should be reviewed promptly

Safe Harbor Statements

  • Local gov. clients can't afford quality routers
  • Hierarchical device priority varies by customer
  • Site access testing is sync. inter-organizational cost
  • Dispatch agencies don't want to identify agents
  • Telecommuting complicates staff availability
  • Escalation procedure is notoriously complex

Economic Limitations

  • Device Quality
  • Device Priority
  • Site Access
  • Agent Identity
  • Staff Availability
  • Escalation Procedure

How should we prioritize these improvements?

Site

Site

NOC

Site

To do this is not to do everything that may be necessary; but it is to make all else easier. And unless we do this, nothing else will avail.
— Henry George
  • Device Quality
  • Device Priority
  • Site Access
  • Agent Identity
  • Staff Availability
  • Escalation Procedure

Reification

Site

Site

NOC

Site

NMS

ITSM

CRM

IAM

Domain
Specific
Business
Logic

Reification

\( \textit{of} \)

\( \textit{prevents} \)

Reification

COrrelated

Failure

Domain
Specific
Business
Logic

Infrastructure &
the System Layer

Generic Architecture

Board, CPU, Memory, Disk, Cards

Login, Desktop/WM, Graphics

init system, Logging, Network

Drivers, Scheduling, IPC, lib

Settings GUI, File Viewer, Web Browser*

Application

Userland

System

Kernel

Hardware

c

c

c

c

c

\( \Bigg \rbrace \)

d

d

d

d

d

threads
 processes 
services
sessions

...

service
init system
process

App

Usr

Sys

Krn

Hw

Apple

Aqua

Carbon,
Darwin

xnu

??

OSX

Operating Systems

Apple

Aqua

Carbon,
Darwin

xnu

??

OSX

General

Environment

Integral,
Executive

HAL

*.exe

NT

App

Usr

Sys

Krn

Hw

Operating Systems

Steve Ballmer shouting "DEVELOPERS, DEVELOPERS, DEVELOPERS" at a Microsoft Windows conference.

There's no chance that the iPhone is going to get any significant market share.
No chance.

—Steve Ballmer, 2007

Apple

Aqua

Carbon,
Darwin

xnu

??

OSX

General

Environment

Integral,
Executive

HAL

*.exe

NT

App

Usr

Sys

Krn

Hw

Operating Systems

Anything

GNOME

Linux

RPM

RHEL

systemd

Traditional init

systemd
systemd
systemd

Independent services

  • Boot scripts
  • Resource management
  • Data access
  • Configuration tooling

Complete centralization

  • Boot sequence
  • Containerization
  • Binary data format
  • System-wide DSL
  • Everything is a unit
    (no longer a file)

systemd

Apple

Aqua

Carbon,
Darwin

xnu

??

OSX

General

Environment

Integral,
Executive

HAL

*.exe

NT

Anything

GNOME

Linux

RPM

RHEL

systemd

App

Usr

Sys

Krn

Hw

Operating Systems

We have taken
all the IBM software
and optimized it for
the Red Hat OpenShift platform.

—Arvind Krishna, CEO
(2022, four years after Red Hat aquisition)

I have a suspicion that a lot of people are using containers where they probably shouldn’t.

—Lennart Poettering, systemd creator

Apple

Aqua

Carbon,
Darwin

xnu

??

OSX

General

Environment

Integral,
Executive

HAL

*.exe

NT

App

Usr

Sys

Krn

Hw

Operating Systems

GNU

Linux/Hurd

Guix CLI, sshd

Guix System,

Shepherd

Guix

FOSS*

Anything

GNOME

Linux

RPM

RHEL

systemd

 the original daemon-managing daemon

update configuration or invoke service actions

manage daemons

command

interact via shepherd

reply

init, monitor, restart

Service A

Service B

User

1.

2.

3.

 the original daemon-managing daemon

One language; many forms of business logic

Service A

Service B

User

Service-
specific

Service-interactive

User-interactive

Holistic

Virtualization

Userland

System

Kernel

Hardware

Application

In a nutshell:

  • Hardware and Kernel slow
  • Everything else (can be) fast

slow

fast

Virtualization

System

Kernel

Hardware

Login sessions are units of system virtualization

Session Manager

Userland

Application

Userland

Application

slow

fast

session

Virtualization

System

Kernel

Hardware

Session Manager

Window system

Userland

Application

Application

Application

Windows are units of userland virtualization

slow

fast

session

window

Virtualization

System

Kernel

Hardware

Session Manager

Window system

Userland

Application

Application

Browser

Tab

Tab

Tab

Some applications virtualize themselves

slow

fast

session

window

Virtualization

Userland

System

Kernel

Hardware

Application

slow

fast

Virtual Machines

Units of hardware virtualization

Hypervisor

Userland

System

Kernel

Hardware

Application

Userland

System

Kernel

Application

vm

slow

fast

Containers

Userland

System

Kernel

Hardware

Application

Userland

System

Application

Container Runtime

container

slow

fast

Units of kernel virtualization

Userland

System

Hardware

Application

Userland

System

Application

Container RUNTIME

Hypervisor

Kernel

Kernel

Container Runtime

Userland

System

Application

Userland

System

Application

container

vm

Many organizations which containerize also hypervize

vm

Production

Guile

Guix CLI

SSHD

Guix System

RUNTIME

Kernel

Hypervisor

Hardware

container

Userland

System

Application

Language VM

Interface (CI/User)

MSP-adminstrated

negotiated
environment

Example Configuration

vm

...
Guix System

RUNTIME

Kernel

Hypervisor

Server/Instance

container

Bare Metal

Custom Linux

...
Guix System

compilation,
configuration
installation

vm

Server/Instance

Hypervisor

Linux Libre

...
Guix System

configuration

1.

2.

3.

negotiated environments

Common Infrastructure

VPN

Example Deployment Tiers

dev
int
prod

SSHD

Guile

vm

Guix CLI

Guix System

Linux Libre

SSHD

Guile

vm

Guix CLI

Guix System

Linux Libre

Unit Testing

vm

Bare Metal

Hypervisor

VPC*

SSHD

Guile

Guix CLI

Guix System

Linux Libre

Production

Testing  Fallback

Wielding Infrastructure

Configuration

Management

Old-School System Administration

Sysadmin

Production OS

Direct filesystem edits

Detailed administration

Granular system feedback

\( \cdots \)

User script initialization

Procedural changes

initial state

edited state

scripted state

viable state (?)

Reckless improvisation

Communion with The Machine

Infrastructure Provisioning/Automation

DevOps
\( \text{``Engineer"} \)

OS

Declarative DSL

Provisioning
System

Sequential
Filesystem
Changes

???

DevOps
\( \text{``Engineer"} \)

+ Virtualized OS

Declarative DSL

Provisioning
System

Sequential
Filesystem
Changes

Backup system image

+ Backup
Repository

Retrieve & restore backup

Infrastructure Provisioning/Automation

\( \text{``Developer"} \)

Declarative DSL

Reproducible Operating Systems

The Store

Build
System

Symbolic
Filesystem

single system instance

\( \tt derivations \)

\( \big \} \space \tt outputs \space \checkmark \)

Irresponsibly rad hacking montage

Reconfigure to last generation

reconfiguration (atomic)

generation

\( G_{n+2} \)

\( G_{n+1} \)

\( G_n \)

\( G_{n+1} \)

\( G_{[\![0,n+2]\!]} \)

Reconfigure to any generation

one or more files

Filesystem

User

FILE

access

(Disk, NAS, RAID controller, etc.)

The Store

FILE

symlink

access
reference

Filesystem

User

The Store

FILE

Build System

FILE*

symlink

access

Filesystem

error
create
reference

User

Build errors
can't interfere
with users!

The Store

FILE*

symlink

reference

FILE

Filesystem

Build System

User

access
reconfigure

STATE Monad

The Store

FILE*

STATE Monad

symlink

reference

FILE

access

Filesystem

User

\( G_{n+1} \)

\( G_n \)

The Store

generation0

Directory

File

Directory

State Monad

generation2

FILE*

generation1

File

File

Build System

modify
1 file

create
2 files

The Store

Reconfiguration

generation0

Directory

File

Directory

State Monad

generation2

FILE*

generation1

File

File

Build System

modify
1 file

create
2 files

Symlinked
Filesystem

File

Directory

FILE*

File

Directory

Reconfiguration

generation0

Directory

File

Directory

State Monad

generation2

FILE*

generation1

File

File

Build System

modify
1 file

create
2 files

Symlinked
Filesystem

Directory

File

Directory

Userland

Kernel

Hardware

Application

Generation

Filesystem

System

Generation

The Store

 

Build System

State Monad

delimited system-layer virtualization

Userland

Kernel

Hardware

Application

Gen

System

Gen

The Store

Userland

System

Kernel

Hardware

Application

Userland

System

Application

Container Runtime

\( C_n \)

filesystem modified "inline"

changes require rerouting
data flow between containers

\( C_{n+1} \)

If reproducibility is so important,
why isn't everyone using NixOS?

2003

Nix

2015

NixOS Foundation

2024

🤔

2006

NixOS

  • Fragmented Ecosystem

    • Nix, Nixpkgs, NixOS, NixOps, Hydra, various utilities

    • Written in C++, Haskell, and Nix DSL

    • Configured via hardcoded extensions of Nix DSL

    • Documentation inconsistent or even incomplete

    • Haskell community 😬

  • Nix DSL

    • Textbook scope-bloated DSL

    • Too complex to coexist peacefully with other DSLs

    • Not powerful or flexible enough for general purpose

    • Depends heavily on shell scripting, config embed

    • Ecosystem extends the DSL layer by layer

    • Extension tools unavailable to end user

Nix

The Store

C++

Haskell

NixOS DSL

Nix DSL

NixOps DSL

NixOps

Nixpkgs

NixOS

package manager

package repository

operating system

infrastructure

Nix Ecosystem

Nix

NixOS DSL

Nix DSL

NixOps DSL

NixOps

Nixpkgs

NixOS

Haskell

Vendor

Client

design

configuration

language

service

Nix

The Store

Nixpkgs

C++

Haskell

NixOS DSL

NixOS (operating system)

Nix DSL

NixOps (infrastructure)

NixOps DSL

systemd

HUBRIS

C

Guix

The Store

C++

Guile

Guix System

Guix Packages

package manager

package repository

operating system

init system

Guix Deploy

infrastructure

GNU Guix

  • Homoiconic Abstract Syntax Tree (LISP)

  • Procedural Syntax Transformation (macros)

  • Quasi-functional paradigm (w/ optional OOP)

  • Lexical & dynamic scope

  • Delimited continuations

  • REPL-driven service introspection, interactivity

  • Strong Linux environment integration

  • 30 years of practical extensions

Coding

Expansion

Compilation & Runtime

Application

(not at all to scale)

Generated uniform procedural codebase

EDSL

EDSL

EDSL

Module

Module

Expression logging

Syntax data
at runtime
(CI/CD boon)

Pattern-based
programming

Guix

The Store

C++

Shepherd

Guix System

Guix Packages

Guile

Guix Deploy

Guix

The Store

C++

Guix System

Guix Packages

Guile

Guix Deploy

systemd

Guix Deploy

Guix

The Store

C++

Guix System

Guix Packages

Guile

Guix

The Store

C++

Guile

Shepherd

Guix System

Guix Packages

Guix Deploy

int
prod

CI/CD

Infrastructure

dev

Integration Services

Platforms

\( \Bigg \{ \)

Third-party
development

Automation Services

Corporate Policy

Mgmt.

Admin.

Designer Systems LLC | Logo Designer Systems LLC | Logo

Service Architecture

Infrastructure

Integration

Policy

Automation

Architecture

Platforms

Mgmt.

CI/CD

Designer Systems LLC | Logo Designer Systems LLC | Logo

Admin.

Third-party
development

Infrastructure

Integration

Policy

Automation

Architecture

Platforms

Mgmt.

Admin.

Third-party
development

CI/CD

Designer Systems LLC | Logo Designer Systems LLC | Logo

Conclusion

  • GNU Guix is a second-generation
    Reproducible Operating System
    • Infrastructure agnostic
    • Holistic systems control
    • Extensible configuration
  • Designer Systems extends Guix
    • Reified business languages
    • System-layer service design
    • Composed application layer

\( \copyright \) 2022-2024 DESIGNER SYSTEMS LLC
All Rights Reserved

Thank you!

Business traffic

\( \cdots \)

NOC

dev
int 
prod

CI/CD

\( \checkmark G_{n+1} \)

Business traffic

Reconfigure

Manual or CI/CD

\( \checkmark \space \text{Int, E2E tests} \)

\( \xRightarrow{\tt generation} G_{n+1} \)

Route traffic to integration

Business traffic

(as production fallback)

Take down production

\( \checkmark \space \text{Reboot, e.g.} \)

Route traffic to production

Business traffic

\( \cdots \)

(as test platform)

Risky merge

\( \text{\textdagger \space System failure} \)

Reconfigure

\( \checkmark \text{Generation} \space G_n \)

Careful merge

\( \checkmark \text{Unit tests} \)

The Int Tier

NOC

int 
prod
dev

whiteboard