Introduction
This is a live streamed presentation. You will automatically follow the presenter and see the slide they're currently on.
This is a live streamed presentation. You will automatically follow the presenter and see the slide they're currently on.
No presentation currently in progress.
Make a free appointment today!
s
y
s
t
e
m
s
Waiting for humans
Presented by Calvin T. Nesbitt
Founder, Principal Engineer
Founder, Principal Engineer
calvin@designer.systems
This you must always bear in mind,
what is the nature of the whole,
and what is my nature,
and how is this related to that,
and what kind of a part it is
of what kind of a whole.
—Marcus Aurelius
consumer expectations
extent of modern technology
Vendor
Client
service
V
one service to many organizations
C
C
C
many services to one organization
V
V
V
C
V
V
V
C
\( \approx \) fees
V
V
V
network & switching costs
configuration costs
C
\( \bold a \)
V
\( \bold b \)
network & switching costs
configuration costs
C
matrix complexity grows
V
V
V
C
C
C
config costs rise
C
C
C
V
V
V
reduced config costs
increased N&S costs
many-to-many service-organization mapping
M
V
V
V
C
C
C
multisourcing
\( - \space \text{network costs} \)
standard
cutover
procedure
\( - \space \text{switching costs} \)
staff specialization
& automation
\( - \space \text{config costs} \)
economies
of scale
\( - \space \text{config costs} \)
M
V
V
V
C
C
C
The Vendor must grow
M
V
V
V
C
C
C
Generalization
increased MSP fees
larger client pools
increased config costs
M
C
C
C
V
\( \bold v \)
V
\( \text{nas}(\bold v, x) \propto \text{cfg}(\bold v) \)
Never attribute to malice or stupidity that which can be explained by moderately rational individuals following incentives in a complex system.
—Douglas W. Hubbard
Competitive MSPs mitigate platform decay via
V
V
V
C
C
C
V
V
V
C
C
C
custom scale
custom form
general service
destructuring
V
V
V
C
C
C
Robotic Process Automation
Operations Platform Integration
Service-Oriented Architecture
Implementation Strata
input
output
data vendor
client
digest
client
client
control vendors
data vendor
data vendor
Data Flow Category
hook (reactive)
timer (proactive)
data vendors
vendor-agnostic
data structure
stateless functions
Digest
Direct vendor↔vendor integration
Interfaces between platform APIs (hopefully)
\( O_a \)
\( O_b \)
\( O_c \)
Composable output services
data vendor
control vendors
data vendor
data vendor
ACME, Inc.
\( \text{complex vendor} \)
input
digest
API
data vendor
client
client
client
data vendor
data vendor
Nadir LLC
\( \text{complex client} \)
input
output
The significance of the chrysalis is change.
—Thomas Harris
IT Service
Management
Identity
& Access
Management
Customer
Relationship
Management
Network
Monitoring
Systems
NMS
ITSM
CRM
IAM
presentation
procedure*
Stages of information development
narrative
(subjective)
business process as data structure
(objective)
human
reification
HMC pattern
permissions?
pattern
(hardware, networking, virtualization, cloud, multi-cloud, etc.)
Corporate Policy
Corporate Policy
Promises:
Corporate Policy
Realities:
implementation
Integration mitigates vendor change
Architecture tolerates
administration change
Corporate Policy
Automation reflects management change
Designer Systems LLC
Full Customer
Ownership
Complexity
Flexibility
Factoring
Growth
vs. Entropy
automation
integration
architecture
input
digest
output
\( \text{``An input automation"} \)
\( \text{``A digest integration"} \)
\( \text{``An output structure"} \)
automation
integration
architecture
input
digest
output
bloat
automation
integration
architecture
input
digest
output
\( \text{Refactoring} \)
automation
reified via
architecture
based on
integration
supervised by
Full customer ownership
No proprietary code
No license fees
Microcontracts
Layered Statements of Work
Guaranteed Maximum Price
Unconditional prorated cancellation
Service Architecture
SoW-1
Start Date
End Date
Guaranteed Maximum Price
Service Architecture
SoW-1
Platform Integration
SoW-2
Start Date
End Date
Guaranteed Maximum Price
Service Architecture
SoW-1
Policy Automation
SoW-3
Platform Integration
SoW-2
Start Date
End Date
Guaranteed Maximum Price
Service Architecture
SoW-1
Policy Automation
SoW-3
Platform Integration
SoW-2
complete:
\( CTC = GMP \)
half-elapsed:
\( CTC \approx \frac{GMP}{2} \)
pending:
\( CTC = \$0 \)
Unconditional
Prorated Cancellation
\( CTC \approx GMP \cdot \frac{\text{elapsed days}}{\text{total days}} \)
Documentation
Report & Proposal
SoW-1 Agreement
Documentation
Report & Proposal
SoW-1 Agreement
\( \bf Birds \)
A true story of correlated failure.
Afternoon of Thursday, July 4, 2019
Gainesville, VA (Artist's rendition)
Municipal Junction Boxes
High Availability Pair
\(\approx\) 600 endpoints
(Gainesville and surrounding counties)
No service
interruption
log message:
backup down,
no impact
Undisclosed
Location, VA
Gainesville, VA
Centerville, VA
1. log
3. query
4. technician
Undisclosed
2. ticket
Gainesville, VA
Undisclosed
Centerville, VA
Technician
Massive traffic jam
Highway utility corridor
Analyst
Gainesville, VA
Undisclosed
Centerville, VA
ticket update:
Cause: Birds
(no resolution)
RNA
Analyst
escalation
What did the Analyst forget?
escalation
Management
Municipal
customer's
"cost-saving
devices"
OS update
Router (L3)
Switch (L2)
Router (L3)
Switch (L2)
update
reboot
OSPF DR election (Layer 3)
STP Storm
STP Trunk
Router (L3)
Switch (L2)
Device flooded;
can't route WAN;
enters boot loop
Router (L3)
Switch (L2)
Analyst
CTO
It was birds.
What's going on?
You're fired.
Ah.
It's not the birds, it's
How should we prioritize these improvements?
To do this is not to do everything that may be necessary; but it is to make all else easier. And unless we do this, nothing else will avail.
— Henry George
\( \textit{of} \)
\( \textit{prevents} \)
Board, CPU, Memory, Disk, Cards
Login, Desktop/WM, Graphics
init system, Logging, Network
Drivers, Scheduling, IPC, lib
Settings GUI, File Viewer, Web Browser*
Application
Userland
System
Kernel
Hardware
c
c
c
c
c
\( \Bigg \rbrace \)
d
d
d
d
d
threads
processes
services
sessions
service
init system
process
App
Usr
Sys
Krn
Hw
Apple
Aqua
Carbon,
Darwin
xnu
??
OSX
Apple
Aqua
Carbon,
Darwin
xnu
??
OSX
General
Environment
Integral,
Executive
HAL
*.exe
NT
App
Usr
Sys
Krn
Hw
There's no chance that the iPhone is going to get any significant market share.
No chance.
—Steve Ballmer, 2007
Apple
Aqua
Carbon,
Darwin
xnu
??
OSX
General
Environment
Integral,
Executive
HAL
*.exe
NT
App
Usr
Sys
Krn
Hw
Anything
GNOME
Linux
RPM
RHEL
systemd
Traditional init
systemd
systemd
systemd
Independent services
Complete centralization
systemd
Apple
Aqua
Carbon,
Darwin
xnu
??
OSX
General
Environment
Integral,
Executive
HAL
*.exe
NT
Anything
GNOME
Linux
RPM
RHEL
systemd
App
Usr
Sys
Krn
Hw
We have taken
all the IBM software
and optimized it for
the Red Hat OpenShift platform.
—Arvind Krishna, CEO
(2022, four years after Red Hat aquisition)
I have a suspicion that a lot of people are using containers where they probably shouldn’t.
—Lennart Poettering, systemd creator
Apple
Aqua
Carbon,
Darwin
xnu
??
OSX
General
Environment
Integral,
Executive
HAL
*.exe
NT
App
Usr
Sys
Krn
Hw
GNU
Linux/Hurd
Guix CLI, sshd
Guix System,
Shepherd
Guix
FOSS*
Anything
GNOME
Linux
RPM
RHEL
systemd
the original daemon-managing daemon
update configuration or invoke service actions
manage daemons
command
interact via shepherd
reply
init, monitor, restart
Service A
Service B
User
1.
2.
3.
the original daemon-managing daemon
One language; many forms of business logic
Service A
Service B
User
Service-
specific
Service-interactive
User-interactive
Holistic
Userland
System
Kernel
Hardware
Application
In a nutshell:
slow
fast
System
Kernel
Hardware
Login sessions are units of system virtualization
Userland
Application
Userland
Application
slow
fast
session
System
Kernel
Hardware
Userland
Application
Application
Application
Windows are units of userland virtualization
slow
fast
session
window
System
Kernel
Hardware
Userland
Application
Application
Tab
Tab
Tab
Some applications virtualize themselves
slow
fast
session
window
Userland
System
Kernel
Hardware
Application
slow
fast
Units of hardware virtualization
Userland
System
Kernel
Hardware
Application
Userland
System
Kernel
Application
vm
slow
fast
Userland
System
Kernel
Hardware
Application
Userland
System
Application
container
slow
fast
Units of kernel virtualization
Userland
System
Hardware
Application
Userland
System
Application
Kernel
Kernel
Userland
System
Application
Userland
System
Application
container
vm
Many organizations which containerize also hypervize
vm
Production
Guix CLI
Guix System
Kernel
Hardware
container
Userland
System
Application
Language VM
Interface (CI/User)
MSP-adminstrated
negotiated
environment
vm
...
Guix System
Kernel
Server/Instance
container
Bare Metal
Custom Linux
...
Guix System
compilation,
configuration
installation
vm
Server/Instance
Linux Libre
...
Guix System
configuration1.
2.
3.
negotiated environments
dev
int
prod
vm
Guix CLI
Guix System
Linux Libre
vm
Guix CLI
Guix System
Linux Libre
Unit Testing
vm
Bare Metal
Guix CLI
Guix System
Linux Libre
Production
Testing Fallback
Wielding Infrastructure
Old-School System Administration
Sysadmin
Production OS
Direct filesystem edits
Detailed administration
Granular system feedback
\( \cdots \)
User script initialization
Procedural changes
initial state
edited state
scripted state
viable state (?)
Reckless improvisation
Communion with The Machine
Infrastructure Provisioning/Automation
DevOps
\( \text{``Engineer"} \)
OS
Declarative DSL
Provisioning
System
Sequential
Filesystem
Changes
???
DevOps
\( \text{``Engineer"} \)
+ Virtualized OS
Declarative DSL
Provisioning
System
Sequential
Filesystem
Changes
Backup system image
+ Backup
Repository
Retrieve & restore backup
Infrastructure Provisioning/Automation
\( \text{``Developer"} \)
Declarative DSL
Reproducible Operating Systems
The Store
Build
System
Symbolic
Filesystem
single system instance
\( \tt derivations \)
\( \big \} \space \tt outputs \space \checkmark \)
Irresponsibly rad hacking montage
Reconfigure to last generation
reconfiguration (atomic)
generation
\( G_{n+2} \)
\( G_{n+1} \)
\( G_n \)
\( G_{n+1} \)
\( G_{[\![0,n+2]\!]} \)
Reconfigure to any generation
one or more files
access
(Disk, NAS, RAID controller, etc.)
symlink
access
reference
symlink
access
error
create
reference
Build errors
can't interfere
with users!
symlink
reference
access
reconfigure
symlink
reference
access
\( G_{n+1} \)
\( G_n \)
generation0
Directory
File
Directory
State Monad
generation2
generation1
File
File
Build System
modify
1 file
create
2 files
generation0
Directory
File
Directory
State Monad
generation2
generation1
File
File
Build System
modify
1 file
create
2 files
Symlinked
Filesystem
File
Directory
File
Directory
generation0
Directory
File
Directory
State Monad
generation2
generation1
File
File
Build System
modify
1 file
create
2 files
Symlinked
Filesystem
Directory
File
Directory
Userland
Kernel
Hardware
Application
System
Build System
State Monad
delimited system-layer virtualization
Userland
Kernel
Hardware
Application
System
Userland
System
Kernel
Hardware
Application
Userland
System
Application
\( C_n \)
filesystem modified "inline"
changes require rerouting
data flow between containers
\( C_{n+1} \)
If reproducibility is so important,
why isn't everyone using NixOS?
2003
Nix
2015
NixOS Foundation
2024
🤔
2006
NixOS
Fragmented Ecosystem
Nix, Nixpkgs, NixOS, NixOps, Hydra, various utilities
Written in C++, Haskell, and Nix DSL
Configured via hardcoded extensions of Nix DSL
Documentation inconsistent or even incomplete
Haskell community 😬
Nix DSL
Textbook scope-bloated DSL
Too complex to coexist peacefully with other DSLs
Not powerful or flexible enough for general purpose
Depends heavily on shell scripting, config embed
Ecosystem extends the DSL layer by layer
Extension tools unavailable to end user
Nix
The Store
NixOS DSL
Nix DSL
NixOps DSL
NixOps
Nixpkgs
NixOS
package manager
package repository
operating system
infrastructure
Nix
NixOS DSL
Nix DSL
NixOps DSL
NixOps
Nixpkgs
NixOS
Vendor
Client
design
configuration
language
service
Nix
The Store
Nixpkgs
NixOS DSL
NixOS (operating system)
Nix DSL
NixOps (infrastructure)
NixOps DSL
Guix
The Store
Guix System
Guix Packages
package manager
package repository
operating system
init system
Guix Deploy
infrastructure
Homoiconic Abstract Syntax Tree (LISP)
Procedural Syntax Transformation (macros)
Quasi-functional paradigm (w/ optional OOP)
Lexical & dynamic scope
Delimited continuations
REPL-driven service introspection, interactivity
Strong Linux environment integration
30 years of practical extensions
Simple Syntax
Coding
Expansion
Compilation & Runtime
Application
(not at all to scale)
Generated uniform procedural codebase
EDSL
EDSL
EDSL
Module
Module
Expression logging
Syntax data
at runtime
(CI/CD boon)
Pattern-based
programming
Guix
The Store
Shepherd
Guix System
Guix Packages
Guix Deploy
Guix
The Store
Guix System
Guix Packages
Guix Deploy
Guix Deploy
Guix
The Store
Guix System
Guix Packages
Guix
The Store
Shepherd
Guix System
Guix Packages
Guix Deploy
int
prod
CI/CD
dev
\( \Bigg \{ \)
Third-party
development
Mgmt.
Admin.
Mgmt.
CI/CD
Admin.
Third-party
development
Mgmt.
Admin.
Third-party
development
CI/CD
Business traffic
\( \cdots \)
NOC
dev
intprod
CI/CD
\( \checkmark G_{n+1} \)
Business traffic
Reconfigure
Manual or CI/CD
\( \checkmark \space \text{Int, E2E tests} \)
\( \xRightarrow{\tt generation} G_{n+1} \)
Route traffic to integration
Business traffic
(as production fallback)
Take down production
\( \checkmark \space \text{Reboot, e.g.} \)
Route traffic to production
Business traffic
\( \cdots \)
(as test platform)
Risky merge
\( \text{\textdagger \space System failure} \)
Reconfigure
\( \checkmark \text{Generation} \space G_n \)
Careful merge
\( \checkmark \text{Unit tests} \)
NOC
intprod
dev
whiteboard