PyWacket Project Page

PyWacket is a loosely-defined integration framework for using several computers/devices to achieve some goal.

It is intended primarily for desktops and servers, but should be general enough to work for almost any application that depends on a collection of software packages running on more than one machine.

At this point, the project has trundled along for several years and consumed multiple hundreds of hours. If done 'properly', it could consume several thousand hours, so clearly setting concrete objectives is more critical than a particular technical capability.

Recently ( since March 2016 ), I've begun to use term machines to include both computers and devices. Modern computers ( particularly the Raspberry Pi ) are able to control small, specialized devices, such as cameras, arrays of sensors, actuators to turn kitchen appliances on and off, etc. They are a very natural extension for a current project in integration.

Paradoxically, expanding the scope of the integration framework to include something like 'smart devices' seems to be creating a sharper focus rather than confusing the original intent of the project - one of the few times in my experience that expanding scope is making it easier to achieve the goal of a project ( so far ).

See ​ PyWacket on semantastic.com.

Also see project sub-pages for Capabilities, Components, To Do List and " Need To Explore".

Overview

Three dimension of conceptual ( or analytic ) framework implied by PyWacket requirements.

• Function - what a user can do and the benefit received.
• Platform - the operating environment, the structural parts both hardware and software infrastructure, different configurations.
• Technology - the processing/transforming required to meet the functional requirements on a given platform.

The categories are not 'pure' - some are fuzzy, others overlap. Certainly there are gaps as well. The categories are adequate for an overview, but may reveal weaknesses once a particular implementation gets down to details.

---

---

Implicit is the model is the Bang/Buck ratio. Bang is what you get ( functional bang ) - Buck is the cost in time, money and effort to get technology going on a given platform.

The overall purpose of the project is to maximize bang for the minimum buck, with a pretty tight rein on the buck budget. Another way to express it is to maximize functional coverage with minimal software dependencies. "Quick and easy" is the watchword.

Maximizing user-level performance and minimizing resource consumption is also an overall goal of the project.

About Computer Languages

Python is the preferred computer language for providing necessary glue code and logic for actual integration ( hence the 'Py' in 'PyWacket' ).

There may be must-have applications in other languages, particularly Java and Node JS, but any language is acceptable if it does the job with little or no effort ( under the guiding auspices of Python ). Erlang anyone ?

In fact, one of the lessons of the project so far is not to get too hung up on language features and perceived advantages of one language over the other. The size of the language community is probably more important than the language itself ...

Given Functional Arenas

Several 'givens' or 'functional arenas' for desktops ( pretty much in this order):

• MS Windows
• Linux
• Apple-OSX , when possible

Note that these desktops can also function as servers for most types of localized network services.

Plus two server-only arenas:

• Linux Servers, including Linux running on a mobile device.
• Cloud Services

Plus support in the longer term for Android, either as servers or 'mobile desktops'. Android is a Java environment so ...

Apple-OSX can be a problem. One would think that Python on a Mac is pretty much the same as Linux, MS Windows is going to be the problem obviously. However such does not seem to be the case for many Python packages. For instance, Greenlets are a problem on the Mac. In fact, installations of many Python packages are easier on MS Windows than other platforms, including Linux.

These are assumed to be public ( VPS, shared or Cloud ) or semi-private ( VPS or Cloud ) servers on the Web/Internet, with no GUI other than HTML5 as a GUI, which makes HTML5 highly desirable and something like a requirement in its own right.

Motivation

Sharing facilities among common core ( peer ), sharing unique facilities ( client-server )

Things that can be measured - "measures of goodness".

• Increase reliability by redundancy over machines
  • Avoid email disasters, zero loss.
  • Backup, zero loss.
  • Example, disk failure on main desktop but I'm up and running again in 15 minutes

• More Screen real estate
  • Browsing on two machines, one might be documentation

• Better distribution of idle machine resources
  • On the average, the typical desktop machine uses under 10% of resources such as CPU, memory
  • Many large disk drives are nearly empty
  • Peak demand on one machine can be offset by moving discrete services to another machines via the network
  • Luckily, most home networks use under 0.1% of total capacity

• Combining machine capacities for better workflow
  • Batch image processing on several machines
  • Pipeline process steps across machines
  • high utilization of multiple machines

• Visibility Across Machines
  • Finding out about services on each machine
  • Unified file tracking and access
  • Shared data and information
  • Less time hunting around for files, passwords, etc.

• Quick and Easy Reconfiguration
  • Moving services around on the fly
  • Redirecting clients to new service location

• Specialization of Machine-OS-User
  • For example, browse on one computer, play music on other
  • Specialized user, commonly admin or not, but maybe audio etc.
  • Personal machine / business machine
  • Various Windows-Linux-Android combos
  • Android - Desktop, Windows / Linux

• Specialization of Configuration
  • Share single mysql server in local network
  • Local Linux server in Windows World

• Autonomy
  • Less dependence on possibly invasive services on the web ... naming no names.
  • Less reliance on the accidents and vagaries of network access via IP, via rich 'featureful' local network.
    • For example, running a local caching proxy to reduce network access, potentially to zero if need be.

Motivations include the overall goals of knowledge management, such as improved performance, better integration and continuous process improvement.

Usage

Note: this section needs to be updated with Home Automation functions, but I still don't have a clear idea of what those functions are.

What I'm doing to produce the value, across multiple machines, multiple OSes, multiple users.

Needs some sort of table/matrix mapping functions back to motivations.

A jumble of stuff, some redundant, some overlapping.

Core Functions

• Email
  • integration
  • archiving

• Links
  • Master links
  • Merging multiple sources, such as Mozilla SQLite DBs
  • Universal-Cosmic browsing history-wiki

• File Management
  • Documents, various doc formats, versioning
  • documents across systems, something like DropBox
  • search for documents across systems, even if machine is turned off or not connected

• File Versioning - Directory Management
  • Synchronize File over machines
  • Backup / Directory Synchronization

• Managing external devices and device IDs, such as USB drives across machine-os
  • Mass backup, copying drives
  • Bootable backup devices

• Cut & Paste Across Different Applications on Different Systems
  • Links
  • Text in general
  • Formatted Text
    • Something like a universal text format, creole markup ?
  • Files - binaries such as images, audio

Messaging

• Messaging
  • status changes
  • logging
  • running history
  • small pieces/payloads of text

• Chat
  • pasting links. text

Multimedia

• Music
  • ripper
  • organizer

• Audio Studio
  • record music
  • make a CD

• Scanning & Processing Images
  • Photo / Video Libraries
  • Presentations

Projects

• Project Management
  • plans, resources, events
  • documents, wikis
  • images, audio
  • software

• General Knowledge Base
  • About configs, programs / params, bash commands etc.
  • Lessons learned, using Shared hosting for instance
  • Info projects, major searches, finding a new printer for ex.
  • Creating public blogs, wikis

Administration

• Configuration Management
  • base install, new install
  • systems recovery, change propagation
  • managing multiple applications

• History & Tracking
  • What I did, when I did it, plus spurious info.
  • Keeping track of message threads
  • Putting the pieces together
  • configuration evolution

Small Office

A collection of small office type of usages. May be the most important, high priority functions. They have economic value that justifies significant time and effort invested on the front end, buying less time and hassle in the long run.

For instance, a marginal improvement in security can save vast amount of time and hassle in the long run.

Definitely an 'under-servered' market. Implies the ability to deliver small quick nearly invisible public presence and security monitors

Major areas are:

• Document production, word processing automation and macros etc.
• Presentations
• Mailings
• Contact Lists
• Contact Histories
• Planning, Tracking, Projects
• Etc.

Needs more attention ...

MS Windows Scenarios

A functional requirement in itself.

• Keep old Windows 2000 / XP technology going after support cutoff
  • Use Linux to filter network traffic and hide Windows vulnerabilities
  • User learns Linux on an old ( even older ) machine
  • Running Windows virtual under Linux

• I'm a Windows user looking for a better world
  • migrating functions across machines
    • en masse
    • over time

Android Scenarios

• Stuff sitting on my Android I need to track
• Support for specialized functions
  • The Android Music Machine !

Home Automation

Uncertain about functions. Heating, lights, potentially anything electrical or electronic, or that can be controlled electrically.

A statement of motivations might include 'to save money', 'to feel more secure', 'more in control' ... and what else ?

#TODO

See Notes/HomeAutomation.

Others

• Social network stuff I don't understand ... see Notes/KnowledgeSharing

PyWacket SubPages

Subpages related to more technical subjects.

• PyWacket/Capabilities
• PyWacket/Components
• PyWacket/NeedToExplore
• PyWacket/ToDo

Obligatory Cat Poem

Charles Baudelaire, The Flowers of Evil (1857), “The Cat”

C'est l'esprit familier du lieu;
Il juge, il préside, il inspire Toutes choses dans son empire;
peut-être est-il fée, est-il dieu?
Quand mes yeux, vers ce chat que j'aime
Tirés comme par un aimant ...

It is the familiar spirit of the place;
It judges, presides, inspires Everything in its empire;
It is perhaps a fairy or a god ?
When my eyes, drawn like a magnet To this cat that I love ...