wiki:Notes/PythonTools

Context Navigation

Python Tools
1. All Platforms
2. Other Tools
  1. Bazaar
  2. Sphinx
3. Linux Only
  1. fail2ban
  2. iotop
  3. Other
4. Also See

Python Tools

Mostly console apps in Python, that is singular Python tools, also see PythonToolSets.

https://docs.python.org/2/howto/curses.html

All Platforms

IPython

Almost a standard in its own right.

https://en.wikipedia.org/wiki/IPython

IPython is a command shell for interactive computing in multiple programming languages, originally developed for the Python programming language, that offers introspection, rich media, shell syntax, tab completion, and history. IPython provides the following features:

Interactive shells (terminal and Qt-based).

A browser-based notebook with support for code, text, mathematical expressions, inline plots and other media.

Support for interactive data visualization and use of GUI toolkits.

Flexible, embeddable interpreters to load into one's own projects.

Tools for parallel computing.

The notebook capability runs on PythonTornado.

https://github.com/ipython/ipython

... The Notebook, Qt console and a number of other pieces are now parts of Jupyter. See the Jupyter installation docs if you want to use these ...

http://jupyter.org/

The Jupyter Notebook is a web application that allows you to create and share documents that contain live code, equations, visualizations and explanatory text. Uses include: data cleaning and transformation, numerical simulation, statistical modeling, machine learning and much more.

https://pypi.python.org/pypi/jupyter

http://jupyter.readthedocs.org/en/latest/index.html

The fine print ... mostly the same a IPython itself.

https://github.com/jupyter/notebook/blob/master/setup.py

install_requires = [

'jinja2', 'tornado>=4', 'ipython_genutils', 'traitlets', 'jupyter_core', 'jupyter_client', 'nbformat', 'nbconvert', 'ipykernel', # bless IPython kernel for now

]

In any case, IPython uses a lot of subprocess module stuff [ "tools for parallel computing" ] so Windows is not quite as clean a Linux ... as usual.

Installation dependencies, most are generally useful and are required by many other python packages.

https://ipython.org/ipython-doc/2/install/install.html

jinja2, needed for the notebook

sphinx, needed for nbconvert

pyzmq, needed for IPython’s parallel computing features, qt console and notebook

pygments, used by nbconvert and the Qt console for syntax highlighting

tornado, needed by the web-based notebook

nose, used by the test suite

readline (on OS X) or pyreadline (on Windows), needed for the terminal

PyReadline

https://ipython.org/ipython-doc/2/install/install.html#windows

... IPython by default runs in a terminal window, but the normal terminal application supplied by Microsoft Windows is very primitive. You may want to download the excellent and free Console application instead, which is a far superior tool.

https://pypi.python.org/pypi/pyreadline

PyReadline is part of IPython. ~~Windows only~~ ... In the not too distant past, I had trouble running PyReadline on Windows ( the console readline function has been a consistent irritation over the years ), but the developers say it runs on Windows systems. Haven't tried it yet.

http://pythonhosted.org/pyreadline/introduction.html#a-python-implementation-of-gnu-readline

Pyreadline is a package inspired by GNU readline which aims to improve the command line editing experience. In most UNIX based pythons GNU readline is available and used by python but on windows this is not the case.

Pexpect

Pexpect is a powerful command-level integration tool, but was missing in this category for a long time. It couldn't run on Windows without a Cygwin dependency. So ... it works on Linux-like systems only, too bad from an integration perspective.

However, as of Pexpect version 4.0:

Pexpect can be used on Windows and POSIX systems. However, pexpect.spawn and pexpect.run() are only available on POSIX, where the pty module is present in the standard library.

http://pexpect.readthedocs.org/en/stable/install.html

Now the bad news ...

Pexpect can be used on Windows to wait for a pattern to be produced by a child process, using pexpect.popen_spawn.PopenSpawn?, or a file descriptor, using pexpect.fdpexpect.fdspawn. This should be considered experimental for now.

pexpect.spawn and pexpect.run() are not available on Windows, as they rely on Unix pseudoterminals (ptys). Cross platform code must not use these.

Not ideal, in fact seems to be barely usable, but the hope of 'quick and easy' workarounds looms in the near future.

PyPi - https://pypi.python.org/pypi/pexpect/

Docs - http://pexpect.readthedocs.org/en/stable/

Repos - https://github.com/pexpect/pexpect

... Pexpect is a pure Python module for spawning child applications; controlling them; and responding to expected patterns in their output. Pexpect works like Don Libes' Expect. Pexpect allows your script to spawn a child application and control it as if a human were typing commands ...

Helpful - http://helpful.knobs-dials.com/index.php/Python_usage_notes/pty_and_pexpect_notes

Good example of using for network admin - http://blog.pythonicneteng.com/2012/11/pexpect-module.html

Current (2016/4/10) dev version says:

https://github.com/pexpect/pexpect/blob/master/pexpect/screen.py

warnings.warn(("pexpect.screen and pexpect.ANSI are deprecated. "

"We recommend using pyte to emulate a terminal screen: " "https://pypi.python.org/pypi/pyte"), stacklevel=2)

More compatible with Windows ?

pyte

https://pypi.python.org/pypi/pyte

http://pyte.readthedocs.org/en/latest/

https://github.com/selectel/pyte

https://github.com/selectel/pyte/blob/master/setup.py

setup(name="pyte", ...

install_requires=wcwidth?,
setup_requires=pytest-runner?,
tests_require=["pytest"

See https://pypi.python.org/pypi/wcwidth and https://github.com/jquast/wcwidth

This Library is mainly for those implementing a Terminal Emulator, or programs that carefully produce output to be interpreted by one.

psutil

https://github.com/giampaolo/psutil

"psutil (python system and process utilities) is a cross-platform library for retrieving information on running processes and system utilization (CPU, memory, disks, network) in Python."

useful mainly for system monitoring, profiling and limiting process resources and management of running processes.

implements many functionalities offered by command line tools such as:
- ps, top, lsof,
- netstat, ifconfig, pmap.
- who, df, kill, free,
- nice, ionice, iostat, iotop,
- uptime, pidof, tty, taskset,

currently supports Linux, Windows, OSX, FreeBSD and Sun Solaris, both 32-bit and 64-bit architectures, with Python versions from 2.4 to 3.4. PyPy is also known to work.

https://pypi.python.org/pypi/psutil

https://github.com/giampaolo/psutil

https://pythonhosted.org/psutil/

Extra! -> http://grodola.blogspot.com/search/label/psutil

Sunday, November 6, 2016

OK, this is a big one. Starting from psutil 5.0.0 you can query multiple Process information around twice as fast than with previous versions ...

Circus

Windows ??? Uses psutil, also requires pyzmq and tornado.

https://pypi.python.org/pypi/circus

Circus is a program that runs and watches processes and sockets. Circus can be used as a library or through the command line.

http://circus.readthedocs.org/en/latest/

... Circus provides a command-line script call circusd that can be used to manage processes organized in one or more watchers ...

http://circus.readthedocs.org/en/latest/tutorial/step-by-step/#examples

Circus is tested on Mac OS X and Linux with the latest Python 2.6, 2.7, 3.2 and 3.3. To run a full Circus, you will also need libzmq, libevent & virtualenv.

Not Windows apparently, not sure why.

Glances

https://pypi.python.org/pypi/Glances

https://github.com/nicolargo/glances

Glances is a cross-platform curses-based system monitoring tool written in Python. It uses the psutil library to get information from your system.

Also integrated with PythonBottle ...

CMD2

Useful Python console extension, if a bit limited. It's small, 64K.

https://pypi.python.org/pypi/cmd2

Enhancements for standard library’s cmd module.

Drop-in replacement adds several features for command-prompt tools

Some IPython like features

CMD3

Interesting approach, but only works with PythonGridComputing#CloudMesh

https://pypi.python.org/pypi/cmd3

cmd3 - A dynamic CMD shell with plugins ...

cmd is an extension for the CMD class with the following features:

commands are loaded from plugin directories
usage of docopts as part of the command creation
variable substitution
execution of python commands
scripts loadable from a script directory

http://cloudmesh.github.io/cmd3/

https://github.com/cloudmesh/cmd3

File / Directory Modules

Python OS File Mehtods

http://www.tutorialspoint.com/python/os_file_methods.htm

File / Directory Watchers

WatchDog

https://pypi.python.org/pypi/watchdog

Python API library and shell utilities to monitor file system events.

Directory monitoring made easy with: a cross-platform API and a shell tool to run commands in response to directory changes.

https://github.com/gorakhargosh/watchdog

Depends on: PyYAML, argh, pathtools

http://pythonhosted.org//watchdog/

 The following example program will monitor the current directory recursively
 for filesystem changes and simply log them to the console:

import sys
import time
import logging

from watchdog.observers import Observer
from watchdog.events import LoggingEventHandler

if __name__ == "__main__":

    logging.basicConfig(level=logging.INFO,
                        format='%(asctime)s - %(message)s',
                        datefmt='%Y-%m-%d %H:%M:%S')

    path = sys.argv[1] if len(sys.argv) > 1 else '.'

    event_handler = LoggingEventHandler()
    observer = Observer()
    observer.schedule(event_handler, path, recursive=True)
    observer.start()

    try:
        while True:
            time.sleep(1)
    except KeyboardInterrupt:
        observer.stop()

    observer.join()

http://stackoverflow.com/questions/18599339/python-watchdog-monitoring-file-for-changes

Instead of LoggingEventHandler? define your handler:

#!/usr/bin/python

import time

from watchdog.observers import Observer
from watchdog.events import FileSystemEventHandler

class MyHandler(FileSystemEventHandler):

    def on_modified(self, event):
        print "Got it!"

if __name__ == "__main__":

    event_handler = MyHandler()
    observer = Observer()
    observer.schedule(event_handler, path='.', recursive=False)
    observer.start()

    try:
        while True:
            time.sleep(1)
    except KeyboardInterrupt:
        observer.stop()

    observer.join()

on_modified is called when a file or directory is modified.

http://sysmagazine.com/posts/140649/

from watchdog.observers import Observer
from watchdog.events import FileSystemEventHandler

class Handler(FileSystemEventHandler):
    def on_created(self, event):
        print event

    def on_deleted(self, event):
        print event

    def on_moved(self, event):
        print event


if __name__ == "__main__":

observer = Observer()
observer.schedule(Handler(), path='/path/to/smth', recursive=True)
observer.start()

try:
    while True:
        time.sleep(0.1)
except KeyboardInterrupt:
    observer.stop()

observer.join()

On an output of a script we had:

  <DirCreatedEvent: src_path=/path/to/smth/foo>
  <FileCreatedEvent: src_path=/path/to/smth/bar>
  <FileMovedEvent: src_path=/path/to/smth/bar, dest_path=/path/to/smth/baz>
  <DirCreatedEvent: src_path=/path/to/smth/foo/foz>
  <FileMovedEvent: src_path=/path/to/smth/baz, dest_path=/path/to/smth/foo/quz>
  <FileCreatedEvent: src_path=/path/to/smth/hw>
  <FileDeletedEvent: src_path=/path/to/smth/foo/quz>
  ...

http://brunorocha.org/python/watching-a-directory-for-file-changes-with-python.html

Interesting:

http://ginstrom.com/scribbles/2012/05/10/continuous-integration-in-python-using-watchdog/

I use watchdog to run continuous integration on my projects: every time a file changes (e.g. is saved, deleted, or created) in the directory I am watching, a script will automatically run unit tests, compile libraries, build docs, and run other tests, as necessary.

http://blog.philippklaus.de/2011/08/watching-directories-for-changes-using-python_-_an-overview/

Good cross-platform info.

Other File Watchers

https://github.com/mkaz/fswatch

Python script that monitors a Mac OS X folder and on change syncs with remote site

https://code.google.com/p/pyfilesystem/

Dependencies: boto, paramiko, six, django, dexml, wx

https://github.com/gregghz/Watcher - enough ?

http://codeprogress.com/python/libraries/pyqt/showPyQTExample.php?index=419&key=QFileSystemWatcherDirChange

PyQT,QFileSystemWatcher: Detect Changes in a Directory QFileSystemWatcher

Summary : This example shows how to detect directory changes with QFileSystemWatcher in PyQT. This code simply watches "/home/developer/Desktop" directory and shows in QLabel if any change has been made in directory(If any of its contents is changed or removed).

https://github.com/mdipierro/workflow

workflow.py is a minimalist [ sub-minimalist? ] file based workflow engine. It runs as a background process and can automate certain tasks such as deleting old files, emailing you when new files are created or run a script to process new files.

SSH Tools

Fabric

http://www.fabfile.org/

Fabric is a Python (2.5-2.7) library and command-line tool for streamlining the use of SSH for application deployment or systems administration tasks.

Fabric Documentation	http://docs.fabfile.org/en/1.10/
How to use Fabric	http://www.pythonforbeginners.com/systems-programming/how-to-use-fabric-in-python/

From http://docs.fabfile.org/en/1.10/tutorial.html

"Fabric is a Python (2.5-2.7) library and command-line tool for streamlining the use of SSH for application deployment or systems administration tasks.

More specifically, Fabric is:

A tool that lets you execute arbitrary Python functions via the command line; A library of subroutines (built on top of a lower-level library) to make executing shell commands over SSH easy and Pythonic.

Naturally, most users combine these two things, using Fabric to write and execute Python functions, or tasks, to automate interactions with remote servers."

From http://www.fabfile.org/:

It provides a basic suite of operations for executing local or remote shell commands (normally or via sudo) and uploading/downloading files, as well as auxiliary functionality such as prompting the running user for input, or aborting execution.

Typical use involves creating a Python module containing one or more functions, then executing them via the fab command-line tool.

Below is a small but complete “fabfile” containing a single task:

from fabric.api import run

def host_type():
    run('uname -s')

Once a task is defined, it may be run on one or more servers, like so:

$ fab -H localhost,linuxbox host_type
[localhost] run: uname -s
[localhost] out: Darwin
[linuxbox] run: uname -s
[linuxbox] out: Linux

Done.
Disconnecting from localhost... done.
Disconnecting from linuxbox... done.

In addition to use via the fab tool, Fabric’s components may be imported into other Python code, providing a Pythonic interface to the SSH protocol suite at a higher level than that provided by e.g. the Paramiko library (which Fabric itself uses.)

Installing Fabric On Windows

From http://www.fabfile.org/installing.html:

Windows users who already have ActiveState’s ActivePython distribution installed may find Fabric is best installed with its package manager, PyPM.

Below is example output from an installation of Fabric via pypm:

C:\> pypm install fabric
The following packages will be installed into "%APPDATA%\Python" (2.7):
 paramiko-1.7.8 pycrypto-2.4 fabric-1.3.0
Get: [pypm-free.activestate.com] fabric 1.3.0
Get: [pypm-free.activestate.com] paramiko 1.7.8
Get: [pypm-free.activestate.com] pycrypto 2.4
Installing paramiko-1.7.8
Installing pycrypto-2.4
Installing fabric-1.3.0
Fixing script %APPDATA%\Python\Scripts\fab-script.py
C:\>

http://stackoverflow.com/questions/1917193/installing-fabric-on-windows-error-no-module-called-readline?rq=1

I have no problem installing Fabric on Windows with ActivePython? (w/ PyPM):

C:\> pypm install fabric
Ready to perform these actions:
The following packages will be installed:
 fabric-0.9.0 pycrypto-2.0.1
Get: [pypm.activestate.com] fabric 0.9.0-1
Get: [pypm.activestate.com] pycrypto 2.0.1-1
Installing fabric-0.9.0
Fixing script C:\Users\sridharr\AppData\Roaming\Python\Scripts\fab-script.py
Installing pycrypto-2.0.1

Getting Python Fabric setup in Windows - May 30th, 2013

http://thepracticalsysadmin.com/getting-python-fabric-setup-in-windows/

It seems like the Python integration in Windows is very different than it is in the Linux world where everything is all bundled up nice and neatly.

There are several separate, seemingly unrelated pieces that all need to fit together to get Python and Fabric working correctly in a Windows environment ...

I recommend going with Python 2.7, rather than 3.3 because it has much better backwards compatibility ...

http://stackoverflow.com/questions/4777503/how-does-one-get-the-output-of-fabric-execution-fab-exe-on-windows-7

The solution to getting fabric running on Windows 7:

easy_install fabric

manually install pycrypto-2.0.1.win-amd64-py2.6.exe from: http://www.voidspace.org.uk/python/modules.shtml

manually install pywin32-214.win-amd64-py2.6.exe from: http://sourceforge.net/projects/pywin32/files/

Pip still doesn’t work quite right when attempting to install various Python packages. To get that part working you will need MinGW32 installed ( http://www.mingw.org/wiki/InstallationHOWTOforMinGW ) ...

Other Tools

Bazaar

Bazaar is a relatively quick and easy distributed revision system written in Python ( if there is such a thing as a 'quick and easy distributed' anything ) from Canonical, as in Ubuntu Linux. Runs well on Windows too.

A mature, active and well supported project. See RevisionControl#Bazaar.

The source code as of Jan 2017 is at http://bazaar.launchpad.net/~bzr-pqm/bzr/bzr.dev/files. Very clean and Pythonic, good use of comprehensions, etc.

Has both a decent GUI and lots of little line command scripts, good example of a Python line command environment.

http://bazaar.launchpad.net/~bzr-pqm/bzr/bzr.dev/files/head:/bzrlib/.

Sphinx

https://pypi.python.org/pypi/Sphinx

Sphinx is a tool that makes it easy to create intelligent and beautiful documentation for Python projects (or other documents consisting of multiple reStructuredText sources) ...

...

Among its features are the following:

Output formats: HTML (including derivative formats such as HTML Help, Epub and Qt Help), plain text, manual pages and LaTeX or direct PDF output using rst2pdf

Extensive cross-references: semantic markup and automatic links for functions, classes, glossary terms and similar pieces of information

Hierarchical structure: easy definition of a document tree, with automatic links to siblings, parents and children

Automatic indices: general index as well as a module index

Code handling: automatic highlighting using the Pygments highlighter

Flexible HTML output using the Jinja 2 templating engine

Various extensions are available, e.g. for automatic testing of snippets and inclusion of appropriately formatted docstrings

Setuptools integration

http://sphinx-doc.org/

Well supported. Nearly a standard in its own right.

Linux Only

fail2ban

http://www.fail2ban.org/wiki/index.php/Main_Page

http://www.fail2ban.org/wiki/index.php/Commands

iotop

http://guichaz.free.fr/iotop/

Other

paramiko

Widely used.

https://pypi.python.org/pypi/paramiko/

SSH2 protocol library

This is a library for making SSH2 connections (client or server). Emphasis is on using SSH2 as an alternative to SSL for making secure connections between python scripts. All major ciphers and hash methods are supported. SFTP client and server mode are both supported too. Required packages: pyCrypto

Also See

https://launchpad.net/rapache

Rapache is a simple tool for remotely managing and configuring an apache2 instance without having to hassle around through configuration files.

Rapache is a python + GTK tool that allows you to configure apache2 and all of its modules ...

Vhost managing
Virtualhost normalization to conform Debian policy
Module managing
Syntax highliting when showing confs source
Config files backup and quick restore Plugin system
SSL Plugin, Basic_Auth plugin, Mono plugin, Logging Plugin
Apache start/stop and status

https://wiki.ubuntu.com/Rapache

Version 0.8 ???

Last modified 9 months ago Last modified on 03/07/2017 11:57:28 AM

Download in other formats:

Plain Text