If you can see it, you can automate it: Sikuli

Who's driving who?

Project Sikuli

Computer vision + scripting

Let's see it!

About Project Sikuli

• MIT project
• Free as in freedom (MIT)
• ... and beer
• Jython on Java
• Cross-platform

Alternatives

• APIs
• Web tools
• User simulators

API (Application Programming Interface)

• App-specific
• Language-specific
• Badly documented
• Freaking hard!

Python-UNO bridge

import socket  # only needed on win32-OOo3.0.0
import uno
localContext = uno.getComponentContext()
resolver = localContext.ServiceManager.createInstanceWithContext(
   "com.sun.star.bridge.UnoUrlResolver", localContext )
ctx = resolver.resolve( "uno:socket,host=localhost,port=2002;urp;StarOffice.ComponentContext" )
smgr = ctx.ServiceManager
desktop = smgr.createInstanceWithContext( "com.sun.star.frame.Desktop",ctx)
model = desktop.getCurrentComponent()
text = model.Text
cursor = text.createTextCursor()
text.insertString( cursor, "Hello World", 0 )
ctx.ServiceManager

Web tools

• Selenium, Windmill
• GreaseMonkey
• Mechanize
• Web APIs

User simulators

• PyWinAuto
• autohotkey
• dogtail
• Sikuli

Installing

java -version

sudo aptitude install openjdk-6-jre wmctrl libcv4
libcvaux4 libhighgui4

~/Sikuli-IDE/sikuli-ide.sh

Windows: sikuli.bat instead of .exe

Lifecycle

1. Create with IDE
2. Save myscript.sikuli/
3. Load myscript.sikuli/
4. Export myscript.skl
5. sikuli-ide.sh myscript.skl

Similarity

switchApp('Klondike')
cards = findAll()
for card in cards:
   doubleClick(card)

Typing

import time

switchApp('Klondike')
type('g', KEY_ALT)
type('n')
time.sleep(3)

Waiting for Guido

import re

switchApp('Firefox')
type('t', KEY_CTRL)
type('l', KEY_CTRL)
type('http://twitter.com/gvanrossum\n')
guido = wait(, 60)

Paste buffer hack for text

dragDrop(guido.getCenter().below(50),
   guido.getCenter().below(250))
type('c', KEY_CTRL)
guidosays = Env.getClipboard().strip()
popup(guidosays)
guidosays = re.search('1\.(.*?)\d{1,2}:\d\d', guidosays,
   re.DOTALL).group(1)
guidosays = guidosays.strip()

User interaction

popup(txt)

answer = input(question)

selection = capture(question)

User interaction example

foods = VDict()
foods[] = 'apple'
foods[] = 'cobb salad'
foods[] = 'salmon'
foods[] = 'pork chop'
foods[] = 'swiss cheese'
foods[] = 'biscuit'


switchApp(r'foods.jpg')
dish = None
while not dish:
   target = capture("Your order, please?")
   dish = foods[target]
dish = dish[0]
fries = input('Would you like fries with that (y/n)?')
if fries.lower().strip()[0] == 'y':
   dish += ' with fries'
popup(dish + ' coming right up!')

JUnit testing

View >> Unit Test

methods of implied class

Run button, Test Trace tab

Python modules

import mylocalmodule

sudo easy_install-2.5 pyparsing

import sys
sys.path.append('/usr/local/lib/python2.5/' +
                'site-packages/pyparsing-1.5.5-py2.5.egg/')

Java libraries

from java.lang import Math
popup(str(Math.pow(2,8)))

Warts

Source: Center for Biological Informatics of the U.S. Geological Survey.

Debugging

• popup()
• print()
• python logging
• pdb (-Dsikuli.console=false)

IDE

Image capture and display is revolutionary, BUT

• Poor space/tab indentation
• Error messages: nongraphical, hidden
• Weak code colorization
• No Undo
• No Search

Graphical is fragile

Scale, themes, colors

Window positions

Mouse pointer

Portability

Cross-platform?

• Sikuli itself: YOU BETCHA
• Sikuli scripts: GOOD LUCK WITH THAT
• Same OS, different machine: IFFY

Bulletproofing: take control!

• switchApp()
• Pull windows onscreen
• Set sizes
• Scroll

When find() fails

try:
    btn = find()
except findFailed:
    popup('Phooey!')
# or
setThrowException(False)
btn = find() or find()

O RLY?

making programming simple, versatile, and useful

Or maybe

making programming fun, versatile, and useful

Easy?

YES: Short, clear scripts

NO: Troubleshooting, debugging, bulletproofing

Help

• sikuli.org >> contact >> questions
• catherine.devlin@gmail.com
• catherinedevlin.blogspot.com

Community

PyOhio 2011, July 30-31, Columbus

PyCon 2011, March 9-17, Atlanta

Turtles