python | rael.io

Jupyter Image Extractor

Tue, 06 Mar 2018 11:23:56 +0200

Parses a Jupyter book html and extracts the images.

Build

Build the container:

ellie:ws rael$ make docker-build

Expected output:

docker build . -t 'jupyter-image-extractor'
Sending build context to Docker daemon 12.46MB
Step 1/1 : FROM python:3-onbuild
# Executing 3 build triggers
---> Running in be4957e8a2d2
Collecting beautifulsoup4 (from -r requirements.txt (line 1))
Downloading https://files.pythonhosted.org/packages/9e/d4/10f46e5cfac773e22707237bfcd51bbffeaf0a576b0a847ec7ab15bd7ace/beautifulsoup4-4.6.0-py3-none-any.whl (86kB)
Installing collected packages: beautifulsoup4
Successfully installed beautifulsoup4-4.6.0
Removing intermediate container be4957e8a2d2
---> 2b1181b0bde2
Successfully built 2b1181b0bde2
Successfully tagged jupyter-image-extractor:latest

Execute

Run the script with docker:

ellie:ws rael$ make docker-run

Expected output:

docker run -it --rm --name 'jupyter-image-extractor' -v "/Users/rael/Documents/Code/python/ws:/code" -w /code 'jupyter-image-extractor' python jupyterImageSaver.py Jupyter\ Notebook\ Viewer.html
./Jupyter Notebook Viewer_files/nav_logo.svg
./images/base64-img-1.png
./images/base64-img-2.png
./images/base64-img-3.png
./images/base64-img-4.png
./images/base64-img-5.png
./images/base64-img-6.png

PyBot

Thu, 02 Mar 2017 11:23:56 +0200

pybot

Wrapper for python-telegram-bot to allow dynamic plug-in architecture, an attempt to make python-telegram-bot more hubottish.

Execution flow

bin/pybot starts the bot with the selected adapter and using the configuration options defined in conf/pybot.conf or the command-line arguments.
The selected adapter receives the message and sends it to the brain.py.
The brain.py loads in runtime all the .py files in the ./memory folder each time and try to execute a defined method, for example hear(message.text).
Each module in ./memory with that method returns a response, the brain.py sends it to the pybot.py and it sends the response back to the chat.

As the methods are dynamically loaded, you can edit and add the files in ./memory without need to restart pybot.py, and they will be reloaded on the next message. Overkill but funny.

Makefile targets

usage: Show this help
setup-venv: Setup virtualenv
lint: Run code linter to check code style
telegram: Run pybot with the telegram adapter
docker-build: Build the docker image for running pybot
docker-telegram: Run with telegram adapter in the docker container
docker-lint: Run pep8 in the docker container
docker-clean: Remove the docker image

Execution example

make docker-telegram

docker run -it --rm --name pybot -v /Users/rael/Code/python/pybot:/usr/src/pybot -w /usr/src/pybot 'pybot' bin/pybot telegram
Starting pybot using conf token file. CTRL-C to quit.
2017-04-17 07:35:54,392 - pybot.interfaces.telegram - INFO - Bot raelbot up and ready!
2017-04-17 07:36:39,371 - pybot.brain - INFO - 116133952, Hello world!, 53693428, 2017-04-17 07:36:39,"53693428";

Contributing

Contributions of all sizes are welcome.

Attributions

Header image from python-telegram-bot.org

License

You may copy, distribute and modify the software provided that modifications are described and licensed for free under GNU General Public License v3.0. Derivatives works (including modifications or anything statically linked to the library) can only be redistributed under LGPL-3, but applications that use the library don’t have to be.

GTA Crew Exporter

Wed, 06 Nov 2013 11:23:56 +0200

Overview

Get information about crew members on Rockstar GTAV social club.

Prerequisites

python | http://www.python.org/download/
selenium for python | https://pypi.python.org/pypi/selenium
firefox | http://www.mozilla.org/en-US/firefox/new/

** Firefox must be in english due to Rockstar Social Club source code, in the final version it will not be necessary. **

Installation

Linux

[327] root@eva:~# aptitude install python-pip
The following NEW packages will be installed:
python-pip python-pkg-resources{a} python-setuptools{a}
0 packages upgraded, 3 newly installed, 0 to remove and 18 not upgraded.
Need to get 648 kB of archives. After unpacking 2,938 kB will be used.
Do you want to continue? [Y/n/?] y
Get: 1 http://ftp.es.debian.org/debian/ jessie/main python-pkg-resources all 0.6.49-2 [62.5 kB]
Get: 2 http://ftp.es.debian.org/debian/ jessie/main python-setuptools all 0.6.49-2 [320 kB
Get: 3 http://ftp.es.debian.org/debian/ jessie/main python-pip all 1.4.1-2 [266 kB
Fetched 648 kB in 2s (223 kB/s)
Selecting previously unselected package python-pkg-resources.
(Reading database ... 156790 files and directories currently installed.)
Unpacking python-pkg-resources (from .../python-pkg-resources_0.6.49-2_all.deb) ...
Selecting previously unselected package python-setuptools.
Unpacking python-setuptools (from .../python-setuptools_0.6.49-2_all.deb) ...
Selecting previously unselected package python-pip.
Unpacking python-pip (from .../python-pip_1.4.1-2_all.deb) ...
Processing triggers for man-db ...
Setting up python-pkg-resources (0.6.49-2) ...
Setting up python-setuptools (0.6.49-2) ...
Setting up python-pip (1.4.1-2) ...
[328] root@eva:~# easy_install selenium
Searching for selenium
Reading http://pypi.python.org/simple/selenium/
Best match: selenium 2.37.2
Downloading https://pypi.python.org/packages/source/s/selenium/selenium-2.37.2.tar.gz#md5=f3fffaae0bc789676c4e2ab285f7b04f
Processing selenium-2.37.2.tar.gz
Writing /tmp/easy_install-shFcgY/selenium-2.37.2/setup.cfg
Running selenium-2.37.2/setup.py -q bdist_egg --dist-dir /tmp/easy_install-shFcgY/selenium-2.37.2/egg-dist-tmp-_CdLcZ
Adding selenium 2.37.2 to easy-install.pth file
Installed /usr/local/lib/python2.7/dist-packages/selenium-2.37.2-py2.7.egg
Processing dependencies for selenium
Finished processing dependencies for selenium

Mac

Download and install python from python.org

Then install selenium using pip:

[502] rael@mbw:~$ sudo pip install selenium
Downloading/unpacking selenium
Downloading selenium-2.37.2.tar.gz (2.6MB): 2.6MB downloaded
Running setup.py egg_info for package selenium
Installing collected packages: selenium
Running setup.py install for selenium
Successfully installed selenium
Cleaning up...

Windows

Not tested yet, but I guess similar as in Mac.

Usage

Command interface

./gtav_crew_exporter.py --help
gtav_crew_exporter.py -c <crew_name> [-u <username> -p <password>] [-o <output_file>] [-v]

Flag	Description
–help	View the basic usage
-c, –crew	The name of the crew to export
-u, –username=	The username of your Rockstar social club
-p, –password=	The password of your Rockstar social club
-v	Verbose mode, for debug.

Examples

Only members and ranks

gtav_crew_exporter.py -c elotrolado

$ ./gtav_crew_exporter.py -c elotrolado
Crew: elotrolado
Crew Size: 300 members
!! Without login and password, only username and rank are available:
Crew Members :
leader, nihael, http://socialclub.rockstargames.com/member/nihael
commissioner, FJTR23, http://socialclub.rockstargames.com/member/fjtr23
lieutenant, Racso25, http://socialclub.rockstargames.com/member/racso25
representative, AntonLaveyX, http://socialclub.rockstargames.com/member/antonlaveyx
representative, beerna, http://socialclub.rockstargames.com/member/beerna
representative, lierhoff, http://socialclub.rockstargames.com/member/lierhoff
representative, raelga, http://socialclub.rockstargames.com/member/raelga
representative, rubi7410, http://socialclub.rockstargames.com/member/rubi7410
representative, Sd-Snatcher, http://socialclub.rockstargames.com/member/sd-snatcher
muscle, _XiloX_, http://socialclub.rockstargames.com/member/_xilox_
muscle, 4LUFLiNT, http://socialclub.rockstargames.com/member/4luflint
muscle, A_Rebato, http://socialclub.rockstargames.com/member/a_rebato

gtav_crew_exporter.py -c the_pollasos -u raelga -p ***

$ ./gtav_crew_exporter.py -c the_pollasos -u raelga -p ***
Crew: the_pollasos
Crew Size: 9 members
adrianrgvez, Spain, , PS3, the_pollasos, leader, 108, 5D 4H 17M, All ok.
corderius10, Spain, , PS3, the_pollasos, commissioner, 79, 78H 42M 24S, All ok.
oscartc14, Spain, , PS3, the_pollasos, lieutenant, 11, 8H 34M 36S, All ok.
alfi_de_pumarin9, Spain, , PS3, the_pollasos, representative, 24, 17H 23M 9S, All ok.
capableuncle3020, , , , , representative, , , Private profile.
rodri799, Spain, , PS3, the_pollasos, representative, 12, 14H 47M 50S, All ok.
yex_gr, Spain, yex_, PS3, the_pollasos, representative, 21, 24H 50M 26S, All ok.
OrignalBEAST, United Kingdom, ORIGNALBEASSTT, PS3, lossantosdiamondz, muscle, 24, 48H 40M 23S, All ok.
raelga, Spain, raelga, PS3, elotrolado, muscle, 65, 6D 13H 53M, All ok.

Verbose mode

gtav_crew_exporter.py -c the_pollasos -u raelga -p *** -v

$ ./gtav_crew_exporter.py -v -c the_pollasos -u raelga -p ***
Crew: the_pollasos
DBG : web - starting browser
DBG : web - page fully loaded!
DBG : web - set page in english.
DBG : web - all users visible.
Crew Size: 9 members
DBG : web - page fully loaded!
DBG : web - page fully loaded!
DBG : [adrianrgvez]
DBG : web - page fully loaded!
DBG : [adrianrgvez] main crew: the_pollasos
DBG : [adrianrgvez] PSN ID:
DBG : [adrianrgvez] country: Spain
DBG : web - page fully loaded!
DBG : [adrianrgvez] rank: leader
DBG : [adrianrgvez] playtime: 5D 4H 20M
DBG : [corderius10]
DBG : web - page fully loaded!
DBG : [corderius10] main crew: the_pollasos
DBG : [corderius10] PSN ID:
DBG : [corderius10] country: Spain
DBG : web - page fully loaded!
DBG : [corderius10] rank: commissioner
DBG : [corderius10] playtime: 78H 42M 24S
DBG : [oscartc14]
DBG : web - page fully loaded!
DBG : [oscartc14] main crew: the_pollasos
DBG : [oscartc14] PSN ID:
DBG : [oscartc14] country: Spain
DBG : web - page fully loaded!
DBG : [oscartc14] rank: lieutenant
DBG : [oscartc14] playtime: 8H 34M 36S
DBG : [alfi_de_pumarin9]
DBG : web - page fully loaded!
DBG : [alfi_de_pumarin9] main crew: the_pollasos
DBG : [alfi_de_pumarin9] PSN ID:
DBG : [alfi_de_pumarin9] country: Spain
DBG : web - page fully loaded!
DBG : [alfi_de_pumarin9] rank: representative
DBG : [alfi_de_pumarin9] playtime: 17H 23M 9S
DBG : [capableuncle3020]
DBG : web - page fully loaded!
DBG : [capableuncle3020] Profile is private!
DBG : [rodri799]
DBG : web - page fully loaded!
DBG : [rodri799] main crew: the_pollasos
DBG : [rodri799] PSN ID:
DBG : [rodri799] country: Spain
DBG : web - page fully loaded!
DBG : [rodri799] rank: representative
DBG : [rodri799] playtime: 14H 47M 50S
DBG : [yex_gr]
DBG : web - page fully loaded!
DBG : [yex_gr] main crew: the_pollasos
DBG : [yex_gr] PSN ID: ***
DBG : [yex_gr] country: Spain
DBG : web - page fully loaded!
DBG : [yex_gr] rank: representative
DBG : [yex_gr] playtime: 24H 50M 26S
DBG : [OrignalBEAST]
DBG : web - page fully loaded!
DBG : [OrignalBEAST] main crew: lossantosdiamondz
DBG : [OrignalBEAST] PSN ID: ***
DBG : [OrignalBEAST] country: United Kingdom
DBG : web - page fully loaded!
DBG : [OrignalBEAST] rank: muscle
DBG : [OrignalBEAST] playtime: 48H 40M 23S
DBG : [raelga]
DBG : web - page fully loaded!
DBG : [raelga] main crew: elotrolado
DBG : [raelga] PSN ID: raelga
DBG : [raelga] country: Spain
DBG : web - page fully loaded!
DBG : [raelga] rank: muscle
DBG : [raelga] playtime: 6D 13H 53M
adrianrgvez, Spain, , PS3, the_pollasos, leader, 108, 5D 4H 20M, All ok.
corderius10, Spain, , PS3, the_pollasos, commissioner, 79, 78H 42M 24S, All ok.
oscartc14, Spain, , PS3, the_pollasos, lieutenant, 11, 8H 34M 36S, All ok.
alfi_de_pumarin9, Spain, , PS3, the_pollasos, representative, 24, 17H 23M 9S, All ok.
capableuncle3020, , , , , representative, , , Private profile.
rodri799, Spain, , PS3, the_pollasos, representative, 12, 14H 47M 50S, All ok.
yex_gr, Spain, ****, PS3, the_pollasos, representative, 21, 24H 50M 26S, All ok.
OrignalBEAST, United Kingdom, ****, PS3, lossantosdiamondz, muscle, 24, 48H 40M 23S, All ok.
raelga, Spain, raelga, PS3, elotrolado, muscle, 65, 6D 13H 53M, All ok.