Write Python Weather APP on Heroku(8)

May 11, 2014
Technology

Deploy on Heroku

The deployed version is listed as following:

#!/usr/bin/env python
# -*- coding: utf-8 -*-
# Set default encoding: utf-8
import sys;
reload(sys);
# Setting the utf-8 format
sys.setdefaultencoding("utf8")

import os
import logging
import string
from celery import Celery
from celery.task import periodic_task
from datetime import datetime,timedelta
from os import environ


# For retrieving temperature/humidity
import pywapi
import urllib2
from urllib2 import ProxyHandler
import re
from BeautifulSoup import BeautifulSoup

# For using database(Postgresql)
# from flash.ext.sqlalchemy import SQLAlchemy
from sqlalchemy import create_engine
from sqlalchemy import MetaData, Column, Table, ForeignKey
from sqlalchemy import Integer, String, DateTime


# redis configuration for celery
### Local, should be change to remote when deploying to heroku
# REDIS_URL = environ.get('REDISTOGO_URL', 'redis://localhost')
### Heroku Ways
REDIS_URL = environ.get('REDISCLOUD_URL')


celery = Celery('tasks', broker=REDIS_URL)

# Fetch 


# Define Periodic Tasks
@periodic_task(run_every=timedelta(minutes=60))
# Funciton for fetching data and store it in Postgres
def fetch_and_store_data():
    # Fetching the weather information from Yahoo. 
    Yahoo_Result = pywapi.get_weather_from_yahoo('CHXX0099')
    Current_Temp = string.lower(Yahoo_Result['condition']['temp'])
    Current_Humi = string.lower(Yahoo_Result['atmosphere']['humidity'])
    Tomorrow_Forecast = Yahoo_Result['forecasts'][0]
    Twenty_Four_Hours = Yahoo_Result['forecasts'][1]
    Fourty_Eight_Hours = Yahoo_Result['forecasts'][2]
    Seventy_Two_Hours = Yahoo_Result['forecasts'][3]
    # !!! comment proxy related for deploying to heroku !!! #
    # proxy = urllib2.ProxyHandler({'http': '1xx.x.xx.xxx:2xxx'})
    # opener = urllib2.build_opener(proxy)
    # urllib2.install_opener(opener)
    # !!! comment out end #
    page = urllib2.urlopen("http://www.pm25.in/nanjing")
    soup = BeautifulSoup(page)                      #
    # Find the detailed table from the soup.        #
    table = soup.find('table', {'id':'detail-data'})#
    # Fetch the XuanWuHu, if not, use MaiGaoQiao ins#tead. 
    rows = table.findAll('tr')                      #
    for subrows in rows:                            #
        if "玄武湖" in subrows.text:                #
            XuanwuLake = subrows                    #
        else:                                       #
            if "迈皋桥" in subrows.text:            #
                XuanwuLake = subrows                #
    XuanwuLake_subitem = XuanwuLake.findAll('td')   #
    # Here we will get an array, fetch out the text #for the content from this array.
    # Fetched origin data, different from cnpm25.cn #
    pm_25_orig = XuanwuLake_subitem[4].text         #
    pm_10_orig = XuanwuLake_subitem[5].text
    # Open the Database
    # Local Engine
    # engine = create_engine('postgresql+psycopg2://Trusty:@localhost:5432/mylocaldb',echo=True)
    # Heroku Engine
    db_conn = environ['DATABASE_URL']
    engine = create_engine(db_conn)

    metadata=MetaData(bind=engine)
    
    # Definition of the weather table
    weather_table=Table('weather',metadata,
    	Column('Insert_Time', DateTime(timezone=True),primary_key=True),
    	Column('Temperature', Integer),
    	Column('Humidity', Integer),
    	Column('PmTen', Integer),
    	Column('PmTwoFive', Integer),
    )

    # Create the table in mylocaldb
    metadata.create_all(checkfirst=True)

    # Record Insertion
    # First generate an insertion sentense:
    ins = weather_table.insert()
    # Really insert into the Database
    # ins = weather_table.insert().values(Insert_Time=datetime.datetime.utcnow(), Temperature=25, Humidity=75, PmTen=100, PmTwoFive=55)   # Example
    ins = weather_table.insert().values(Insert_Time=datetime.utcnow(), Temperature=Current_Temp, Humidity=Current_Humi, PmTen=int(pm_10_orig), PmTwoFive=int(pm_25_orig))
    # Connect to engine and execute.
    conn = engine.connect()
    result = conn.execute(ins)

    #return Current_Temp
    return pm_25_orig


# Every 3 minutes we will see debug information, from heroku log
# @periodic_task(run_every=timedelta(minutes=3))
# @periodic_task(run_every=timedelta(seconds=3))
# def print_fib():
#     #logging.info(fib(30))
#     logging.info("This could be viewed in logging!")

This version use remote service, and it will handle the fetching/updating every 60 minutes.

Prevent Heroku from Sleeping

Until now, almost all of the back-end has been set up. We can fetch the data and periodly insert into the database, and our web app will continue to run(Never sleep).
Tomorrow we will try to write a beautiful front-end for our webapp.

Celery and RabbitMQ on ArchLinux

May 10, 2014
Technology

Installation

Install and run rabbitmq server via:

$ yaourt rabbitmq
$ rabbitmq-server

Install celery in python virtual enviroment:

$ workon venv2
(venv2) $ pip install celery

Run Simple Tasks

Following python file, named “tasks.py” defines two tasks:

from celery import Celery

app = Celery('tasks', backend='amqp', broker='amqp://')

@app.task(ignore_result=True)
def print_hello():
    print 'hello there'

@app.task
def gen_prime(x):
    multiples = []
    results = []
    for i in xrange(2, x+1):
        if i not in multiples:
            results.append(i)
            for j in xrange(i*i, x+1, i):
                multiples.append(j)
    return results

Run it via:

celery worker -A tasks &

Now in another terminal you can use python interractive window for:

from tasks import print_hello
from tasks import gen_prime
print_hello()
primes = gen_prime(1000)
primes = gen_prime(50000)
# CTRL+C will stop it. 
# Access the background worker
primes = gen_prime.delay(50000)
# by the worker executing now, because we configured the backend for application
primes.ready()
False
...
True
print primes.get()

Periodic Tasks

Add following lines into the tasks.py:

from celery.task import periodic_task
from datetime import timedelta


@periodic_task(run_every=timedelta(minutes=1))
def print_minutes():
    print 'Hello, 1 minute reached'

@periodic_task(run_every=timedelta(seconds=3))
def every_3_seconds():
    print("Running periodic task!")

Now start the tasks.py via following commands:

$ celery -A tasks worker --loglevel=info --beat

You will see Celery output “Running periodic tasks!” every 3 seconds, while every 1 minutes the “Hello, 1 minute reached” will be printed out.

To Be Thought

How to mirgrate it with our heroku web app ?
The periodic_task is good for fetching pages/datas and generate the result, then store the results into the database.

But the webapp should response to user’s http request, this will be the main task.

Write Python Weather APP on Heroku(5)

May 10, 2014
Technology

In fact this is a migration from sqlite3 to postgresql.

View the historical sqlite3

We will refer to our own design of database. First fetch the data file, this is a sqlite3 file, so we use sqlite3 to view its structure.

$ sqlite3  
SQLite version 3.8.4.3 2014-04-03 16:53:12
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
sqlite> .open ./weather.db
sqlite> .tables
foo
sqlite> .schema foo
CREATE TABLE foo (d_temper integer, d_humi integer, d_pm10 integer, d_pm25 real, d_time timestamp);

Although sqlite3 is supported on heroku, we’d better use heroku’s suggestion, to use postgre for storing out database.

Create Database In Postgres

####Datatime selection:
Postgres provides a very fantanstic way for handling the datatime, it supports the timezone, comparing to GAE’s database, this feature will let us get the current time based on timezone. So we did the following tests:

# CREATE TABLE my_tbl (
mylocaldb(# my_timestamp TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
mylocaldb(# CHECK(EXTRACT(TIMEZONE FROM my_timestamp) = '0')
mylocaldb(# );
CREATE TABLE
mylocaldb=# \dt my_tbl
        List of relations
 Schema |  Name  | Type  | Owner 
--------+--------+-------+-------
 public | my_tbl | table | Trusty
(1 row)

When we want to insert the datatime into table ‘my_tbl’, simply do following:

mylocaldb=# SET timezone = 'UTC';
SET
mylocaldb=# INSERT INTO my_tbl (my_timestamp) VALUES (NOW());
INSERT 0 1

And for querying out the inserted records, we do following:

mylocaldb=# select * from public.my_tbl
;
         my_timestamp          
-------------------------------
 2014-05-10 01:39:52.87532+00
 2014-05-10 01:42:44.130269+00
(2 rows)

mylocaldb=# SET timezone='Asia/Shanghai';
SET
mylocaldb=# select * from public.my_tbl
mylocaldb-# ;
         my_timestamp          
-------------------------------
 2014-05-10 09:39:52.87532+08
 2014-05-10 09:42:44.130269+08
(2 rows)

mylocaldb=# SET timezone='America/Los_Angeles';
SET
mylocaldb=# select * from public.my_tbl;
         my_timestamp          
-------------------------------
 2014-05-09 18:39:52.87532-07
 2014-05-09 18:42:44.130269-07
(2 rows)

We can see the output formats depends on our “timezone” value.

Select Other Data Formats

We listed following table for describing the Data we inserted:
Timestamp integer integer integer integer
Insert_Time Temperature Humidity PMTen PMTwoFive

Thus the sql sentense is as following:

mylocaldb=# CREATE TABLE weather (
mylocaldb(# my_timestamp TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
mylocaldb(# Temperature ^C
mylocaldb=# CREATE TABLE weather (
mylocaldb(# Insert_Time TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
mylocaldb(# Temperature integer,
mylocaldb(# Humidity integer,
mylocaldb(# PMTen integer,
mylocaldb(# PMTwoFive integer,
mylocaldb(#  CHECK(EXTRACT(TIMEZONE FROM Insert_Time) = '0'));

Check the tables:

mylocaldb=# \dt
        List of relations
 Schema |  Name   | Type  | Owner 
--------+---------+-------+-------
 public | my_tbl  | table | Trusty
 public | user    | table | Trusty
 public | weather | table | Trusty
(3 rows)

Insert one record:

mylocaldb=# INSERT INTO weather (Insert_Time, Temperature, Humidity, PMTen, PMTwoFive) VALUES(NOW(), 25, 80, 150, 75);
INSERT 0 1

Displaying the inserted record:

mylocaldb=# SET timezone='Asia/Shanghai';
SET
mylocaldb=# select * from public.weather;
          insert_time          | temperature | humidity | pmten | pmtwofive 
-------------------------------+-------------+----------+-------+-----------
 2014-05-10 10:38:27.276043+08 |          25 |       80 |   150 |        75
(1 row)

Database Operation

We need to create just once database, So this function should be Check_Or_Create().
We need to insert records, so Insert_Record() should be written.
Other Operation, modification or delete shouldn’t care at the very beginning.
We will use a new file for recording all of the function.

The code for Create and Insert record into weather table is listed as following:

from sqlalchemy import create_engine
from sqlalchemy import MetaData, Column, Table, ForeignKey
from sqlalchemy import Integer, String, DateTime
import datetime

# Create the engine for connecting the local database
# How to connect the remote engine, heroku postgres? 
# Yes, it's possible. The DATABASE_URL environment variable provided by heroku fits perfectly as argument for create_engine. Behind the scene, it's a postgresql database, which is perfectly handled by sqlalchemy.
# 
# The way to do it may vary depending on the framework you use, but there shouldn't be any difficulty.

engine = create_engine('postgresql+psycopg2://Trusty:@localhost:5432/mylocaldb',echo=True)
metadata=MetaData(bind=engine)

# Definition of the weather table
weather_table=Table('weather',metadata,
	Column('Insert_Time', DateTime(timezone=True),primary_key=True),
	Column('Temperature', Integer),
	Column('Humidity', Integer),
	Column('PmTen', Integer),
	Column('PmTwoFive', Integer),
)

# Create the table in mylocaldb
metadata.create_all(checkfirst=True)

# The actual SQL Language
#CREATE TABLE weather (
#	"Insert_Time" TIMESTAMP WITH TIME ZONE NOT NULL, 
#	"Temperature" INTEGER, 
#	"Humidity" INTEGER, 
#	"PmTen" INTEGER, 
#	"PmTwoFive" INTEGER, 
#	PRIMARY KEY ("Insert_Time")
#)

# mylocaldb=# select * from public.weather;
#  Insert_Time | Temperature | Humidity | PmTen | PmTwoFive 
# -------------+-------------+----------+-------+-----------
# (0 rows)

# Record Insertion
# First generate an insertion sentense:
ins = weather_table.insert()
#>>> str(ins)
#'INSERT INTO weather ("Insert_Time", "Temperature", "Humidity", "PmTen", "PmTwoFive") VALUES (%(Insert_Time)s, %(Temperature)s, %(Humidity)s, %(PmTen)s, %(PmTwoFive)s)'

ins = weather_table.insert().values(Insert_Time=datetime.datetime.utcnow(), Temperature=25, Humidity=75, PmTen=100, PmTwoFive=55)

#>>> str(ins)
#'INSERT INTO weather ("Insert_Time", "Temperature", "Humidity", "PmTen", "PmTwoFive") VALUES (%(Insert_Time)s, %(Temperature)s, %(Humidity)s, %(PmTen)s, %(PmTwoFive)s)'
#
#>>> ins.compile().params
#{'PmTen': 100, 'PmTwoFive': 55, 'Temperature': 25, 'Insert_Time': datetime.datetime(2014, 5, 10, 5, 58, 21, 677234), 'Humidity': 75}

# Connect to engine and execute. 
conn = engine.connect()
# >>> conn
# <sqlalchemy.engine.base.Connection object at 0x2715890>
result = conn.execute(ins)
# View result in psql
# mylocaldb=# select * from public.weather;
#           Insert_Time          | Temperature | Humidity | PmTen | PmTwoFive 
# -------------------------------+-------------+----------+-------+-----------
#  2014-05-10 05:58:21.677234+08 |          25 |       75 |   100 |        55

The critical functions has been pointed out in the above python file. Now we will consider how to run these functions at background. This will lead to next topic, “Run multiple process in a single Heroku dyno”.

Write Python Weather APP on Heroku(6)

May 10, 2014
Technology

Risk

On Google App Engine it’s very convinient to setup a crontab task, while in heroku setting up a crontab task will occupy the material, thus will use another dyno, each dyno will cost $30/month. For avoiding this, we will re-design our Weather App.
Following is the detailed explanation on heroku’s dyno:
Heroku allows you to run one free dyno (or actually they give you 720 free dyno hours per month, which corresponds to one dyno constantly running). This means that if you choose to run one web dyno and one worker dyno (celery in this case), you’ll be charged for 720 dyno hours. However, if you have a very small project, or your’re working on a project that hasn’t been released yet, you can avoid this cost.
A heroku dyno is like a process, and in this process you can actually spawn new processes, as long as you stay within the limit of 512 mb ram (the process also only has one CPU core). Heroku suggests that you use foreman when you run your application on your local machine, but you can actually use foreman on heroku, in order to run multiple processes in a single dyno.
On Heroku, we don’t have physical machines; in fact there isn’t the concept of “machine” at all. Instead, Heroku has Dynos, which are described as “lightweight containers” for UNIX processes. From their documentation:

[A Dyno] can run any command available in its default environment combined with your app’s slug

Solution

Celerywww.celeryproject.org
or
Honchohttps://github.com/nickstenning/honcho
We will try Honcho first, because it’s based on python, so won’t affect our code format.

Celery Way

Install redis and celery:

pip install redis celery

Remember to use pip freeze to update the requirement.txt file.
We should also enable the RedisToGo plugin, install it via CLI:

$ heroku addons:add rediscloud

If you don’t have a credit card, then your installation of plugin will be fail. We will register an account on www.redislabs.com, then we will continue our setting. ]

heroku config:set REDISCLOUD_URL="http://Resouce_Name:Redis_Passwod@pub-redis-xxxx.xxx.xxx..garantiadata.com:1xxx3"

Your heroku app will restart, then we can test this redis database via following commands:

$ heroku run python
Running `python` attached to terminal... up, run.8246
Python 2.7.6 (default, Jan 16 2014, 02:39:37) 
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> import urlparse
>>> import redis
>>> url = urlparse.urlparse(os.environ.get('REDISCLOUD_URL'))
>>> r = redis.Redis(host=url.hostname, port=url.port, password=url.password)
>>> r.set('foo','bar')
True
>>> r.get('foo')
'bar'

An Article of using celery is right after this article, so we end this article and in next one we will re-design the web app to fit the celery way.

Write Python Weather APP on Heroku

May 9, 2014
Technology

Accounting Setting

First you should have heroku accounting, then create an app on heroku, write down its repository information, mine is listed as following:

Your app, python-weather-app, has been created.    
App URL:    
http://python-weather-app.herokuapp.com/   
Git URL:    
git@heroku.com:python-weather-app.git      
Use the following code to set up your app for local development:

git clone git@heroku.com:python-weather-app.git -o heroku

Suggested next steps    
 Get started with Heroku.   
 Add some collaborators.    
 Check out some of our great add-ons.

Before you continue, make sure you have install heroku tools:

$ yaourt -S heroku-toolbelt

Create HelloWorld App

Use github for recording all of the source code.
Create a repository on github, mine is at “git@github.com:kkkttt/herokuWeatherApp.git”, then:

$ pwd
/home/Trusty/code/herokuWeatherApp
$ touch README.md
$ git init
$ git add README.md
$ git commit -m "first commit"
$ git remote add origin git@github.com:kkkttt/herokuWeatherApp.git
$ git push -u origin master

First login with heroku:

$ heroku login
Enter your Heroku credentials.
Email: xxxxxx@gmail.com
Password (typing will be hidden): 
Authentication successful.

Now create the folder which holds WeatherApp and create the venv, later we will use virtual environment for working:

$ virtualenv2 venv
New python executable in venv/bin/python2
Also creating executable in venv/bin/python
Installing setuptools, pip...done.
$ source venv/bin/activate
(venv) $ pip install Flask gunicorn

Now we write a very simple python file, name it “hello.py”:

import os
from flask import Flask

app = Flask(__name__)

@app.route('/')
def hello():
    return 'Hello World!'

Create a Procfile in the root directory which holds our own App:

$ cat Procfile 
web: gunicorn hello:app

Use foreman for preview the web app:

$ foreman start
14:53:37 web.1  | started with pid 30946
14:53:37 web.1  | 2014-05-09 14:53:37 [30946] [INFO] Starting gunicorn 18.0
14:53:37 web.1  | 2014-05-09 14:53:37 [30946] [INFO] Listening at: http://0.0.0.0:5000 (30946)
....

Open your browser and visit “http://127.0.0.1:5000” and you can see “Hello World” is in browser.
Now make the requirement file in the root folder:

$ pip freeze>requirements.txt
(venv)$ cat requirements.txt 
Flask==0.10.1
Jinja2==2.7.2
MarkupSafe==0.23
Werkzeug==0.9.4
gunicorn==18.0
itsdangerous==0.24
wsgiref==0.1.2

Deploy It To Heroku

We have to ignore the temp files, so we add following into our .gitignores file:

$ cat .gitignore
*~
*.pyc
venv/

We add the app into the heroku:

$ heroku git:remote -a python-weather-app
Git remote heroku added
(venv)$ git remote -v
heroku	git@heroku.com:python-weather-app.git (fetch)
heroku	git@heroku.com:python-weather-app.git (push)
origin	git@github.com:kkkttt/herokuWeatherApp.git (fetch)
origin	git@github.com:kkkttt/herokuWeatherApp.git (push)

Deploy:

$ git push heroku master

Now open your browser and visit “http://python-weather-app.herokuapp.com/" you will see the webpage displays “Hello World!".

很惭愧，就做了一点微小的工作

Write Python Weather APP on Heroku(8)

Deploy on Heroku

Prevent Heroku from Sleeping

Celery and RabbitMQ on ArchLinux

Installation

Run Simple Tasks

Periodic Tasks

To Be Thought

Write Python Weather APP on Heroku(5)

View the historical sqlite3

Create Database In Postgres

Select Other Data Formats

Database Operation

Write Python Weather APP on Heroku(6)

Risk

Solution

Celery Way

Write Python Weather APP on Heroku

Accounting Setting

Create HelloWorld App

Deploy It To Heroku